Data Guardians: What is a DWH and Why Company Data Is Almost Useless Without It

Large companies in Kazakhstan today face challenges with data storage and utilization. Key information is stored in disparate systems. Employees spend a significant amount of time consolidating and processing this data. Why should large companies consider implementing a Data Warehouse (DWH) tool, and how should they go about it?

Timur Kashkinbayev, CTO and Project Architect at OneCode IT Company

What is DWH

I have been working in the IT field for over 12 years. I have worked with large holding companies in Kazakhstan and developed mobile applications, web applications, web portals, and accounting systems. Large companies today struggle with data storage and consolidation.

Data is the new "oil." The more data a company has, the more effectively it can attract new customers, develop growth strategies, and strengthen its market position. A useful tool for storing and processing data is a Data Warehouse (DWH). This is a single corporate archive of data from various sources: systems, departments, and companies.

Foreign companies have long transitioned to DWH and utilize all its advantages. Unfortunately, however, companies in Kazakhstan are still adopting such types of storage to a lesser extent.

Existing problems in large companies

Most companies have a vast number of information systems. Data may be unstructured or outdated. Managing and analyzing large volumes of data requires powerful technologies and resources. Different systems and data sources may use different formats and standards, which complicates the process and increases the time required for processing.

A corporate Data Warehouse can serve as a solution to these problems. DWH consolidates important business information and allows for the automatic preparation of reports.

Case study from practice: At one of my previous workplaces, the IT department was tasked with collecting and processing sales data for the past three years. The report was prepared three months later because employees had to restore databases and export data into a unified table format. Another department manually prepared reporting documents because the company had changed several accounting systems over the past few years. Such requests can come in daily, so a solution was needed to reduce labor costs and save time. We began working on data collection for DWH, and Data Engineers developed the system.

Advantages of implementing DWH

DWH differs from the ordinary databases used in business. For example, ordinary databases store information from specific systems where they are installed. In the HR department's database, you can only see data about employees, and there will be no data about supplies.

DWH is built on a different principle: this corporate storage consolidates information from all company departments. If the company is large, obtaining data from different sources requires collecting permissions and access rights. In DWH, everything needed is already available in a ready-to-use format.

Additionally, a Data Warehouse can reflect all historical data and archival records of your company, which is very convenient from an analytical perspective. Typically, ordinary databases are not used to store information from the last ten years of a company's operations. However, for DWH, such a timeframe is normal. A Data Warehouse always contains the latest versions of data because all information is fed in regularly.

Businesses use Data Warehouses to create reports and perform analytical processing. By using a warehouse, company executives can justify important decisions by backing up their ideas with quantitative data. This is the foundation of Business Intelligence (BI). Having measurement results for hundreds of indicators at hand allows one to formulate hypotheses and conduct experiments.

Another advantage of DWH is that it is an ideal source for machine learning (ML) and Data Science technologies, as well as for new high-load systems within the company's ecosystem.

DWH easily adapts to current business logic and is very popular among business analysts. This type of storage is suitable for large companies, especially retail chains, manufacturing companies, and the banking sector.

How data migration works

Every CTO of a large company should already be thinking about how to transition to DWH. DWH is not a ready-made box solution. This type of storage needs to be adapted to the company's requirements. In simpler terms, it must be created from scratch depending on the number of accounting systems and the data storage architecture.

After the code is written, the next step is to migrate data from old accounting systems. This is usually done by Data Engineers and Data Analysts. They can easily understand the data storage format. However, certain difficulties may arise here.

The main problem encountered when transitioning to DWH independently is migrating a "snapshot" of all data from 1C into DWH. Why 1C? Because companies in the CIS region often use 1C systems for internal accounting, although other systems may also be involved.

Migrating data from 1C to DWH, exceeding more than a million records, takes a long time. This can be avoided. I propose using a methodology that I apply in my data integration module. It is necessary to ensure a common "language" for communication to facilitate data exchange between different 1C systems and DWH.

To ensure that each system can understand the data structure it will receive in the message, we propose adopting a unified JSON format standard within the company. This format is ideal for both 1C and systems built on other platforms. You can develop this standard yourself, regulating it internally within the company. It is similar to an international communication language.

Maintaining data relevance

After you have created a DWH storage base containing the full volume of data from 1C, you can build data marts or connect external systems to it. However, it is very important to maintain data relevance.

Typically, at OneCode, we clearly distinguish the "temperature" of data. "Cold" data refers to information that we use very rarely and which may be updated from once a day to once a month. "Warm" data requires relevance to be maintained from 15 minutes to 1 hour. Typically, this data is used infrequently, and any delay will not affect its relevance. However, the most relevant are "hot" data, because they influence the decision-making process in the company and must be exported instantly, within one minute.

If your company's operations use only "cold" data, you do not need to use a data bus; instead, you can maintain their relevance daily by exporting a "snapshot" of data from 1C.

However, if you need "hot" and "warm" data, we recommend using a message broker, such as Apache Kafka or others, which are not built on API requests. Apache Kafka is a distributed messaging system between server applications in real-time. It has high throughput, scalability, and reliability, and is therefore used by companies that work with large volumes of data.

A Data Warehouse represents a stable source used for detailed information viewing. With its help, you can process queries in any company at high speed.