Big data, in recent years, is not just a buzzword, but also a growing need for ambitious companies. With skyrocketing data scale and stringent requirements on data freshness, big data-related scenarios are becoming complicated and multi-dimensional. Thus, many companies use real-time data warehouses to meet their business demand.
But data warehouses are not the only option. An emerging category of databases, Hybrid Analytical/Transactional Processing (HTAP) databases, can serve you just as well as data warehouses, if not better. HTAP databases can handle Online Transactional Processing (OLTP) workloads and respond quickly to big data analytical requests in real time.
In this post, we will share the usage, cost, and technology selection of real-time big data analytics. We’ll also introduce TiDB, an open-source HTAP database designed for massive data. We hope this will help you in your search for a big data architecture for your organization.
For different roles in an organization, real-time data analytics plays a different part in their decision-making process:
These three roles often need real-time data analytics to make decisions. In addition, because they query long time spans, the data scale must be taken into account. It is challenging for the infrastructure software to provide real-time data analytics on a large data scale.
As the business evolves, users are no longer satisfied with performing data analytics on only offline data. They want to base their analytics on fresher data—even on data on the fly. To perform analytics on fresh data, their Online Transactional Processing (OLAP) architecture needs an OLAP-oriented warehouse on top of it.
The current data solutions each have their own problems:
In addition, the existing big data platforms based on the above technologies have several disadvantages: complicated extract, transform, load (ETL), high storage costs, lengthy data transfer processes, and convoluted tech stacks.
In an effort to build and maintain a system like the one above, users must bear the high costs of a fully-functional data team:
Building a team that covers all these positions will cost the company a fortune, not to mention that some high-level talents are rare in the labor market. Building the team might take six months to a year, and developing the data platform another one to two years. In this process, you might need to introduce various technologies and increase the maintenance complexity.
TiDB is an open-source, distributed database that supports HTAP workloads. It features horizontal scalability, strong consistency, high availability, and real-time data analytics. In addition, TiDB is MySQL compatible and easy to manage.
One of TiDB’s biggest strengths is its HTAP architecture. In TiDB, the row store and column store can coexist and provide service for different kinds of queries.
TiDB also supports elastic scaling:
TiDB supports standard SQL syntax and the MySQL protocol:
TiDB is easy to manage:
TiDB provides a variety of tools to import, export, backup, and restore data:
By default, these tools operate on all tables. However if you only need a subset of tables, they also support table filters to specify the subsets.
TiDB users enjoy many community resources:
Many companies have adopted TiDB to empower their business with HTAP capabilities. One example is ZTO Express.
ZTO Express is a leading express delivery company in China and one of the largest express delivery companies in the world. They used to use Oracle Exadata, but as their business quickly developed, it no longer met their requirements:
To build a real-time data warehouse, they tested HBase and Kudu. But Kudu was not compatible with the existing technology stack, and HBase had very limited support for multi-dimensional queries. Finally, they adopted TiDB to solve their problems.
ZTO Express built their system based on TiDB as follows:
This new database infrastructure has brought them the following benefits:
For more details on how ZTO Express uses TiDB, read their story about their 300% IT efficiency boost.
Department of Information Technologies: https://www.ibu.edu.ba/department-of-information-technologies/