Musings on Data, Part 1: lakes, houses, clouds, etc.

Renu Tewari

January 11, 2025

2024 has been a remarkable year for AI and BI. While AI has been dominating all discourse since chatGPT exploded on the scene in 2022, BI and related technologies made quiet architectural leaps in 2024. The buzz began with Iceberg, the open table format for lakehouses, and the acquisition of Tabular, the company created around managing Iceberg tables. Tabular was founded by the original creators of Iceberg when they were at Netflix, and Databricks, the company that first introduced us to the concept of a lakehouse with Delta lake, later acquired Tabular for a pretty penny. To understand why an open table format is a big deal let’s start at the very beginning, a very good place to start (apologies Julie Andrews).

Traditionally, Data Warehouses were designed for OLAP queries on structured data with data sizes mostly under a terabyte. Teradata, aptly named, was one such industry leader in the OLAP space. Warehouses typically had a 3-tier architecture and supported multiple schema designs to logically organize the data: star (a fact table with multiple dimension tables), snowflake (a fact table with multiple dimension tables and intermediate dimension tables), and galaxy (multiple fact tables sharing multiple dimension tables). As data and primarily unstructured data grew, “bigdata” processing took off with Hadoop, inspired by Google’s seminal papers, becoming the primary player. Network speeds were still low, with 1GBps available for the few big players. Shared storage over SANs was expensive. Computing with locally attached storage was the only viable alternative that the Hadoop filesystem (HDFS) was designed around. The term datalake became synonymous with Hadoop MapReduce (MR) and HDFS and data sizes grew to multiple petabytes. Data was processed, cleaned, and decorated (joined with operational data) in the datalake and then ETLed into a data warehouse for business analytics running OLAP queries. Hive from Facebook and Impala from Cloudera tried to change that by bringing analytics directly to the datalake. Thus warehouse concepts of tables, partitions, and SQL-like languages evolved to bring some structure to the data in the datalake. Although both Hive and Impala soon lost the race to Spark and other analytical engines, Hive Metastore (HMS) endured. Hive Metastore was quite independent of Hive and provided the table catalog with table definitions, columns names, data types, serde, and storage file path etc. Soon Spark/MR +HMS+HDFS became the de facto bigdata stack.

While datalakes grew, another revolution was happening with cloud architectures becoming more commercially viable. Cloud storage, especially the blob/object storage pioneered by Amazon’s S3, became dirt cheap. The traditional data warehouse soon evolved into a cloud warehouse, led by Snowflake, with disaggregated compute and storage, each scaling independently. Soon, every cloud provider had a cloud warehouse offering. In parallel, datalakes moved to the cloud and all cloud vendors offered a bigdata stack. Network throughput was not a big concern. Bigdata seemed perfect for the high throughput with high latency model of cloud storage and compute. The cloud architecture had done a complete 180 degrees on the design of the bigdata stack. Instead of compute being local to storage it became disaggregated and shared via a REST API. The HDFS getBlockLocations API was set free as all blocks were remote. The de facto cloud bigdata stack now had Spark/MR+HMS+HDFS shim+cloud object store (e.g., S3).

For datalakes in the cloud, leveraging a cloud object store for reading and writing tables with Hive metastore catalog threw up new challenges. To understand this better let’s consider what exactly represents a datalake table? In HDFS it was basically a collection of files in a directory with a table schema and a partitioning scheme (typically the partition key is based on the date column). Anyone building filesystems knows that the ls command on a very large directory is slow as it reads multiple blocks from disk (ls -l is even slower). The list command on a cloud store’s bucket is, similarly, not something you want to do regularly. HMS required a listing of the table directory or bucket to get the set of files belonging to a table. Moreover, there was no consistency. Data was being written and read without locks. New files can be created and old files can be deleted. The table listing was not guaranteed to always return the correct and latest list of files. The table contents could change while a query was running. The datalake on the cloud was rather basic!

With data becoming larger it was expensive to ETL it into a warehouse for OLAP analytics. But the cloud datalake was too inconsistent for running a traditional warehouse query that expected a consistent response. Database folks love ACID. A better table format was needed to provide ACID guarantees on top of the datalake. This need resulted in the birth of the lakehouse. While Databricks introduced DeltaLake, the Hadoop community had Apache Hudi (aptly named for Hadoop update, delete, insert), and Netflix introduced Iceberg. The basic idea behind all of them is to avoid listing files in a directory/bucket by directly keeping the list as part of the table metadata, snapshot isolation for reads and writes, and providing consistency by supporting copy-on-write or merge-on-read or both. It is unclear which specific table format will stand the test of time but currently, Iceberg, has the most momentum. Recently, AWS announced S3 tables that provides a managed Iceberg table service backed with data in S3. A table service at a minimum helps manage snapshots, small file compaction, clustering, retention policies, etc.

With an open table format, the warehouse can be deconstructed into distinct components and free from vendor lock-in. Without that, if you own the format, you own the data (if data is the new oil then does that make you the Saudi prince!). With community managed table formats, the entire lakehouse can now be democratized, consisting of a variety of compute engines, a standard table format and data backed by cloud storage. Even Spark, the dominant datalake engine, is showing its age with too many parameters to tune, lack of checkpointing, shuffle scaling issues, and tricky resource sizing problems. With a democratized lakehouse, new special purpose engines can slice and dice the data in the lakehouse without any expensive transformations and copying.

(Note that in all this the file format has remained unchanged with columnar ones like Parquet being standard for on disk and Apache Arrow for in memory). The lakehouse stack now becomes BYOE+open table format+BYOC+cloud object store with data sizes in the exabytes. (BYOE= bring your own engine, BYOC = bring our own catalog). Engines could be anything from Spark, Trino/Presto, Dremio, Flink, or frameworks for AI training like TF and Pytorch. And catalogs can be anything from Databricks’ Unity, Snowflake’s Polaris, Linkedin’s Openhouse and even our good old HMS.

The deconstructed lakehouse, however, is not the end of the story for data. It is in fact the tip of the iceberg (some pun intended). There is operational data in relational and noSQL datastores, real time data in streaming platforms like Kafka, unstructured data in NAS filers and everything else in object stores. With AI and ML training becoming a necessity for businesses, more multi-modal data from images, zoom videos, customer support call archives, JIRA tickets, wiki pages, pdf documents, slack archives etc. will become part of the data pipeline. Where do we go from here as the world of BI and AI converge around even more data? More musings on this in Part 2.