Coming Tidal Waves in the (Data) Lake

From Warehouse to Lake to Lakehouse: A Brief Voyage Through Time

Imagine a fortress (data warehouse) — structured, sturdy, great for analytics but too stiff to dance with new data types or scale playfully. Then came the data lake — a dumping ground of CSVs, logs, and random JSONs, cheap and flexible, yet chaotic. Finally — Lakehouse! It delivers warehouse-grade governance and lake-style openness in one slick architecture. Open table formats like Iceberg, Delta Lake, and Hudi brought ACID transactions, time travel, schema evolution, and query pruning to object storage. Now that’s harmony. Lakehouse has since become the mainstream for big data analytics, as evident by AWS establishment of S3 table as one of its first class citizens at the end of 2024.

Lakehouse: The Layered Cake of Modern Data Architecture

Here’s the architecture laid out visually in this graph:

1. Storage (Base Layer) — AWS S3, Azure Blob, GCS: cheap and scalable storage.
2. File Format — Parquet (and friends ORC/Avro): efficient, columnar formats for structured data.
3. Open Table Format — Iceberg, Delta Lake, Hudi: adds ACID, versioning, partitioning, schema evolution.
4. Metadata/Catalog Layer — Iceberg Catalog, Unity Catalog, Polaris, AWS Glue, Apache Gravitino.
5. Compute Engines — Spark, Trino, Presto, DuckDB.

Lakehouse key differentiations

Catalog/Metadata Service

A catalog is crucial in a modern lakehouse architecture, particularly when using table formats like Apache Iceberg, because it serves as the central hub for managing and organizing table metadata. This enables a range of functionalities that are essential for efficient, scalable, and reliable data management within the lakehouse environment. Iceberg’s architecture revolves around a layered metadata system that elegantly orchestrates table state and operations. Here’s how the magic happens:

• Catalog: Holds the pointer to the active metadata file for a table.
• Metadata File (metadata.json): Describes schema, partition specs, and current snapshot ID.
• Manifest List (Avro): Points to manifest files for a snapshot.
• Manifest File (Avro): Lists actual data files (e.g., Parquet), with stats like row counts, partition info, and column min/max.
• Data Files: Stored in Parquet for efficient analytics processing.

When performing CRUD ops, updates create new metadata, manifests, and snapshots atomically — guaranteeing transactional correctness and time-travel functionality. Readers follow the catalog → metadata → manifests → data files path for efficient query planning, skipping expensive directory listings.

Parquet — The File Format Powerhouse

Parquet has become the de facto file format for modern lake house. Parquet delivers a hybrid-columnar approach, organizing data in chunks of columns — boosting both compression and query speed. It trims storage size by up to ~87% and accelerates reads dramatically — even 34× in benchmark comparisons with CSV.

Parquet supports:

• Column-wise compression and encoding (snappy, gzip, LZO, zstd, etc.)
• Dictionary and run-length encoding for dense data
• Column pruning and predicate pushdown for faster, targeted reads

In lakehouse contexts, Parquet shines as the data storage layer under table formats like Iceberg, enabling performant, queryable data landscapes.

The Metadata Wave

Due to the central role of metadata/catalog service in the lakehouse. It’s drawn attention from big, established players as well as startups to battle for the strategic high ground. Big players want to solidify their offerings, and newcomers want to out-innovate incumbents from either the breath and ease-of-use angles.

2.1 Databricks Acquisition of Tabular

In a bold move, Databricks acquired Tabular (founded by Iceberg’s creators), signaling how deeply fundamental metadata services are to modern lakehouses. It’s like buying the DJ who runs the metadata party.

2.2 Apache Gravitino: Metadata’s Meta

Meet Apache Gravitino — a metadata service that doesn’t just manage lakehouse tables; it spans multi- and hybrid-cloud, files outside the lakehouse, and even AI models. Think of it as the mega-librarian who knows not just where your books are, but where your coffee cup, earbuds, and training model for GPT are too. This “metadata-of-metadata” approach promises universal visibility and governance across varied assets.

2.3 DuckDB’s DuckLake: SQL Metadata, Simplified

DuckLake swaps Iceberg’s manifest layers for a familiar SQL-backed metadata catalog (DuckDB/Postgres/SQLite). This gives you simpler metadata management, ACID transactions, and cross-table consistency — but adds the overhead of managing yet another DB. It’s like trading in a multi-step ladder for a single-step elevator… one that needs its own electrical maintenance.

The File Format Wave

While Parquet now realms supreme in structure, semi-structured data format in Lakehouse, it was introduced in early 2010s before the current AI explosion. It doesn’t have enough provision for truly understructured data or vectors. Enter LanceDB and its Lance file format: reengineered to natively support vectors, blobs, images — basically unleashing AI unstructured data into lakehouses. It sacrifices Parquet’s row-group optimizations and compression, but brings first-class multimodal handling, primed for AI workloads. It’s the new skinny jeans of file formats — tight and uncompressed, but oh — so stylish for modern data. My bet is that Lance will augment, not replace, Parquet in future data stacks.

Parting Thoughts

• A warehouse is an all-organized library where you’re allowed only to the recommended list.
• A data lake is like your attic — everything’s there, but good luck finding your passport or getting rid of junks.
• A lakehouse? Finally, the librarian with a Roomba walks in and says, “I got you, buddy.”

New AI use cases will demand skills updates from the librarian. So keep an eye on the horizon: the tides are rising, and they’re bringing metadata catalysts and AI-ready formats — just the right waves to surf into the future.