The Triad of Open Table Formats [1/n] — History, Motivation & Design Philosophy

Context

Over the last decade, Open Table Formats have become the foundation of modern lakehouse architectures. Apache Hudi, Apache Iceberg and Delta Lake emerged from very different operational pain points, yet eventually converged toward a shared goal: bringing warehouse-grade guarantees to open object storage systems.

Having worked primarily with Delta Lake first-hand at Atlassian, I wanted to better understand how the other formats evolved, what problems they were originally designed to solve and why Apache Iceberg in particular has seen such rapid industry adoption.

This blog explores the historical context, architectural motivations and design philosophies behind all three systems. More than just features, I wanted to understand the operational pain each system was reacting to — because in distributed systems, architecture is often a reflection of the accumulated operational scar.

A Look Into the Past

Before diving into Hudi, Iceberg and Delta Lake, it helps to understand the world that existed before them.

Around 2008–2012, the industry was transitioning from traditional data warehouses toward Hadoop-based data lake systems. This coincided with the explosion of:

• clickstream data
• logs
• sensor data
• social media data
• semi-structured and unstructured datasets

Traditional warehouses struggled because they required:

• schema definition upfront
• ETL before storage
• expensive proprietary infrastructure

At the same time, the Apache Hadoop ecosystem — primarily HDFS and MapReduce — made it possible to store enormous volumes of raw data cheaply.

This gave rise to the idea of the Data Lake.

The term itself was coined around 2010 by James Dixon while at Pentaho.

His famous analogy described:

• Data marts as bottled water — cleaned, processed and packaged
• Data lakes as natural bodies of water — raw, flexible and usable for many purposes

The core philosophy behind data lakes was radical for its time:

Store everything first. Model later.

Early data lake architectures roughly looked like:

Raw Data
   ↓
HDFS (Parquet / ORC / Avro)
   ↓
Hive / Pig / Spark

These systems enabled:

• cheap horizontal scaling
• schema-on-read
• distributed compute
• raw file storage

But they also lacked:

• ACID guarantees
• reliable concurrent writes
• schema evolution
• transactions
• robust metadata management

As the industry moved toward cloud-native object stores like:

• Amazon S3
• ADLS
• GCS

the economics improved dramatically, but the correctness problems remained.

This eventually led to the rise of Lakehouse architectures — systems that combined the flexibility of data lakes with warehouse-grade guarantees.

So What Exactly Is “Open” in Open Table Formats?

When people say:

• “open table format”
• “open data lake”
• “open lakehouse”

the word open is often overloaded.

In practice, it usually refers to three different kinds of openness.

1. Open Storage Formats

At the lowest layer, data is stored in open formats such as:

• Parquet
• ORC
• Avro

These formats are:

• publicly documented
• widely supported
• engine-independent

Meaning the data itself is not locked inside proprietary warehouse internals.

A Parquet file written by Spark can also be read by:

• Trino
• DuckDB
• Flink
• Pandas
• Snowflake

and many others.

2. Open Table Specifications

Formats like:

• Apache Hudi
• Apache Iceberg
• Delta Lake

define:

• metadata layout
• transaction semantics
• snapshot rules
• schema evolution behaviour
• commit protocols

through public specifications.

This means:

• Spark can write
• Trino can read
• Flink can stream
• multiple engines can interoperate

without tight vendor coupling.

Iceberg, in particular, pushed this idea aggressively by treating the table abstraction itself as an open specification.

It became:

a shared interoperability contract.

3. Storage-Compute Decoupling

Traditional warehouses tightly coupled:

• storage
• metadata
• compute
• execution engine

Open lakehouses broke this apart.

Now organisations could mix:

• S3 for storage
• Spark for ETL
• Trino for BI
• Flink for streaming
• DuckDB for local analysis

all operating on the same tables.

Cloud object storage dramatically changed economics and naturally led organisations to ask:

“Why should my data belong to a single compute engine?”

Open table formats emerged as the answer.

Hive & Spark: Precursors to Modern Open Table Formats

Hive and later Apache Spark were foundational precursors to modern open table formats because they normalised the idea of querying open file formats directly from distributed storage.

However, they still lacked:

• atomic commits
• snapshot isolation
• reliable concurrent writes
• rollback semantics

That missing transactional layer is exactly what Hudi, Iceberg and Delta Lake introduced — albeit through very different architectural philosophies.

Hudi’s write path began life as a Spark library focused on incremental processing and efficient upserts for mutable datasets. Iceberg, as covered later, took heavy inspiration from the ubiquity and limitations of the Hive table format, rethinking table state and metadata management from first principles for petabyte-scale analytics. Delta Lake, meanwhile, was laser-focused on bringing warehouse-grade ACID reliability to the enterprise Spark ecosystem through a transaction-log-centric design.

Year Format Created At 2016 Apache Hudi Uber Technologies 2017 Apache Iceberg Netflix 2019 Delta Lake Databricks

Apache Hudi

Original Problem It Tried to Solve

“How do we efficiently support streaming upserts and incremental ingestion on a data lake?”

At the time, most data lakes were:

• append-only
• batch-oriented
• poor at handling updates and deletes

This was painful for:

• CDC pipelines
• mutable datasets
• near-real-time analytics

Uber’s Core Pain

At Uber scale:

• trip data continuously changed
• events arrived late or out of order
• datasets required frequent corrections

Appending new Parquet files indefinitely created:

• stale records
• duplicates
• expensive partition rewrites

Hudi’s Core Conceptual Shift

Traditional data lakes largely treated files as immutable forever.

Hudi challenged that assumption by treating datasets as living mutable entities.

Instead of repeatedly recomputing entire partitions, Hudi introduced:

• record-level upserts
• indexed mutations
• incremental processing semantics

This enabled data lakes to behave more like mutable operational systems rather than static analytical archives.

Key Innovations

1. Record-Level Upserts

Hudi enabled:

• inserts
• updates
• deletes

directly on Parquet-based storage.

2. Incremental Consumption

Downstream systems could ask:

“What changed since commit X?”

instead of rescanning entire datasets repeatedly.

This dramatically reduced:

• compute cost
• latency
• operational complexity

3. Timeline-Based Table State

Hudi introduced:

• commit timelines
• transactional metadata
• versioned table state

This enabled:

• snapshot isolation
• rollback
• recovery semantics

4. Copy-on-Write vs Merge-on-Read

Copy-on-Write (CoW)

Updates rewrite Parquet files immediately.

Benefits:

• simpler reads
• better analytical scan performance

Tradeoff:

• more expensive writes

Merge-on-Read (MoR)

Updates first land in delta log files before eventually compacting into Parquet.

Benefits:

• fast ingestion
• low-latency writes

Tradeoff:

• more complex reads

Conceptually, this feels very similar to log-structured and LSM-tree-inspired storage engines.

Architectural Bias

Hudi’s worldview was fundamentally:

“The write path is the bottleneck.”

It optimised heavily for:

• streaming ingestion
• mutable datasets
• CDC pipelines
• incremental recomputation

Apache Iceberg

Original Problem It Tried to Solve

“How do we make massive analytical tables reliable and scalable across multiple compute engines?”

At Netflix scale, Hive-style tables were beginning to break down.

Problems included:

• millions of partitions
• directory listing bottlenecks
• unsafe concurrent writes
• painful partition evolution
• metadata scalability issues

The Problem with Hive-Style Tables

Traditional Hive tables relied heavily on directory structures such as:

/date=2026-05-01/

for partition discovery.

This tightly coupled:

• table state
• partitioning
• physical directory layout

That model became increasingly fragile on cloud object stores.

Iceberg effectively redesigned analytical tables from first principles.

Iceberg’s Core Innovations

Iceberg introduced:

• snapshot-based tables
• manifest trees
• hidden partitioning
• field-ID-based schema evolution
• engine-neutral metadata

The key insight was:

The table abstraction itself needed redesigning.

Key Architectural Ideas

1. Hidden Partitioning

Iceberg introduced hidden partition transforms that enabled:

• partition evolution
• cleaner query logic
• decoupling of queries from physical layout

2. Stable Field IDs

Instead of relying purely on column names or positions, Iceberg used stable field IDs.

This enabled safe:

• schema evolution
• column renames
• column reordering

across engines.

3. Snapshot-Based Metadata

Iceberg introduced snapshot-based table state managed through manifest trees.

This enabled:

• atomic reads
• time travel
• scalable metadata planning
• snapshot isolation

without relying on filesystem directory discovery.

Architectural Bias

Iceberg optimised heavily for:

• metadata scalability
• analytical query planning
• engine interoperability
• correctness on object stores

It wasn’t merely introducing new features — it was redefining how analytical tables should behave in a cloud-native, multi-engine world.

<aside> 💡

https://www.alluxio.io/videos/apache-iceberg-a-table-format-for-hige-analytic-datasets?utm_source=chatgpt.com is one of the earliest talks I found on Apache Iceberg, presented by its creator Ryan Blue.

On watching it, I realised Iceberg was, in many ways, ahead of its time. Its emphasis on open specification and standardisation laid the groundwork for multi-engine interoperability long before the industry fully appreciated how important that would become.

What struck me most was that Iceberg didn’t merely introduce new features — it introduced the very idea of a table format as a first-class abstraction over object storage.

Many of the things that feel obvious in hindsight today were deliberate design choices back then:

• using stable field IDs instead of column names/positions to enable safe schema evolution
• decoupling table state from directory structure
• snapshot-based metadata management for time travel and atomic reads
• hidden partitioning to avoid brittle query logic

You can almost see years of operational pain encoded into the design.

Interestingly, part of Iceberg’s eventual ubiquity stems from the fact that the Hive table format had already become the industry’s “lowest common denominator” — much like CSV for tabular data. Ryan Blue uses a great analogy in the talk: the goal was to evolve tables from the “CSV end of the spectrum” toward something more like Parquet — structured, scalable and explicit in its semantics.

</aside>

Delta Lake

Original Problem It Tried to Solve

“How do we bring warehouse-grade ACID guarantees to Spark-based data lakes?”

Before Delta Lake:

• Spark pipelines frequently corrupted tables
• partial writes were common
• schema evolution was fragile
• concurrent jobs created inconsistencies

At Databricks scale, this made production pipelines operationally brittle.

Delta Lake’s Core Insight

Delta’s key insight was elegant:

Object stores themselves did not need to support transactions.

Instead, transactional correctness could be achieved through a carefully managed append-only transaction log layered above immutable Parquet files.

This database-inspired WAL-style design became Delta Lake’s defining architectural choice.

Core Innovations

Delta introduced:

• append-only transaction logs
• optimistic concurrency control
• ACID transactions on object storage
• rollback semantics
• time travel
• unified batch + streaming semantics

Its philosophy was fundamentally pragmatic:

Bring database transaction guarantees into the Spark ecosystem with minimal operational complexity.

Why Build Delta When Iceberg and Hudi Already Existed?

By the time Delta Lake emerged, both Hudi and Iceberg already existed — but they were solving different problems.

Hudi focused on:

• streaming ingestion
• incremental processing
• efficient upserts

Iceberg focused on:

• metadata scalability
• clean table abstractions
• multi-engine interoperability

Delta Lake, meanwhile, focused on making Spark-based data lakes operationally reliable through:

• ACID transactions
• optimistic concurrency control
• append-only transaction logs
• unified batch + streaming semantics

Concluding Thoughts

Diving into the historical context behind these systems made one thing increasingly clear:

each format emerged from a very different operational pain point.

Format Original Pain Hudi Mutable streaming datasets Iceberg Metadata scalability & multi-engine analytics Delta Lake Reliable transactional Spark pipelines

That is why their architectures diverged so significantly. Yet despite their differences, the broader ecosystem has steadily converged toward interoperability because organisations increasingly demand:

• openness
• engine portability
• avoidance of vendor lock-in

<aside> 💡

What I found most fascinating is that Open Table Formats are not really about storage alone. They are fundamentally about:

• metadata management
• transaction semantics
• concurrency control
• correctness guarantees
• distributed systems design

all layered above open file formats like Parquet sitting inside cloud object stores.

</aside>

I hope to delve deeper into the architecture of each of these systems in the subsequent blogs.

References:

• https://www.uber.com/in/en/blog/hoodie/?from_challenge=1
• https://www.uber.com/in/en/blog/uber-big-data-platform/
• https://www.alluxio.io/videos/apache-iceberg-a-table-format-for-hige-analytic-datasets?utm_source=chatgpt.com
• https://thamizhelango.medium.com/why-netflix-created-apache-iceberg-a-revolutionary-table-format-for-big-data-1cc9ba1858d3
• https://www.databricks.com/company/newsroom/press-releases/databricks-launches-delta-combine-best-data-lakes-data-warehouses-streaming-systems
• https://www.databricks.com/blog/2022/06/30/open-sourcing-all-of-delta-lake.html

Disclaimer: While the ideas and core structure of this blog were conceptualised by the author, AI tools were used to assist in drafting and refining the content and as a sparring assistant.