The Triad (Hudi, Iceberg, Delta) of Open Table Formats [2/n] — Architecture Deep Dive

In the previous blog in the series, we explored the history and design philosophies behind Hudi, Iceberg and Delta Lake. In this post, we’ll examine how those philosophies translate into architecture.

In this post, we’ll look under the hood at how each system works: how table state is managed, how metadata is organised, how updates and deletes are implemented, how concurrency is handled and how each format approaches schema evolution, partitioning and query planning.

As we will see, the architectural centre of gravity for each format is remarkably different: Hudi revolves around the Timeline, Iceberg around the Metadata Tree and Delta around the Transaction Log. These foundational choices continue to shape their performance characteristics and operational tradeoffs today.

Apache Hudi

As covered in the previous blog, Hudi’s worldview was fundamentally shaped by Uber’s operational pain point: the write path was the bottleneck. Traditional Hive-style data lakes were good for large immutable batch datasets, but struggled with continuously mutating datasets generated by ride events, CDC streams and operational databases.

Hudi therefore optimised heavily for:

• high-frequency streaming ingestion
• mutable datasets
• upserts and deletes
• CDC pipelines
• incremental recomputation
• near real-time analytics

Instead of treating the data lake as a passive collection of Parquet files, Hudi introduced the notion of a transactional table abstraction sitting on top of object storage.

To achieve this, Hudi introduced several architectural innovations:

• a table abstraction that tracks all files belonging to a table instead of relying purely on partition-directory discovery
• a timeline abstraction that acts like a transaction log of all activity on the table, enabling snapshot isolation, rollback, incremental processing and time travel
• a dual-storage design inspired by LSM-tree/append-oriented systems, where updates are first appended to log files before being compacted into columnar base files.
• asynchronous table services like compaction, clustering and cleanup to continuously optimise storage layout in the background

Unlike traditional warehouse systems that optimise primarily for scan performance, Hudi consciously trades off some read complexity in favour of significantly lower write amplification and faster ingestion throughput.

Key Concepts

Hudi Timeline & Metadata Layer

The Hudi Timeline is the heart of Hudi’s architecture.

Every action performed on a Hudi table is recorded as an event on the timeline:

• commits
• delta commits
• compactions
• clustering operations
• cleanups
• rollbacks
• restores

Each action is associated with an instant time, allowing Hudi to reconstruct the exact state of the table at any point in time.

Conceptually, the timeline behaves similarly to:

• a database transaction log
• a WAL (Write Ahead Log)
• or an event-sourced state machine

This enables several powerful capabilities:

• Snapshot isolation: Readers query a consistent instant
• Time travel: Query historical snapshots
• Incremental processing: Consume changes since instant X
• Rollback & recovery: Restore failed commits
• Concurrent writers: Timeline-mediated coordination

Hudi uses **TrueTime-style semantics** for commit ordering to maintain globally ordered transactional state across distributed writers.

A subtle but important distinction from Hive-style lakes is that the table state is no longer inferred purely from filesystem listings. Instead:

Table State = Filesystem State + Timeline Metadata

This architectural shift is what transforms a passive data lake into a transactional lakehouse table.

Metadata Table

As data lakes grow to millions of files, one of the biggest bottlenecks becomes metadata operations themselves:

• listing partitions
• discovering files
• pruning candidate files
• locating records for updates
• collecting column statistics

Traditional Hive-style lakes rely heavily on recursive object-store directory listing operations which become increasingly expensive and slow at scale.

To solve this, Hudi introduced the Metadata Table — an internal Hudi-managed table that stores metadata about the dataset itself.

The metadata table is itself implemented as a specialised MOR table and maintains:

• file listings
• partition metadata
• bloom filters
• column statistics
• record indexes
• secondary indexes

This allows Hudi to avoid expensive filesystem scans and instead perform metadata lookups directly through optimised indexed structures.

Conceptually:

Object Store Listing
        ↓
Metadata Table Lookup

The metadata table is one of Hudi’s most important scalability optimisations because cloud object stores are extremely inefficient at high-frequency metadata discovery workloads.

Together, the Timeline and Metadata Table form Hudi’s transactional metadata backbone — enabling database-like table semantics on top of immutable cloud object storage.

Hudi File Layout

Physically, Hudi still stores data on open object storage using formats like Parquet and Avro. However, Hudi layers a richer logical structure on top of these files.

At the highest level:

Partitions → File Groups → File Slices → Base + Log Files

Partitions

Partitions are represented as directories, similar to Hive tables.

Example:

/orders/date=2026-05-27/

However, unlike Hive where partitions are mostly independent directories, Hudi tracks partition metadata centrally through the timeline and metadata table.

File Groups

Within each partition, Hudi organises data into file groups.

A file group is identified by a unique:

fileId

A file group represents the complete evolution history of a subset of records.

This is an extremely important concept because Hudi maintains the invariant:

record_key → file_group

Once a record key maps to a file group, that mapping is immutable.

This dramatically simplifies:

• updates
• indexing
• record location
• incremental processing

Conceptually, a file group behaves somewhat like a shard or segment responsible for a subset of records.

File Slices

A file group evolves over time through multiple file slices.

Each file slice corresponds to a specific commit instant and may contain:

• a base data file
• zero or more delta log files

Example:

fileGroupA
 ├── slice@T1 → parquet
 ├── slice@T2 → parquet + logs
 └── slice@T3 → compacted parquet

Readers consult the timeline to determine which file slice is valid for a given snapshot query.

Base Files vs Log Files

Hudi stores data in two complementary physical formats:

• Read/scan optimised: Parquet
• Write/update optimised: Avro log files

Parquet files provide:

• efficient columnar scans
• compression
• vectorised reads
• analytical query performance

Avro log files provide:

• append-efficient writes
• low-latency ingestion
• reduced rewrite amplification

This dual-storage strategy is central to Hudi’s architecture.

HFile & SSTable-style Metadata Storage

For metadata and indexing workloads, Hudi uses HFile, the immutable sorted key-value file format originally used by Apache HBase.

HFiles are heavily inspired by SSTable-style storage engines:

• immutable
• sorted by key
• block indexed
• optimised for random lookups

This is important because Parquet and Avro are optimised for large analytical scans not fast point lookups.

Hudi therefore uses:

• Parquet for analytical data
• Avro for streaming delta logs
• HFile for metadata/index lookup paths

Example:

customer_id → candidate file groups

HFiles allow Hudi to quickly jump to relevant metadata blocks instead of scanning large Parquet datasets.

Architecturally, this gives Hudi a hybrid design:

• Analytical scans: Parquet
• Streaming writes: Avro logs
• Metadata & indexing: HFile

This is also where Hudi most strongly resembles LSM/SSTable-inspired storage systems:

• append-oriented writes
• immutable file generation
• background compaction
• sorted key-value metadata structures

Hudi Table Types

Hudi supports two table types with fundamentally different storage tradeoffs.

Copy-On-Write (COW)

In COW tables, updates rewrite Parquet base files directly.

Update → Rewrite Parquet → Commit

Advantages:

• simpler reads
• excellent scan performance
• no merge cost during query execution

Disadvantages:

• higher write amplification
• slower ingestion for heavy update workloads

COW is ideal for:

• BI dashboards
• batch analytics
• read-heavy workloads

Merge-On-Read (MOR)

MOR introduces a more LSM-style architecture.

Updates are first appended to row-oriented delta log files:

Base Parquet + Delta Logs

Background compaction later merges these logs into optimised Parquet files.

This significantly improves write throughput while introducing additional read complexity.

MOR supports two query modes:

• Snapshot Query: Base files + logs merged
• Read Optimised Query: Only compacted Parquet files

This allows the same table to simultaneously serve:

• near real-time operational queries
• and stable analytical workloads

The design strongly resembles LSM-tree systems:

Hudi Query Types

One of Hudi’s most differentiated capabilities is its support for multiple query semantics on the same table.

Snapshot Queries

Returns the latest committed view of the dataset.

For MOR tables this may involve merging:

• base Parquet files
• delta log files

at read time.

Incremental Queries

Instead of scanning the entire dataset repeatedly, Hudi can return:

changes since instant T

This is extremely powerful for:

• CDC pipelines
• incremental ETL
• downstream recomputation
• streaming pipelines

Incremental querying is one of Hudi’s biggest architectural differentiators.

CDC Queries

Hudi can expose actual row-level changes:

• inserts
• updates
• deletes

between commits.

This allows Hudi tables to behave similarly to database change streams.

Time Travel Queries

Since the timeline tracks historical file slices, readers can query older snapshots:

SELECT*FROM ordersTIMESTAMPASOF'2026-05-01'

Read Optimised Queries

Available for MOR tables only.

These queries read only compacted Parquet base files while ignoring delta logs.

This provides:

• lower latency
• simpler execution
• stable scan performance

at the cost of slightly stale results.

Indexing

Efficient upserts require Hudi to quickly determine:

record_key → file_group

Unlike traditional databases, Hudi cannot rely on mutable B+Tree indexes because object storage is immutable and distributed.

Instead, Hudi uses distributed metadata-driven indexing structures optimised for large-scale file pruning and lookup.

Bloom Filter Index

The default indexing strategy.

Each data file stores a Bloom filter in its footer:

record_key → probable file match

Advantages:

• no external dependency
• lightweight
• scalable

Disadvantage:

• probabilistic false positives

HBase Index

Hudi can also use external Apache HBase for exact record location lookups.

This is especially useful for:

• small batches of random lookups
• ultra-low-latency key mapping

though it introduces operational complexity.

Record Index

Newer versions introduced record-level indexes stored in the metadata table itself.

These may be:

• global
• partition-scoped

allowing efficient record location without external systems.

Multi-Modal Indexing

Hudi increasingly treats indexing as a pluggable subsystem supporting:

• metadata indexes
• expression indexes
• secondary indexes
• column statistics
• partition statistics

Table Services

Hudi continuously runs asynchronous background services to maintain storage efficiency and query performance.

These services are deeply integrated into the timeline architecture.

Clustering

Streaming ingestion often creates:

• many small files
• poor sort locality
• fragmented layouts

Clustering reorganises file groups into better physical layouts.

Importantly, clustering in Hudi is fundamentally:

metadata-driven file replacement

— not low-level page reorganization like traditional databases.

This is because object stores are immutable.

Hudi clustering works by:

1. selecting candidate file groups
2. reading existing records
3. repartitioning/sorting them
4. writing new optimised files
5. atomically swapping file slices through the timeline

Clustering can:

• merge small files
• sort by frequently filtered columns
• improve locality
• optimise file sizes

without interrupting ongoing ingestion.

Compaction

Compaction is specific to MOR tables.

It merges:

• Parquet base files
• delta log files

into new optimised Parquet files.

This reduces:

• read amplification
• merge overhead
• query latency

while preserving high ingestion throughput.

Cleanup

Cleanup removes obsolete file slices no longer required for:

• active snapshots
• rollback windows
• time travel retention

This controls storage growth.

Archival

Over time, the timeline itself can become very large.

Archival moves older timeline metadata into compacted historical archives to maintain scalability.

Concurrency Handling

Hudi provides optimistic concurrency control for distributed writers.

Writers:

1. stage tentative changes
2. validate conflicts
3. atomically publish commits to the timeline

The timeline acts as the central coordination mechanism ensuring:

• snapshot isolation
• rollback safety
• multi-writer consistency

This allows Hudi to support:

• streaming ingestion
• concurrent pipelines
• asynchronous table services

on top of fundamentally immutable object storage.

Schema Evolution

Since Hudi tables are mutable and evolve continuously, different file slices in the same table may have been written with different schemas over time.

Example:

T1 → {id, name}
T2 → {id, name, age}
T3 → {id, age}

Hudi handles schema evolution through a combination of:

• Avro schema resolution semantics
• timeline-based schema tracking
• reader-side schema reconciliation

Schemas are stored in:

• commit metadata on the timeline
• Parquet/Avro file metadata
• optionally synced to Hive Metastore / Glue Catalog

At query time, Hudi reconciles:

• writer schema
• reader schema
• file slice schema

to construct a consistent snapshot view.

This allows multiple schema versions to coexist within the same table while supporting operations like:

• adding columns
• deleting columns
• column reordering
• compatible type promotion

Conceptually:

writer schema
     +
reader schema
     ↓
resolved logical schema

For MOR tables, schema handling is more complex since readers may need to reconcile:

• base Parquet schemas
• delta log schemas
• query schema

during merge operations.

Compared to traditional databases where schema changes mutate a centralised catalog, Hudi treats schema evolution as a timeline-aware table concern across immutable distributed files.

Apache Iceberg

As discussed in the previous blog, Apache Iceberg was designed with a different worldview from Hudi. Rather than optimising primarily for write throughput and mutable datasets, Iceberg focused on solving metadata scalability, query planning performance and correctness for extremely large analytical datasets stored on cloud object stores.

Traditional Hive tables relied on directory structures as the primary source of metadata. As tables grew to millions of files, operations such as partition discovery, file listing and query planning became increasingly expensive and error-prone.

Iceberg’s key insight was that metadata should be treated as a first-class concern. Instead of discovering table state from the filesystem, Iceberg explicitly tracks every data file and delete file in table metadata.

This enables:

• scalable query planning for petabyte-scale tables
• engine interoperability
• atomic commits on object stores
• schema and partition evolution
• time travel and snapshot isolation
• row-level mutations

Metadata-driven query planning sits at the heart of Iceberg’s architecture.

To support this, Iceberg introduced several architectural innovations:

• Hidden Partitioning
• Snapshot-based table state management
• A hierarchical metadata tree
• Stable Field IDs for schema evolution
• Optimistic concurrency control
• File-level metadata and statistics

Unlike Hudi’s timeline-centric architecture, Iceberg organises the table around an immutable metadata tree where each write produces a new snapshot while reusing as much existing metadata as possible.

Key Components

Catalog

The Catalog is the entry point into an Iceberg table.

Rather than discovering tables from directory structures, query engines first consult a catalog such as:

• Hive Metastore
• AWS Glue
• REST Catalog
• Nessie
• JDBC Catalog

The catalog maintains a pointer to the latest table metadata file.

Conceptually:

Catalog
    ↓
Current Metadata File

This indirection enables atomic table updates by simply swapping the metadata pointer to a new version.

Metadata Layer

Iceberg organises metadata as a hierarchical tree. Unlike Hudi’s timeline-centric architecture or Delta’s transaction-log-centric architecture, Iceberg fundamentally treats the table as a hierarchy of immutable metadata structures. The metadata tree itself becomes the authoritative representation of table state.

Catalog
   ↓
Metadata File
   ↓
Snapshot
   ↓
Manifest List
   ↓
Manifest Files
   ↓
Data Files / Delete Files

This hierarchy is one of Iceberg’s most important innovations because it allows metadata to scale independently from data volume.

Table Metadata File

The metadata file acts as the root of the metadata tree.

It contains:

• current snapshot pointer
• schema definitions
• partition specifications
• sort orders
• snapshot history
• table properties

Every commit creates a new metadata file.

However, Iceberg avoids rewriting the entire metadata tree by reusing existing manifests and snapshots whenever possible.

This makes snapshot creation extremely lightweight.

Snapshots

A snapshot represents the complete state of a table at a point in time.

Every:

• append
• delete
• merge
• rewrite
• compaction

produces a new snapshot.

Snapshots provide:

• atomic commits
• time travel
• rollback
• snapshot isolation

Unlike traditional databases, snapshots are immutable.

New snapshots reference existing metadata wherever possible rather than duplicating it.

This makes Iceberg’s snapshot model conceptually similar to Git commits.

Manifest Lists

A snapshot points to a Manifest List.

A Manifest List is essentially an index of manifest files belonging to that snapshot.

It stores summary information about:

• partitions
• file counts
• added files
• deleted files

This allows Iceberg to eliminate large portions of metadata during planning before opening individual manifests.

Manifest Files

Manifest files contain file-level metadata.

Each entry represents a data file or delete file and includes:

• file path
• partition values
• record counts
• column statistics
• file size
• snapshot information

Manifest files are the workhorse of Iceberg’s query planning system.

Rather than listing object storage directories, query engines consult manifest metadata to determine which files need to be scanned.

Data Storage Layer

Data Files

Actual data is stored in open formats such as:

• Parquet
• ORC
• Avro

Iceberg itself does not define a storage format.

Instead it manages metadata describing these files.

Each data file belongs to exactly one partition tuple and carries rich metadata used during planning.

Delete Files

To support row-level mutations without rewriting large data files, Iceberg introduces delete files.

Two primary types exist:

Position Deletes

• identify specific rows by file path and row position

Equality Deletes

• identify rows based on column values

Delete files allow Iceberg to implement updates and deletes without immediately rewriting underlying data files.

Periodic maintenance jobs later compact these deletes back into data files.

Hidden Partitioning

Hidden partitioning is one of Iceberg’s most important usability improvements.

Traditional Hive tables expose physical partitioning directly to users.

For example:

WHEREmonth(event_time)='2026-05'

Iceberg decouples logical queries from physical partition layouts.

Users write:

WHERE event_timeBETWEEN ...

while Iceberg automatically evaluates partition transforms during query planning.

For example:

event_time
    ↓
month(event_time)

The resulting partition predicate is used to prune candidate files.

This enables:

• simpler queries
• partition evolution
• engine portability
• better query optimisation

without exposing storage layout details to users.

Metadata-Driven Query Planning

Iceberg’s defining architectural feature is metadata-driven query planning.

When a query arrives:

WHERE event_time> ...
AND price>1000

Iceberg attempts to eliminate files before scanning them.

It uses:

1. Partition pruning
2. Column statistics pruning
3. Delete file pruning

Manifest metadata stores statistics such as:

• min value
• max value
• null count
• value count

allowing entire files to be skipped.

Conceptually:

Query Predicate
      ↓
Partition Pruning
      ↓
Column Stats Pruning
      ↓
Files To Scan

This aggressive file elimination is the primary reason Iceberg scales efficiently to petabyte-sized datasets.

Handling Concurrency

Iceberg uses optimistic concurrency control.

Multiple writers can:

• generate new snapshots independently
• validate against the current table state
• atomically publish commits

A commit succeeds only if the table metadata pointer has not changed unexpectedly.

This provides:

• serializable isolation
• atomic commits
• multi-writer support

without requiring distributed locking systems.

Schema Evolution

Iceberg’s schema evolution is built around stable Field IDs.

Traditional systems identify columns by name:

customer_name

Iceberg assigns an immutable identifier:

field_id = 17

and tracks schema evolution using these IDs.

This enables:

• column renames
• column reordering
• column additions
• column deletions

without breaking query correctness.

Field IDs are widely considered one of Iceberg’s most elegant architectural decisions because they decouple schema semantics from physical column names.

Partition Evolution

Partition specifications are stored as metadata rather than being encoded into directory structures.

As a result, Iceberg can evolve partitioning strategies over time.

Example:

month(event_time)
        ↓
day(event_time)

Older files continue using the original partition specification while new files use the updated specification.

No data rewrite is required for correctness. Query planning dynamically evaluates predicates using the appropriate partition specification for each file.

This is a major departure from Hive-style partition management.

Asynchronous Background Maintenance

Over time, streaming ingestion and row-level mutations create:

• small files
• fragmented layouts
• delete file accumulation
• metadata growth

Iceberg relies on maintenance operations to continuously optimise tables.

A key distinction is that Iceberg itself is not an execution engine. The scheduling and execution of maintenance work is delegated to engines such as Spark, Flink, Dremio or Trino.

Compaction

Compaction merges many small files into larger files.

Benefits include:

• fewer file opens
• improved scan performance
• reduced metadata overhead

Data Rewriting

Data files can be rewritten to:

• improve clustering
• improve sort order
• align with new partition strategies

Unlike partition evolution, these rewrites are performed purely for performance reasons.

Delete File Compaction

Delete files accumulate over time as updates and deletes occur.

Periodic maintenance merges delete information back into data files to reduce read amplification.

Snapshot Expiration & Cleanup

Every write creates a new snapshot.

To prevent metadata growth, old snapshots are periodically expired.

This removes:

• obsolete snapshots
• orphan manifests
• unused metadata files
• unreferenced data files

Without these maintenance operations, metadata growth can eventually impact planning performance and storage costs.

ℹ️ https://www.dremio.com/resources/guides/apache-iceberg-an-architectural-look-under-the-covers/#h-the-iceberg-table-format provides a great walk-through of how the different CRUD operations are implemented and how the data and metadata files layout evolves with them.

Delta Lake

Delta’s worldview was shaped by a different problem than either Hudi or Iceberg. Rather than optimizing primarily for write throughput or metadata scalability, Delta focused on bringing database-style ACID guarantees to enterprise Spark workloads running on cloud object stores.

Prior to Delta, data lakes built on Parquet files suffered from several challenges:

• readers observing partially completed writes
• lack of transactional consistency
• difficult implementation of updates and deletes
• separate processing models for batch and streaming workloads

Delta popularised and operationalised the idea of treating a transactional lakehouse table as the common abstraction for both batch and streaming workloads. By making the transaction log the source of truth, batch jobs, streaming pipelines and interactive queries all operate against the same consistent table state using identical semantics.

At its core, a Delta table is simply:

Parquet Files
      +
Transaction Log
      =
Delta Table

Unlike Iceberg, which materialises table state through a metadata hierarchy, or Hudi, which reconstructs state through the timeline, Delta reconstructs the current state of the table by replaying transactional actions stored in the log.

Delta Transaction Log Protocol

The Delta Lake Transaction Log Protocol is the central specification that defines the behavior of all Delta implementations.

The protocol defines:

• transaction semantics
• commit formats
• concurrency guarantees
• schema management
• table features
• compatibility requirements

The protocol serves as the interoperability layer between different Delta implementations including:

• Delta Spark
• Delta Kernel
• Delta-RS
• Microsoft Fabric
• Vendor-specific implementations

All implementations must correctly interpret and produce protocol-compliant transaction logs.

Protocol Versions vs Table Features

Historically Delta used cumulative protocol versions where enabling a feature required upgrading the minimum reader and writer protocol versions.

This model proved too rigid as the ecosystem expanded.

Modern Delta introduces Table Features, allowing implementations to selectively advertise support for specific capabilities such as:

• Column Mapping
• Change Data Feed
• Deletion Vectors
• Liquid Clustering

This enables feature-level compatibility rather than requiring all implementations to move in lockstep.

File Layout

A Delta table consists of a collection of Parquet files alongside a dedicated transaction log directory.

mytable/
├── _delta_log/
├── data files
├── deletion vectors
└── change data files

The transaction log acts as the authoritative source of truth while the Parquet files store the actual data.

Data Files

Delta stores table data exclusively in Parquet format.

Each transaction adds or removes entire Parquet files from the table state.

Unlike Hudi, Delta does not maintain separate log files containing updated records.

Historically all updates and deletes were implemented through file rewrites, making Delta fundamentally a Copy-on-Write system.

Deletion Vector Files

Modern Delta introduces Deletion Vectors (DV) to avoid rewriting entire Parquet files for row-level modifications.

A deletion vector stores metadata identifying rows that should be considered logically deleted.

Conceptually:

Parquet File
      +
Deletion Vector

Readers apply deletion vectors during query execution to determine visible rows.

This reduces rewrite amplification for updates and deletes while preserving snapshot isolation semantics.

Change Data Files

Delta can optionally materialise row-level changes through the Change Data Feed (CDF).

These files capture:

• inserts
• updates
• deletes

between table versions and allow downstream consumers to incrementally process changes without repeatedly scanning the entire table.

Delta Log Entry

The transaction log consists of a sequence of immutable commit files:

00000000000000000042.json
00000000000000000043.json
00000000000000000044.json

Each commit contains one or more actions describing changes to the table state.

Common actions include:

• AddFile
• RemoveFile
• Metadata
• Protocol
• Transaction

A key design decision is that Delta records changes to files rather than changes to individual rows.

For example, an UPDATE operation is represented as:

RemoveFile(old_file)
+
AddFile(new_file)

rather than an explicit row-level update record.

This keeps the transaction model simple and aligns naturally with immutable object storage.

A notable design decision is that Delta stores transactional actions as append-only JSON commit files rather than binary metadata structures. This keeps commits simple, human-readable and easy to implement across engines, while periodic Parquet checkpoints (covered below) materialise table state to avoid replaying long commit histories.

Delta’s use of JSON is primarily an architectural choice rather than a performance optimisation. The transaction log behaves like a distributed WAL where commits are small, atomic, self-contained files. While binary formats such as Parquet or Avro could improve metadata parsing efficiency, they would add complexity to commit generation and protocol evolution. Delta therefore uses a hybrid approach: JSON for lightweight transactional commits and Parquet checkpoints for efficient snapshot reconstruction at scale.

Checkpoints

Replaying every JSON commit since table creation would eventually become expensive.

To avoid this, Delta periodically writes Parquet checkpoints containing the fully materialised table state.

Checkpoint
      +
Recent JSON Commits
      ↓
Current Snapshot

Readers reconstruct the current table version by:

1. Loading the latest checkpoint
2. Applying subsequent commits

This is conceptually similar to how databases periodically checkpoint their write-ahead logs.

Sidecar Files

Large checkpoints may be split across multiple sidecar files to improve scalability and reduce metadata overhead.

Log Compaction

Delta may also compact historical transaction logs into larger aggregated log files to reduce replay costs.

Last Checkpoint File

The _last_checkpoint file provides a fast pointer to the latest checkpoint, avoiding expensive directory scans during snapshot reconstruction.

Metadata-Driven Query Planning

Although Delta does not use Iceberg’s manifest hierarchy, it still relies heavily on metadata-driven pruning.

Every AddFile action records:

• partition values
• record counts
• file size
• column statistics

including:

• min values
• max values
• null counts

When a query arrives:

WHERE event_date='2026-01-01'
AND price>1000

Delta performs:

1. Partition pruning
2. Statistics-based data skipping
3. Deletion vector filtering

before scanning any Parquet files.

This metadata is periodically materialised into checkpoints, allowing Delta to perform efficient file pruning without traversing object storage listings.

Handling Concurrency

Delta uses Multi-Version Concurrency Control (MVCC) combined with optimistic concurrency control.

Serialisable ACID Writes

Multiple writers may concurrently modify the same table.

Each writer:

1. Reads a snapshot
2. Generates new files
3. Attempts to commit a new transaction log entry
4. Validates that conflicting changes have not occurred

Only one commit succeeds for a given table version.

Conflicting writers must retry against a newer snapshot.

Snapshot Isolation for Reads

Readers always operate against a stable table version.

Even while new commits are occurring, queries continue reading a consistent snapshot without observing partially committed changes.

This provides:

• atomicity
• consistency
• isolation
• durability

on top of eventually consistent object storage systems.

Schema Evolution

Delta stores schema definitions as metadata actions within the transaction log.

Schema changes therefore become part of the transactional history of the table.

Supported operations include:

• column additions
• column deletions
• nested schema evolution
• compatible type widening
• column renaming

Column Mapping

One of Delta’s most significant schema evolution enhancements is Column Mapping.

Historically columns were identified by name, making renames difficult.

Column Mapping introduces stable column identifiers:

Column ID
      ↓
Column Name

allowing columns to be renamed or reordered without changing their logical identity.

This brings Delta’s schema evolution model closer to Iceberg’s field-ID based approach while preserving compatibility with existing tables.

Asynchronous Background Maintenance

Like Hudi and Iceberg, Delta relies on maintenance operations to continuously optimise table layout and metadata.

An important distinction is that Delta itself is not an execution engine. Scheduling and execution of these operations is performed by systems integrating with Delta such as Spark, Databricks Runtime or other compute engines.

OPTIMIZE

Compacts small files into larger Parquet files.

Benefits include:

• reduced file-open overhead
• improved scan efficiency
• lower metadata overhead

VACUUM

Removes:

• obsolete data files
• expired transaction logs
• orphaned metadata

that are no longer required for time travel or snapshot reconstruction.

Z-Ordering

Z-Ordering is a multi-dimensional clustering technique used to improve statistics-based pruning.

Rather than sorting data on a single column, Z-Ordering organises files such that values that are frequently queried together are colocated within the same files.

This improves:

• min/max statistics
• data skipping
• selective query performance

without requiring additional indexes.

Liquid Clustering

Liquid Clustering is Delta’s modern replacement for many traditional partitioning and Z-Ordering workflows.

Rather than relying on rigid partition boundaries, Liquid Clustering continuously organises data around specified clustering columns while allowing file boundaries to evolve over time.

This provides:

• more flexible data layout
• reduced repartitioning requirements
• improved pruning performance
• simplified operational management

for large analytical datasets.

The Takeaway Menu

Summarising the key aspects of Lakehouse Architecture and comparing how the triad of open table formats deal with it:

How CRUD Operations Are Implemented

CREATE / INSERT

Hudi

• Records are written into file groups.
• COW tables write directly to base files.
• MOR tables may append records to log files before compaction.

Iceberg

• New data files are created.
• A new snapshot is committed referencing those files.
• Existing files are never modified.

Delta Lake

• New Parquet files are written.
• Corresponding AddFile actions are recorded in the transaction log.
• A new table version is created through an atomic commit.

READ

Hudi

• Readers reconstruct the table state using the Timeline.
• MOR readers may need to merge base files and log files at query time.

Iceberg

• Readers traverse the metadata tree: Metadata File → Snapshot → Manifest List → Manifest Files → Data Files.
• Query planning aggressively prunes files before scanning.

Delta Lake

• Readers reconstruct the latest snapshot using checkpoints and recent transaction log entries.
• File pruning relies on partition metadata and file-level statistics.

UPDATE

Hudi

• Records are located using indexes.
• COW rewrites affected files.
• MOR appends updated records to log files and reconciles them later through compaction.

Iceberg

• Updates are implemented as a delete + insert operation.
• Typically represented through delete files and new data files.
• Maintenance jobs eventually reconcile and compact these changes.

Delta Lake

• Historically implemented as RemoveFile + AddFile.
• Affected files are rewritten and replaced.
• Modern Delta can use Deletion Vectors to avoid some rewrites.

DELETE

Hudi

• COW rewrites affected files.
• MOR records delete information in log files until compaction occurs.

Iceberg

• Uses Equality Deletes or Position Deletes.
• Delete files are applied during query execution.
• Background maintenance can rewrite data files to eliminate delete files.

Delta Lake

• Historically rewrote affected files.
• Modern Delta often records deletes using Deletion Vectors.
• Readers apply deletion vectors during query execution.

Architectural Contrasts

State Management

Hudi: Timeline-based. Table state is reconstructed from commits, compactions, clustering operations and file slices tracked on the Timeline.

Iceberg: Metadata-tree based. Table state is materialised through a hierarchy of Metadata Files → Snapshots → Manifest Lists → Manifest Files.

Delta Lake: Transaction-log based. Table state is reconstructed by replaying actions in the Delta Log and accelerated through periodic checkpoints.

Metadata Philosophy

Hudi: Metadata primarily exists to support efficient writes, record location, indexing and incremental processing. The Metadata Table accelerates lookups and file discovery.

Iceberg: Metadata is a first-class optimisation layer. Rich file-level metadata, partition metadata and column statistics are used to aggressively prune files before query execution.

Delta Lake: Metadata is primarily transactional. The log records file additions, removals, schema changes and statistics while maintaining ACID guarantees and snapshot isolation.

Partitioning Philosophy

Hudi: Traditional Hive-style partitioning augmented with clustering and indexing.

Iceberg: Hidden Partitioning. Users query logical columns while Iceberg automatically derives partition predicates. Supports partition evolution without rewriting historical data.

Delta Lake: Traditional partitioning with metadata-based pruning. Increasingly moving toward Liquid Clustering to reduce dependence on rigid partition boundaries.

Mutation Strategy

Hudi: Mutations are a first-class concern. COW rewrites files while MOR appends updates to log files and reconciles them later through compaction.

Iceberg: Mutations are treated as metadata operations. Updates and deletes are represented through data files, delete files and rewrite operations.

Delta Lake: Historically Copy-on-Write through AddFile/RemoveFile actions. Modern Delta uses Deletion Vectors to reduce rewrite amplification for updates and deletes.

Schema Evolution

Hudi: Avro-based schema reconciliation. Multiple schema versions can coexist and are resolved through reader/writer schema compatibility.

Iceberg: Stable Field IDs are the source of truth. Renames, reordering and evolution are metadata operations independent of column names.

Delta Lake: Schema is stored in transaction log metadata. Modern Column Mapping introduces stable identifiers that support safe renames and reordering.

Conclusion

In many ways, the evolution of open table formats mirrors the evolution of databases themselves. Hudi borrows ideas from LSM-tree storage engines, Iceberg borrows ideas from immutable metadata structures and snapshot-based planning, while Delta borrows ideas from transaction logs and write-ahead logging systems.

All three ultimately solve the same problem — bringing database-grade guarantees to cloud object storage — but they choose to place complexity in very different layers of the system.

• Hudi places complexity in the write path.
• Iceberg places complexity in metadata management.
• Delta places complexity in the transaction log.

That single design choice explains most of the tradeoffs, strengths and operational characteristics that practitioners observe in production.

An interesting trend over the last few years is that all three formats have gradually borrowed ideas from one another. Iceberg added row-level delete files, Delta introduced Deletion Vectors and Liquid Clustering, while Hudi expanded its metadata and indexing capabilities. Despite this convergence, their original architectural priorities remain clearly visible in where each chooses to place complexity.

References

• https://hudi.apache.org/docs/overview#core-concepts-to-learn
• https://hudi.apache.org/docs/hudi_stack
• https://hudi.apache.org/docs/table_types
• https://www.uber.com/in/en/blog/hoodie/
• https://iceberg.apache.org/docs/latest/?utm_source=chatgpt.com
• https://iceberg.apache.org/spec/#overview
• https://www.dremio.com/resources/guides/apache-iceberg-an-architectural-look-under-the-covers/
• https://www.databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html
• https://github.com/delta-io/delta/blob/master/PROTOCOL.md

Disclaimer: While the ideas and core structure of this blog were conceptualised by the author, AI tools were used to assist in drafting and refining the content and as a sparring assistant.