Building a Scalable, Modernized Data Lakehouse: Delta Lake vs. Apache Iceberg

A practical guide for platform teams evaluating, operating, and evolving an open data architecture on AWS, Azure, or GCP — without migrating your storage.

The Decision That Actually Matters

If you’re a platform team today, you’re probably not starting from a blank slate. You already have data sitting in S3, Azure Data Lake Storage (ADLS Gen2), or GCS. You have pipelines landing both streaming events and nightly batch jobs. You have multiple teams using different compute engines — maybe Spark for data science, SQL for analytics, Flink for real-time — and they all expect consistent, up-to-date data.

The question isn’t “which cloud should we use?” You’ve already answered that. The question is:

How do we put a format, a compute engine, and a catalog on top of our existing storage that gives us ACID guarantees, multi-engine access, streaming + batch unification, and operational sanity — without locking our data to a single vendor?

That question has two serious answers right now: Delta Lake (the engine of Databricks’ Lakehouse) and Apache Iceberg (the open standard increasingly backed by Snowflake, AWS, Google, and the broader open-source ecosystem).

This article doesn’t declare a winner. It gives platform teams the architectural facts to evaluate, operate, and build on either path — or both.

Part 1 — Evaluate: What Are We Actually Choosing Between?

The Problem Open Table Formats Solve

Before comparing Delta and Iceberg, it’s worth being precise about what both are solving.

Raw Parquet files on object storage are fast and cheap, but they have no concept of transactions, schema enforcement, or row-level updates. Two Spark jobs writing to the same partition simultaneously will silently corrupt each other’s output. A GDPR deletion request means manually reading, filtering, and rewriting entire files. A schema change in an upstream source might break downstream readers without warning. And streaming micro-batches landing every five minutes alongside nightly batch jobs creates a coordination nightmare with no native solution.

Open table formats fix this by adding a thin metadata layer on top of your existing Parquet files. Your data stays exactly where it is — in your S3 bucket, your ADLS container, your GCS prefix. The format adds a transaction log and file manifest that gives your collection of files the properties of a proper database table: atomic writes, snapshot isolation, schema enforcement, time travel, and row-level deletes.

Query Engine (Spark / Trino / Flink / Snowflake / Athena / DuckDB)
                              │
                              ▼
              Table Format Metadata Layer
        (transaction log + file manifests + statistics)
                              │
                              ▼
         Your Object Storage (S3 / ADLS / GCS) — unchanged
                              │
                              ▼
              Physical Data Files (Parquet / ORC / Avro)

The catalog (Unity Catalog, Snowflake Horizon, AWS Glue, Apache Polaris) sits above this and maps table names to metadata locations. But the format itself doesn’t require any particular catalog — which is where the portability story begins.

Delta Lake flat transaction log vs Apache Iceberg hierarchical manifest tree

Delta Lake: The Databricks Lakehouse Format

Delta Lake was open-sourced by Databricks in 2019 and has since become a Linux Foundation project. Its defining architectural choice is a flat, append-only transaction log stored as a sequence of JSON files in a _delta_log/ directory alongside your data.

Every write operation appends a new JSON file to that log — 000001.json, 000002.json, and so on. Each JSON file explicitly records: which Parquet files were added, which were removed, and the column-level min/max statistics and null counts for every file added. This flat structure means any engine reading the log gets a complete, ordered record of every transaction without traversing a tree of metadata files.

Automatic Checkpointing keeps this performant over time. Every 10 commits, Delta compacts the JSON log into a single .checkpoint.parquet file. Because the checkpoint is itself a Parquet file, query engines can use standard columnar execution to scan it — making metadata reads (file pruning, partition filtering) extremely fast without parsing dozens of JSON files sequentially.

Key strengths of Delta Lake:

Liquid Clustering replaces traditional Hive-style partitioning entirely. Instead of declaring static partition columns upfront (and being stuck with them), you specify CLUSTER BY (user_id, date) and Delta automatically organizes data on the fly based on write patterns and data skew. You can change clustering keys with a simple ALTER TABLE — no data rewrite required.

Deletion Vectors handle row-level deletes without immediately rewriting entire Parquet files. When a row is deleted or updated, Delta writes a small bitmap file marking those rows as dead. Reads skip them; a background compaction job cleans them up later. This makes high-frequency MERGE/UPSERT operations — typical in CDC pipelines — significantly faster.

Change Data Feed (CDF) automatically captures row-level changes (inserts, updates, deletes) alongside before/after images of each row. Downstream streaming pipelines can consume these changes incrementally without full table scans, making Delta tables an efficient source for event-driven architectures.

Multi-table atomic transactions are native in Delta because all tables share the same log model. You can commit changes across multiple Delta tables simultaneously — all succeed or all roll back.

The Databricks + Photon advantage: When running Delta on Databricks, the Photon execution engine — a native C++ vectorized query engine — replaces the standard JVM-based Spark executor. This eliminates JVM garbage collection overhead, adds SIMD hardware-level parallelism, and includes automated local SSD caching. The performance profile is meaningfully different from vanilla Spark on standard VMs, particularly for large analytical scans and MERGE operations.

Apache Iceberg: The Open Interoperability Standard

Apache Iceberg originated at Netflix in 2017 and became an Apache top-level project in 2020. Its defining architectural choice is a hierarchical metadata tree: every table has a current metadata.json file that points to a Manifest List, which points to a set of Manifest Files (stored as .avro), which each list a subset of the actual Parquet data files.

metadata.json  (current snapshot pointer)
        │
        ▼
  Manifest List  (one entry per manifest file)
        │
        ├── Manifest File A  → [data_file_1.parquet, data_file_2.parquet]
        └── Manifest File B  → [data_file_3.parquet, data_file_4.parquet]

This hierarchy enables a feature Delta doesn’t natively offer: Hidden Partitioning. In traditional Hive-style partitioning (used by Delta), query writers must explicitly filter on partition columns or risk full table scans. In Iceberg, partition transforms are declared at the table level — by day, by bucket, by truncated value — and the engine handles routing automatically. Query writers don’t need to know partition columns. Better still, partition strategies can evolve without rewriting data — old data remains queryable under the new scheme through Partition Evolution.

Key strengths of Iceberg:

Engine-agnostic by design. The Iceberg spec is meticulously defined and has been independently implemented by Spark, Trino, Flink, Athena, BigQuery, Snowflake, DuckDB, and more. A table written by Flink can be read by Athena and updated by Spark without any data conversion. The Apache Iceberg REST catalog API has become the lingua franca of open table interoperability.

Schema evolution safety. Iceberg tracks columns by ID, not name, making column renames, reordering, and type promotions safe across all readers without breaking existing queries.

Snapshot isolation as first-class design. Every read sees a consistent point-in-time snapshot. Time travel is built into the spec, not bolted on.

Vendor-neutral momentum. Snowflake, AWS (S3 Tables, Athena, Glue), Google (BigLake, BigQuery), and the Apache Polaris open-source catalog all speak Iceberg natively. When multiple competing cloud vendors implement the same format, that’s not coincidence — it’s convergence on a standard.

The Snowflake compute advantage on Iceberg: Snowflake’s Virtual Warehouses are not a generic SQL layer on top of object storage. They are a custom-built C++ execution engine with SIMD vectorized processing (similar in architecture to Databricks’ Photon), automated local SSD/NVMe caching that eliminates repeated S3 network fetches, and an Adaptive Scan capability that dynamically adjusts I/O parallelism based on real-time memory and network metrics. For interactive SQL and BI dashboard workloads on Iceberg tables, this delivers performance far beyond vanilla Spark on standard VMs — with zero infrastructure management.

The Honest Comparison: Where Each Has the Edge

Delta Lake (Databricks) vs Apache Iceberg (Snowflake/Open)

The UniForm bridge: Databricks recognized that many teams want Delta’s write performance but Iceberg’s ecosystem reach. Universal Format (UniForm), when enabled on a Delta table, automatically generates Iceberg metadata files alongside the Delta log as part of every write. Snowflake, Trino, Athena, and other Iceberg-native engines can read that table as if it were native Iceberg — with no data copying, no format migration. This is an important practical option for teams that are Delta-primary but want to avoid compute lock-in for reads.

Part 2 — Operate: Life in a Lakehouse

Choosing a format is one decision. Operating it at scale is an ongoing one. Both Delta and Iceberg accumulate small files, stale metadata, and historical snapshots over time — especially in streaming workloads. Left unmanaged, this degrades query performance and inflates storage costs.

Both formats solve the same three operational problems. They just call them different things.

Delta Lake vs Iceberg maintenance operations: checkpointing, compaction, and snapshot cleanup compared

Problem 1: Metadata Bloat (“Squashing the Logs”)

Delta Lake — Automatic Checkpointing

Every 10 commits, Delta automatically compacts its JSON transaction log into a single .checkpoint.parquet file. A query engine reading the table loads the latest checkpoint (fast columnar scan) and replays only the handful of JSON files written since. Without checkpointing, reading a table with 100,000 commits would require parsing 100,000 JSON files sequentially. With it, that's 1 checkpoint + a few recent JSONs — milliseconds instead of minutes. This happens automatically with no configuration required.

Iceberg — Metadata Optimization (rewrite_manifests)

Iceberg’s equivalent problem is manifest fragmentation. Streaming jobs writing every few seconds create thousands of tiny .avro manifest files. During query planning, the engine must open and scan every manifest to determine which data files are relevant — causing slow planning even when the actual data scan is fast. rewrite_manifests compacts small manifests into larger ones (typically 8–16 MB) and groups data files by partition, so the engine can skip entire manifests during pruning. Unlike Delta's automatic checkpointing, this requires explicit orchestration — though table properties like commit.manifest.min-count-to-merge can trigger inline compaction automatically during writes.

Problem 2: Small File Accumulation (“Squashing Tiny Streaming Files”)

This is the most common operational headache in streaming lakehouses. A Kafka consumer writing micro-batches every 30 seconds lands hundreds of 1–10 MB Parquet files per hour. Query engines read far more efficiently from a few 256 MB–1 GB files than from thousands of tiny ones.

Delta Lake — OPTIMIZE command

OPTIMIZE scans active data files, merges small files up to a target size, and writes consolidated Parquet files. Delta marks the old small files as removed in the transaction log but doesn't immediately delete them (they're needed for time travel). This can be run on a schedule or triggered manually.

Iceberg — rewrite_data_files procedure

Iceberg’s equivalent, rewrite_data_files, does the same thing: reads small files, bins them together, writes consolidated files, and records the change as a new snapshot. Like Delta's OPTIMIZE, old files persist until explicitly expired. This must be orchestrated externally via Spark, Glue, or Airflow unless Snowflake is configured as the catalog (more on that below).

Problem 3: Historical File Accumulation (“Cleaning Up Old Snapshots”)

Both formats preserve historical data files for time travel. Without cleanup, your storage bucket grows indefinitely.

Delta Lake — Log Retention + VACUUM

Delta separates this into two controls. delta.logRetentionDuration (default 30 days) governs how long JSON transaction log files are kept before automatic cleanup. VACUUM physically deletes the underlying Parquet data files that are no longer referenced by any active version — files that were overwritten or deleted during MERGE/UPDATE operations. Running VACUUM with a retention threshold (e.g., 7 days) keeps your storage clean without removing files still needed for time travel.

Iceberg — expire_snapshots

Iceberg tracks every commit as a named snapshot. expire_snapshots removes snapshots older than a specified threshold and physically deletes the orphaned data and metadata files they reference. Unlike Delta's two-step model, Iceberg combines log cleanup and data file cleanup into one operation. Retention duration is configurable per table, giving platform teams precise control per dataset.

The Maintenance Sequence (Iceberg, manually orchestrated)

If you’re managing Iceberg outside of a fully managed platform, the right order matters. Running these out of sequence can temporarily double storage costs or leave orphaned files:

1. rewrite_data_files    → compact small data files first
2. rewrite_manifests     → optimize the metadata tree generated by step 1
3. expire_snapshots      → mark old snapshots and files as expired
4. remove_orphan_files   → physically purge failed-write debris from storage

Summary: What Each Format Handles Automatically vs. What You Orchestrate

Who handles Iceberg table maintenance: Databricks, Snowflake-as-catalog, and external catalog configurations

Iceberg Maintenance Operations

Part 3 — Guidance: Building the Right Architecture for Your Environment

The Core Principle: Storage Is Not the Variable

Platform teams often frame this as a migration question. It isn’t. Whether your data lives in S3, ADLS Gen2, or GCS doesn’t change. The variables are:

• The format layer on top of your Parquet files (Delta vs. Iceberg metadata)
• The compute engine reading and writing that format (Photon / Databricks vs. Snowflake Virtual Warehouses vs. open engines like Trino/Athena)
• Who owns the catalog — and therefore who performs maintenance, enforces governance, and controls who can read the table

Choosing well on these three dimensions is the architecture decision. Storage is infrastructure you already have.

Architectural Paths by Starting Point

If your primary compute is Databricks (Azure, AWS, or GCP):

Delta Lake is the natural format choice. You get Photon performance, Liquid Clustering, Change Data Feed, and native Unity Catalog governance out of the box. For any workload where external engines (Snowflake, Athena, Trino) need to read the same tables, enable UniForm — you get Delta’s write performance with Iceberg read compatibility, without duplicating data or running format migrations.

Unity Catalog governs the full stack: tables, ML models, volumes (unstructured files), and fine-grained access control in one place. If your organization also runs AI/ML workloads alongside analytics, the ability to govern model artifacts and training datasets under the same catalog as your production tables is a genuine differentiator.

If your primary SQL layer is Snowflake (on any cloud):

Iceberg is the format that gives you the most flexibility. For your core BI and reporting workloads requiring maximum query performance, strict SLAs, and full warehouse features (materialized views, search optimization), Snowflake-managed internal tables remain the right choice. For large-scale historical archives, ML training data, and any dataset that multiple engines need to access directly from S3/GCS/ADLS, Snowflake-managed Iceberg External Tables give you Snowflake’s compute and automated maintenance on your own storage bucket — with no vendor lock-in on the data files themselves.

The catalog ownership decision matters here: if Snowflake is configured as the catalog (even for data in your own S3 bucket), it handles all background maintenance automatically. If you point Snowflake at an externally managed catalog (Glue, Polaris, Hive), Snowflake becomes a read-only consumer and your team owns the maintenance pipeline.

If you’re multi-cloud or deliberately avoiding primary platform lock-in:

Apache Iceberg with an open catalog (Apache Polaris, Project Nessie, or AWS Glue) is the most defensible long-term architecture. Your data sits in object storage you own. Your format metadata is readable by every major engine. Your catalog is open-source or cloud-native. Compute becomes a pluggable choice — run Snowflake warehouses for SQL, Spark for large transformations, Athena for ad-hoc queries, DuckDB for local development — all against the same tables without data movement or egress costs.

The operational trade-off: you own the maintenance pipeline. rewrite_data_files, rewrite_manifests, expire_snapshots need to be orchestrated — typically via Airflow, AWS Glue, or Spark jobs on a schedule. This is manageable, but it's real work that a Databricks- or Snowflake-managed catalog abstracts away.

How to Think About Compute Engine Choice

The format choice and the compute choice are related but independent. You can run Snowflake compute against Delta tables (via UniForm/Iceberg compatibility). You can run Spark against Iceberg tables. The question is which engine fits which workload.

Compute Engine Choice

The pattern that works in practice for most platform teams is a hybrid by workload type: Snowflake (or Databricks) as the primary compute for SQL and analytics, open engines for transformation and ML, and Iceberg (potentially with UniForm for Delta-primary shops) as the common format underneath. Data doesn’t move. Engines plug in and out as needs change.

What “No Vendor Lock-in” Actually Means

It’s worth being specific, because the phrase is overused.

Lock-in at the storage layer — your bytes being trapped in a proprietary format that only one vendor can read — is what open table formats eliminate. Your Parquet files are your Parquet files. Iceberg metadata is an open spec. Delta metadata is open-source. Neither format holds your data hostage.

Lock-in at the compute layer — being unable to switch query engines without re-ingesting data — is what the Iceberg REST catalog spec and UniForm address. If your tables are Iceberg (or Delta+UniForm), any compliant engine can read them. Switching from Snowflake to Athena for a particular workload is a configuration change, not a migration.

Lock-in at the catalog and governance layer — your metadata, access policies, and lineage being trapped in a vendor’s proprietary catalog — is the frontier where the real architectural decisions are still being made. Apache Polaris (open-source Iceberg REST catalog), Project Nessie (catalog with Git-like branching), and Unity Catalog OSS (Linux Foundation) are the tools that keep this layer open. Investing in this layer as open infrastructure is the highest-leverage move a platform team can make for long-term flexibility.

The Bottom Line

Delta Lake and Apache Iceberg are both serious, production-grade answers to the same set of problems. The right choice isn’t format-first — it’s architecture-first.

Start with your storage. It’s already there. You don’t need to migrate it.

Pick the format that matches your primary compute. Databricks-primary? Delta with UniForm for read interoperability. Snowflake-primary or multi-engine? Iceberg with Snowflake as catalog for managed tables, external catalog for maximum portability.

Design maintenance responsibility explicitly. Managed platforms (Databricks, Snowflake as catalog) handle compaction, checkpointing, and snapshot cleanup automatically. External catalog configurations hand that responsibility to your team. Neither is wrong — but it needs to be a deliberate choice, not a surprise six months into production.

Keep the catalog layer open. Whichever format you choose, investing in open catalog infrastructure (Polaris, Nessie, Unity Catalog OSS) preserves the flexibility to add compute engines, change platforms, or federate across clouds without re-architecting your data.

The data lakehouse isn’t a product you buy. It’s an architecture you build — incrementally, on storage you already own, with formats and engines you can swap as the ecosystem evolves. Both Delta and Iceberg make that possible. The job of a platform team is to make sure the choices you make today don’t become the constraints you apologize for tomorrow.