Use Case 1: Order Status Tracking at Petabyte Scale

A deep dive on Hudi’s MoR + Record-Level Index + Partial Updates + Async Compaction, and why Delta and Iceberg can’t match it on this workload.

TL;DR

Streaming upserts on a hot key are the workload most lakehouse formats quietly fall over on. For a realistic order-status pipeline (50k state transitions per second at peak, 40-column rows, a multi-petabyte history table, and a few-second point-lookup SLA), the only OSS design that holds up is Apache Hudi’s combination of Merge on Read + Partial Updates + Record-Level Index + Async Compaction. The headline number: log volume shrinks by roughly 10x compared to a naïve “write the whole row” approach, S3 PUT cost drops with it, and point lookups stay in low single-digit seconds on a PB table. Delta and Iceberg both lose on this workload. Delta rewrites Parquet on every UPDATE, Iceberg’s v2 MoR is built for delete-heavy patterns, and neither has a record-level index in OSS.

1. The Workload

You run an e-commerce platform at scale. Orders cycle through ~8 states:

placed → paid → packed → shipped → out_for_delivery → delivered → cancelled → returned

At peak, you see 50k+ state transitions per second. The orders table isn’t just orders, it’s the unified order-event store (header, status changes, payments, fulfillment, returns, line items). Across years of retention that combined table runs into the petabytes. The order row is wide. Around 40–60 columns: customer info, shipping address, line items, pricing, payment, fulfillment, timestamps, plus status and status_ts. A single state transition changes exactly two of them: status and status_ts.

SLA

• An order placed at t=0 is queryable as "placed" by t + 10 minutes.
• A single-order lookup (WHERE order_id = 'ABC123') returns in a few seconds on the multi-PB table.
• The pipeline never falls behind during a 10x traffic spike.

This is the workload. Now look at the four hard problems every team hits trying to build it.

2. The Hard Problems in Lakehouse Tech

Lakehouses promise the union of warehouse semantics (ACID, schema, indexes) with data-lake economics (cheap storage, open formats, engine-neutral). For analytical workloads with infrequent updates, they deliver. For high-velocity update workloads like this one, four hard problems show up. Every table format addresses them differently, and the differences are where the cost and SLA outcomes diverge.

Write amplification

Most lakehouses serialize the entire row to disk on every update, even when only a fraction of the columns change. A 40-column row where two columns moved still gets written as 40 columns somewhere on disk. At PB scale and 50k upserts/sec, this means writing terabytes of unchanged data every day. The compute and storage bills follow the byte volume, not the change volume.

The worst version of this is Copy-on-Write, where the same Parquet base file is rewritten whole on every commit that touches any row in it. Hot keys make it dramatically worse: a small number of file groups absorb most of the writes, and those files get rewritten 8, 10, 20 times over the lifetime of a single order.

Indexing for point lookups

OLAP storage formats are built around scan-oriented queries. A primary-key lookup on a wide PB table is the opposite shape of what they’re optimized for. Without a record-level index, every point lookup falls back to partition pruning plus column-stats pruning. That narrows the file set but still requires reading manifest files and scanning data files. Latency stretches from a few seconds (what your support agent needs) to tens of seconds (what a column-stats scan delivers on a PB table).

The fix is a record-level index that resolves a primary key directly to its file group in a single lookup, not partition + column-stats pruning over the whole table.

Ingest vs maintenance competition

Compaction, optimization, and cleaning are not free. They read large amounts of data, write new files, and need their own compute. When they run on the same cluster as ingest, throughput drops and conflict rates rise. When they don’t run, log files accumulate, base files fragment, and read performance degrades.

The challenge is decoupling maintenance from ingest without losing freshness or fault tolerance. Most formats run maintenance as a synchronous Spark job that competes for the same cluster.

3. The Hudi Architecture

3.1 Merge on Read: Cheap Writes

Under MoR, an UPSERT does not rewrite the base Parquet file. It writes a log block: an append-only segment of an Avro/Parquet log file colocated with the base file in the same file group.

Every state transition becomes a small log append. The base file is unchanged. The hot Parquet files that would have been rewritten 8 times across an order’s lifetime are now rewritten only when compaction runs (more on that below).

At 50k upserts/sec, the difference between a log append and a file rewrite is the difference between scaling and falling over.

Two things worth knowing:

1. Log files live in the same file group as the base file they belong to, so the merge logic on read knows exactly where to find them.
2. Log files are append-only. Each commit produces a new log file (or appends a new block to an existing one). They’re cheap to write, cheap to delete.

3.2 MERGE INTO Partial Updates: The 10x Log Shrink

This is the single biggest lever, and it’s the one most teams get wrong because the mechanism is widely misunderstood.

Two common misconceptions:

1. “Set PartialUpdateAvroPayload and you get partial updates." False. That payload class controls how records are merged (null-aware reconciliation on the read/compaction side), but it does not shrink the on-disk log block. If you write a record with 37 nulls and 3 real values, the log block still serializes all 40 fields.
2. “Partial updates come from sending narrow rows in your DataFrame.” Also false. The default Hudi UPSERT path writes the full incoming schema into the log, regardless of how many fields are populated.

The actual mechanism is at the SQL layer. On a Hudi 1.0+ MOR table, when you run:

MERGE INTO orders t
USING status_updates s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET
  t.status     = s.status,
  t.status_ts  = s.status_ts

Hudi’s MERGE INTO planner inspects the UPDATE SET clause, builds a narrow Avro schema containing only order_id, status, status_ts, and emits a log data block written against that narrow schema. The other 37 columns are not present on disk in the log block at all. Not nulls, not defaults, just absent from the schema.

This behavior is controlled by hoodie.spark.sql.merge.into.partial.updates, which defaults to true since Hudi 1.0.0. The writer auto-derives the partial schema and stamps it on the log block via hoodie.write.partial.update.schema (you don't set this manually).

At read or compaction time, the log block is merged back into the base file according to the table’s PartialUpdateMode. Two modes exist:

• IGNORE_DEFAULTS: fields with default values in the partial record are skipped during merge; values from the prior version are kept.
• FILL_UNAVAILABLE: fields marked "unavailable" via hoodie.write.partial.update.unavailable.value are skipped, using the prior version instead.

Concrete numbers on the wire and disk:

• Full 40-column row, Avro-encoded: ~2 KB per record
• Partial 3-field block, Avro-encoded: ~150 bytes per record

That’s a ~13x reduction in log byte volume. The downstream effects compound:

• S3 PUT cost drops because log files cross size thresholds less often.
• Compaction reads less data when it folds logs back into base files.
• MoR reads merge less data at query time. The on-the-fly merge per file group scans kilobytes instead of megabytes.
• You can run more delta commits between compactions because the log doesn’t bloat as fast.

Architecturally, what this means for the ingest pipeline:

The cleanest pattern is:

1. Run a Spark SQL MERGE INTO job on a tight cadence (every 1–5 minutes) that applies the staging delta to the orders table.
2. The MERGE INTO planner builds the narrow schema automatically. Your pipeline never has to think about it.

This is not a micro-optimization. On a status-update workload, it is often the single thing that decides whether you meet or miss the freshness SLA.

3.3 Record-Level Index: Both Sides of the Pipeline

The Record-Level Index (RLI) is a partition inside Hudi’s metadata table, a separate Hudi-managed table at <basepath>/.hoodie/metadata/. The RLI partition maps record key → file group with HFile-backed point-lookup performance.

It does work on both sides of the pipeline:

On ingest: when the writer receives a batch of upserts, it needs to know which file group each order_id belongs to. The default index strategy (Bloom) probes Parquet file footers for matching record-key ranges. At PB scale, that's hundreds of thousands of footer reads per commit and a long tail of false-positive scans. The Bucket index avoids that but locks you into a fixed bucket count and pre-sharded ingest.

RLI gives you exact file-group lookup in O(1) HFile reads. No bloom probes, no scans, no false positives, and crucially no hot-partition skew when one popular SKU floods the stream into a single partition. RLI’s hash distributes record-key-to-file-group mappings across its own file groups.

On reads: a query like WHERE order_id = 'ABC123' is rewritten to first consult RLI, get the exact file group, and only read that file group's base + log files. On a PB-scale table with thousands of file groups, this is the difference between a 30+ second partition scan and a few-second lookup (metadata HFile read + base + log fetch, mostly bound by S3 latency, not Spark planning).

This is the feature OSS Iceberg and Delta don’t have an equivalent for. They rely on partition pruning + column-stats pruning to narrow the file set, but neither resolves to a single file group from a primary key.

3.4 Async Compaction: Decoupled Compute

If MoR is “cheap writes,” compaction is “pay later.” Logs accumulate in front of base files; eventually you need to fold them back in or read performance degrades and storage bloats.

In Hudi, compaction runs as a separate scheduled action on the timeline. Two modes:

• Inline compaction: the writer blocks every N commits to run compaction. Predictable but couples ingest throughput to compaction throughput.
• Async compaction: a separate process (a different Spark job, often on a different cluster) reads the compaction plan from the timeline and executes it. Writers keep ingesting. Readers keep querying. Compaction runs to its own cadence.

For the order-status workload, async is the only option. The recipe:

1. Set hoodie.compact.schedule.inline = true on the ingest writer (the default is false). This makes the writer attempt to schedule a compaction plan after each write. Fast, because "scheduling" only writes a plan instant to the timeline; it doesn't execute the merge. A plan is a manifest of which file groups need merging.
2. Run a separate compaction job (HoodieCompactor from hudi-utilities, or a standalone Spark job) that reads pending compaction plans from the timeline and executes them.
3. The compaction job runs on its own cluster, with its own resource profile (more CPU, less network), on a cadence matched to log growth (not ingest throughput).

The decoupling matters for cost too. Compaction is bursty. It processes large batches of logs in one shot, then idles. Running it on its own cluster lets you spin that cluster up on a schedule, run for an hour, spin it down. You don’t pay for compaction headroom on the ingest cluster 24/7.

Writer Configuration

# Table type: write log blocks, not rewrites
hoodie.datasource.write.table.type        = MERGE_ON_READ
# Partial updates: log only the columns the MERGE INTO touched
# (default: true on Hudi 1.0+ MOR tables; included here for clarity)
hoodie.spark.sql.merge.into.partial.updates = true

# Record-Level Index in the metadata table
hoodie.metadata.enable                    = true
hoodie.metadata.record.index.enable       = true
hoodie.index.type.                        = RECORD_INDEX

# Async compaction/Cleaning: don't block ingest
hoodie.compact.inline                     = false
hoodie.clean.async                        = true

Reader Configuration (Spark)

hoodie.enable.data.skipping               = true
hoodie.metadata.record.index.enable       = true

4. Hudi vs Delta: Deep Comparison

Delta is the closest competitor in the “transactional lakehouse” space. Here is precisely where it diverges on this workload.

4.1 Update Semantics

Delta’s MERGE INTO and UPDATE are implemented as copy-on-write under the hood. The optimizer identifies files containing matching rows, reads them, applies updates, and writes new files. There is no log-block equivalent.

Deletion Vectors (Delta 3.0+) add a “merge-on-read” variant, but only for deletes. A DV is a bitmap marking deleted rows; readers skip those rows. For an UPDATE, the row still needs to be written to a new file (with the new values), and the old row is marked via DV. So UPDATE still incurs a write.

For the order-status workload, where every state transition is an UPDATE, this is a complete loss. You pay the full rewrite cost on every update.

4.2 No Partial Update Semantics

This is where the contrast with Hudi is sharpest. Both engines support the same SQL syntax: MERGE INTO t USING s ... UPDATE SET col1 = ..., col2 = .... But the on-disk effect is completely different.

• Hudi 1.0+ MoR: the planner inspects the UPDATE SET clause and writes a log data block whose schema contains only the listed columns. The 37 untouched columns are not on disk in the log block.
• Delta: the same SQL writes the full 40-column row into a new Parquet file (with DV marking the old row in 3.0+, or full rewrite of the old file pre-3.0). The 37 untouched columns travel anyway.

Delta has no on-disk representation where only the changed columns are persisted and merged at read time. The “column-level” semantics live in the SQL layer; the physical write is whole-row.

4.3 No Record-Level Index

Delta has no record-level index in OSS. It relies on:

• File-level statistics in the transaction log (min/max per column)
• Z-ordering / liquid clustering for spatial locality
• Bloom filter indexes (limited, file-level, not record-level)

For point lookups, these help narrow the file set but don’t resolve to a single file group from a primary key. You still scan multiple files.

4.4 OPTIMIZE Competes With Ingest

Delta’s compaction is OPTIMIZE, which can also re-Z-order. It runs as a synchronous Spark job. Running it concurrently with ingest is supported via OCC, but practically it competes for cluster resources and increases conflict-retry rates. The clean "separate cluster" model that Hudi async compaction enables is harder to wire up in Delta.

4.5 Where Delta Wins

For pure analytical workloads with infrequent updates (slowly-changing dimensions, batch ETL), Delta is excellent, often simpler to operate, and has a more mature SQL ecosystem on Databricks. It’s not built for high-velocity, partial-column streaming updates on hot keys.

5. Hudi vs Iceberg: Deep Comparison

Iceberg is the other major contender. The comparison is more nuanced because Iceberg v2 introduces merge-on-read primitives, but they’re built for a different shape of problem.

5.1 Iceberg v2 MoR: Built for Delete-Heavy, Not Update-Heavy

Iceberg v2 introduced two delete file formats:

• Positional deletes: a file listing (filepath, rowpos) pairs for rows to skip
• Equality deletes: a file listing predicate values; rows matching are skipped

For an UPDATE, the canonical Iceberg approach is:

1. Write a delete file marking the old row
2. Write a new data file with the new row

Two file writes per update. The “MoR” here means readers have to merge data files with delete files at scan time, not that updates avoid rewriting data.

For the order-status workload: every state transition writes a delete-file entry plus a new data row. Storage and write cost are comparable to CoW. The “savings” go to delete-heavy workloads (GDPR right-to-be-forgotten, soft deletes), not update-heavy ones.

5.2 No Partial Update Semantics

Iceberg has no concept of partial-schema updates. The new data file written for an UPDATE (whether via copy-on-write or v2 merge-on-read) contains the full 40-column row. No log blocks. No narrow-schema data files. No column-level merge semantics at the storage layer.

The same MERGE INTO ... UPDATE SET status = ... that produces a 3-column log block on Hudi produces a 40-column row in a new data file on Iceberg.

5.3 No Record-Level Index in OSS

Iceberg’s read-path pruning relies on:

• Hidden partitioning (transforms on partition fields)
• Manifest-level pruning (min/max stats per file at the manifest layer)
• File-level Parquet column stats

These are excellent for analytic queries with partition predicates. For point lookups on a non-partitioned key like order_id, you fall back to scanning manifest files and matching column stats. At PB scale, this is hundreds of file reads per lookup. Tens of seconds at best, not few-second territory.

5.4 Equality Deletes Add Read Cost

When you adopt v2 equality deletes for updates, every reader of the table has to also read all relevant equality-delete files and apply them. At high update velocity, the delete-file count grows, and every read query pays a per-row check cost. The performance cliff is real. Iceberg’s rewrite_data_files action is the cleanup, but it doesn't run as smoothly async as Hudi's compaction.

5.5 Where Iceberg Wins

Iceberg is excellent for:

• Large append-mostly analytical tables
• Tables queried by many engines (Spark, Trino, Flink, Snowflake; Iceberg’s catalog story is the cleanest)
• Workloads dominated by partition predicates and time-range queries
• Soft-delete or GDPR-style workloads where v2 equality deletes shine

It is not built for few-second point lookups on a primary key over a multi-PB updating table.

6. Production Checklist

If you’re putting this design into production, here’s what to verify before flipping the switch.

MERGE INTO and partial updates

• Hudi version is 1.0.0 or later (partial-update log blocks ship in 1.0+).
• Table type is MERGE_ON_READ. Partial-update log blocks are an MoR-only feature.
• hoodie.spark.sql.merge.into.partial.updates left at default (true). Verify the docs/configs once if you've inherited a custom override.
• Ingest pipeline applies updates via Spark SQL MERGE INTO (directly, or inside Structured Streaming forEachBatch). Plain DataFrame UPSERT does not trigger partial-schema log blocks.
• UPDATE SET lists only the columns that actually change. The narrow schema is derived from this clause.
• event_ts is monotonic enough at the record-key level that out-of-order delivery is rare; the precombine field handles the outliers.

Index sizing

• RLI file-group count tuned for cardinality. Default: 5–100 file groups for the RLI partition; tune up if order_id cardinality is in the billions.
• Metadata table HFile compaction tuned (hoodie.metadata.compact.max.delta.commits).

Compaction cadence

• hoodie.compact.inline.max.delta.commits set to a value that bounds log file size (5–10 is typical).
• Separate compaction job scheduled with cluster sizing matched to log growth.

Cleaner

• hoodie.cleaner.policy = KEEP_LATEST_COMMITS with enough retention for in-flight queries and time-travel needs.
• Async cleaning enabled (hoodie.clean.async = true). Never let cleaning block ingest.

Monitoring

• Track per-file-group log size (alert if any group’s log > 10x its base).
• Track compaction lag (instants pending vs completed).
• Track ingest commit latency P50/P95/P99.
• Track RLI lookup latency from sample point-lookup queries.

7. Conclusion

Lakehouse table formats are not interchangeable. They embed design tradeoffs that show up as cost and SLA outcomes at scale, and the order-status workload is one of the sharpest places those tradeoffs surface.

For wide rows, tiny change footprints, high update velocity, hot-key point lookups, and a PB-scale history:

• CoW always loses on cost because it rewrites entire base files for every change, dragging unchanged columns along for the ride.
• Iceberg v2 MoR doesn’t help because it’s designed for delete patterns, not partial updates.
• Delta’s deletion vectors don’t help because UPDATE still rewrites the row.

Hudi’s design is the only OSS combination that pulls the four right levers together:

1. MoR turns writes into appends.
2. MERGE INTO partial updates turn the log block into “only what changed.”
3. Record-Level Index keeps both ingest tagging and point lookups in O(1) territory.
4. Async compaction decouples the eventual rewrite from the ingest cadence.

The cost outcome (somewhere in the 4–7x range on operational spend for a realistic PB pipeline) falls out of the design, not from heroic tuning. Get the four pieces right and the rest of the system has room to breathe.