🧭Taming the Small File Monster in Fintech: Iceberg & Kudu

In FinTech, “small files” aren’t just a storage nuisance; they are a financial and operational risk. If a fraud detection query takes 10 minutes instead of 10 seconds because it’s wading through a million tiny files, the fraudulent transaction has already cleared the bank. To survive this, you need Compaction — the technical “surgery” that squashes fragmented shards into high-performance blocks. But the way Apache Iceberg and Apache Kudu perform this surgery is fundamentally different: one is a scheduled house-cleaning, the other is a self-healing organism.

The Use Case: Real-Time Fraud Detection

Imagine a global payment processor like Paypal or Zelle. Every second, thousands of transactions pour in flooding the system every second. Each event is a 2KB JSON snippet written to your data lake. By the time your fraud detection engine wakes up to analyze the pattern, it is staring into a digital graveyard of five million tiny files scattered across the storage layer.

When a data scientist tries to run a query to detect a fraud pattern across the last 24 hours, the query engine spends 90% of its time just “opening and closing” these millions of files. The dashboard that should take 5 seconds now takes 5 minutes. The fraud is detected too late.

Solution 1: Apache Iceberg (The Modern Lakehouse)

Iceberg is a Table Format. It doesn’t own the “writing” process; it just keeps the books (metadata). Because it sits on top of immutable object stores like HDFS/S3, it cannot “edit” a file. It can only replace it.

The Detailed Compaction Workflow

Iceberg uses a procedure called rewrite_data_files. Here is exactly what happens when you trigger it:

1. File Selection (The Planning Phase): Iceberg scans its metadata to find “unoptimized” files. It looks for files smaller than your target-file-size-bytes (default 512MB) or files with too many associated "Delete Files."
2. Bin-Packing: It groups these small files into “File Groups.” Each group is a bundle of data that, when merged, will result in one perfectly sized Parquet file.
3. The Rewrite (The Execution Phase): A Spark or Flink job has to be run to read these groups, applies any row-level deletes (merging them into the data), and writes out a brand-new, clean Parquet file.
4. The Atomic Swap (The Commit): This is the most critical step. Iceberg creates a new metadata snapshot. a) It points to the new large file. b) It “removes” the pointers to the old small files. c) Crucially: The old files stay on HDFS/S3 for “Time Travel” until you run an expire_snapshots command.

Solution 2. Apache Kudu: The (Real-time Engine)

Kudu is a Storage Engine. It owns the disks and the processes. It doesn’t wait for you to run a job; it has a “Maintenance Manager” that acts like a built-in janitor.

The Detailed Compaction Workflow Kudu handles data in a “Log-Structured” way. It never lets small files accumulate to a breaking point.

1. MemRowSet to DiskRowSet: When transactions arrive, they sit in memory (MemRowSet). When that memory is full, Kudu flushes it to a DiskRowSet. This creates a relatively small on-disk file.
2. Delta Compaction (Minor): If a FinTech record (like a transaction status) is updated, Kudu doesn’t rewrite the data file. It writes a “Delta” to a separate log. The Maintenance Manager periodically merges these small delta logs together so that a “Read” doesn’t have to check 100 different logs to find the current status of one transaction.
3. RowSet Compaction (Major): Kudu constantly watches its DiskRowSets. If it sees multiple small RowSets that cover overlapping “Primary Key” ranges, it merges them into a single, large, sorted DiskRowSet.
4. The “Live” Update: Because Kudu manages the tablet, it can do this while the system is under load. It doesn’t need a separate Spark cluster to wake up and “fix” things.

The Architect’s Final Verdict: When to recommend what?

Use Iceberg (The “Archive & Audit” King): If your FinTech team needs a single source of truth for billions of historical transactions. You use Iceberg because it’s cheap (HDFS/S3) and standardized. You accept the “Small File Problem” as a maintenance task you run every night or every hour. A good tip is, use Merge-on-Read for the daytime ingestion (fast writes) and run a Major Compaction at midnight to turn everything into optimized Parquet.

Use Kudu (The “Hot Data” Champion): If you are building a Live Ledger or Fraud Dashboard. If the data needs to be updated constantly and queried immediately, you cannot wait for an “Iceberg Compaction Job” to finish. Kudu’s internal, automated management ensures that the “Small File Problem” never actually happens because it’s being fixed every few milliseconds.