We Migrated a Monolithic Redshift Cluster to Apache Iceberg on EMR — And Cut Costs by 50%

The real story: what broke, what we decided, and the numbers that made it worth it

— -

At some point, every data team hits the wall.

For us, it was a 16-node RA3 Redshift cluster managing approximately 2 petabytes of advertising finance data. The system worked — until it didn’t. Daily processing consumed 16 to 17 hours, leaving almost no headroom for anything else: no ad-hoc queries, no concurrent revenue reporting jobs, no room for the business to grow.

And the business was growing. New advertising products, acquisitions, expanded reporting requirements. Every new source of revenue needed a slice of the same exhausted cluster.

We had a choice: scale Redshift horizontally and keep paying the premium, or rethink the architecture from scratch.

We chose the latter. This is the story of how we migrated from a monolithic Redshift warehouse to a horizontally scalable, EMR-based Medallion architecture with Apache Iceberg — and what we learned along the way.

The problem with the old architecture

The original finance data warehouse was built around a centralised Redshift cluster. At the time, it made sense: Redshift was fast, familiar, and easy to query. But Redshift is fundamentally a coupled compute-and-storage system. When your data grows, you scale the whole cluster, not just the part that’s under pressure.

By the time we started planning the migration, we had three compounding problems:

The cluster was saturated. With 16–17 hours of daily batch processing, concurrent workloads — ad-hoc finance queries, revenue reconciliation jobs — were constantly queuing. The cluster had essentially become a single-purpose machine.

Reprocessing windows were painful. Advertising conversion data can restate up to 90 days back. On a monolithic cluster, running a 90-day backfill while keeping daily jobs running was a capacity nightmare. We had to choose between freshness and stability.

The cost curve was unsustainable. RA3 nodes are expensive. Scaling the cluster to handle growing data volumes meant a near-linear increase in spend, with no architectural ceiling in sight.

Why we chose EMR + Medallion, and not just a bigger Redshift

Before committing to EMR, we seriously evaluated the alternatives:

Bigger Redshift: More nodes, more RA3. Would have unblocked us for 12–18 months, but didn’t solve the fundamental coupling of compute and storage. Cost would scale linearly with data volume. Rejected.

Redshift Serverless: Attractive for ad-hoc queries, but at the time, it wasn’t suited for long-running, high-throughput batch workloads with complex joins across billions of rows. Rejected for our core use case.

Snowflake: Genuinely considered. Compute-storage separation, good concurrency. But we were already deeply invested in the AWS ecosystem — LakeFormation, Glue, S3, IAM. Switching vendors would have meant re-engineering every integration. Rejected on operational grounds.

EMR with Spark: Decouples compute from storage entirely. Scale workers up for heavy batch jobs, scale down when idle. Store everything in S3. Pay for compute hours, not node-months. This was the path.

The Medallion architecture (Bronze → Silver → Gold) was the natural companion to EMR because it enforces a clean separation of concerns at the data layer, not just the infrastructure layer.

The architecture we built

Data ingestion: metadata-driven, not scan-driven

The first design decision was how to detect upstream changes. The naive approach — scanning trillions of rows across fact tables to detect what changed — was a non-starter at our scale.

Instead, we built a metadata-driven ingestion layer. The upstream data platform emits SNS notifications whenever fact table partitions are updated. We consume these via SQS (as a buffer for the 10,000+ daily message volume), trigger Lambda functions that parse the partition metadata, and persist the state to DynamoDB. The compute layer only processes the partitions it’s told have changed.

This pattern alone eliminated a significant amount of wasted compute compared to full-scan approaches.

The three layers

Bronze layer is deliberately minimal. Raw data lands here with light transformations — deduplication, type casting, schema validation. No business logic. No aggregation. The rule is: if you wouldn’t be comfortable explaining why a transformation belongs here, it doesn’t belong here.

Silver layer applies product-specific filtering and initial business transformations. This is where scope is established — which advertising product lines are included, which records are relevant. Data entering the Gold layer is already narrowed to the right set of records.

Gold layer is where the real complexity lives. Revenue spreading across billions of trafficking records using impression share methodology, multi-source data integration, multi-currency conversion (local currencies to USD), and product categorisation logic flexible enough to be updated without reprocessing upstream layers.

After Gold, a dimension enrichment layer joins in campaign, advertiser, and creative metadata — adding the reporting dimensions that finance teams actually query against.

Decoupled layers via async messaging

Each layer communicates with the next via DynamoDB messages rather than direct calls. This was a key architectural choice. It means each layer operates independently: Bronze doesn’t know or care what Silver does with its output. Silver doesn’t wait on Gold. If one layer fails, it can be retried in isolation without cascading failures across the pipeline.

The architecture behaves like a set of loosely coupled microservices operating on data, which made debugging, partial reprocessing, and layer-level upgrades dramatically easier than the old monolithic approach.

Production cluster sizing

The production system runs 200 to 400 EMR nodes, scaling elastically based on workload. That sounds large, but the key difference from the old Redshift architecture is that we only pay for those nodes during active processing hours. When jobs complete, the cluster scales down. The old Redshift cluster was running 24/7 whether or not it was doing useful work.

The Iceberg migration: why we did it, and what it unlocked

Once the EMR Medallion architecture was stable, the next logical step was migrating from raw S3 Parquet files to Apache Iceberg table format. This is the part of the story most articles gloss over — not just what Iceberg is, but why it mattered specifically for our workload.

The problem with raw S3 Parquet

Our Gold layer tables were stored as Parquet files in S3, partitioned by date. This worked, but had a set of friction points that compounded over time:

• No ACID guarantees. Partial writes, failed jobs, and concurrent updates could leave tables in inconsistent states. Recovery required manual intervention.
• Schema changes were painful. Adding a new column required coordinating with every downstream consumer and often rewriting historical partitions.
• Reprocessing was expensive. The 90-day conversion restatement window meant rewriting up to 90 date partitions regularly. With raw Parquet, this meant deleting and rewriting files — no transactional safety net.
• No time travel. When finance teams needed to reconcile a prior month’s numbers against the version of the data that existed at month-close, we had no clean way to do that.

What Iceberg gave us

Hidden partitioning was the immediate win. With raw Parquet, query performance depended heavily on whether the analyst remembered to include the partition column in their WHERE clause. Iceberg’s metadata layer handles partition pruning automatically — queries filter efficiently even when users don’t know the physical partition scheme.

ACID transactions meant we could run 90-day restatements without fear. Iceberg’s atomic commit model ensures that a failed reprocessing job leaves the table in its last known good state, not a half-written mess. Recovery went from a manual, hours-long process to a retry.

Schema evolution without rewriting. We could add new reporting dimensions — new advertiser attributes, new creative metadata fields — by adding columns to the Iceberg schema. Existing Parquet files were read with null values for the new columns. No backfill required.

Time travel for month-close reconciliation. Finance teams can now query the exact snapshot of the data that existed at any historical point. This was a compliance and audit requirement that was genuinely difficult to meet with raw file storage.

The performance and cost numbers

The results after migrating to the full EMR + Iceberg architecture:

• Daily processing time reduced by ~30% — from 16–17 hours to 10–12 hours
• Hardware costs reduced by ~50% — from 200 nodes to approximately 100 nodes for equivalent throughput, driven by Iceberg’s metadata-optimised query planning and better compression ratios
• Query performance improved 30–40% on analytical workloads, due to metadata-based predicate pushdown reducing unnecessary data scans

The node reduction was the number that got finance leadership’s attention. Halving the node count while handling growing data volumes is the kind of outcome that justifies a migration.

What we’d do differently

No migration at this scale is clean. A few things we’d change:

Invest in the dev environment earlier. We eventually created a dedicated development environment using AWS Service Catalog — 10-node test EMR clusters with pre-configured bootstrap scripts. We should have done this at the start of the project. The first few months of testing in production-adjacent environments created unnecessary risk and slowed iteration.

Write the layer contracts before writing the code. The async inter-layer messaging worked well, but the contract between layers (what fields are guaranteed, what’s optional, what error states are signalled) evolved informally. We spent time debugging failures that were really contract mismatches. A formal schema registry for inter-layer messages would have saved weeks.

Plan the Iceberg migration in parallel, not in sequence. We treated the EMR Medallion migration and the Iceberg migration as sequential projects. In retrospect, designing for Iceberg from the beginning would have avoided a second round of schema and partition design work. The two architectures are complementary enough to be planned together.

Key takeaways

If you’re evaluating a similar migration, here’s what I’d want you to know:

The case for migrating off a monolithic Redshift cluster is rarely about Redshift being bad. It’s about the fundamental mismatch between coupled compute-storage and workloads that need to scale asymmetrically — lots of compute for batch processing, very little for idle periods.

EMR solves the compute-storage coupling. Apache Iceberg solves the data management layer — ACID, schema evolution, time travel, and efficient metadata-based querying — that raw S3 Parquet can’t provide.

Together, they gave us a system that handles 2+ petabytes, supports 90-day restatement windows, lets finance teams do time-travel queries for month-close reconciliation, and costs half as much to run as the architecture it replaced.

The migration is not trivial. But at the scale where your data warehouse is eating your entire engineering capacity just to stay alive, the question isn’t whether to migrate. It’s how to do it without breaking the revenue numbers that finance depends on.

We did it. The numbers held. And we came out the other side with headroom to grow.

I’m a Senior Data Engineer with 19+ years of experience building enterprise-scale data platforms. If you found this useful or have questions about the architecture, connect with me on LinkedIn.

Tags: Data Engineering · Apache Iceberg · AWS EMR · Redshift · Data Lakehouse · Big Data · Amazon Web Services · Data Architecture