Scaling Apache Iceberg: Best Practices for Reliable Table Maintenance

Practical ways to manage metadata bloat and handle concurrent writes without sacrificing data lake performance

Author: Raghav Jha

Apache Iceberg may be the bedrock of your modern lakehouse, but without consistent operational oversight its advantages quickly erode: query latencies climb, storage bills grow, and reliability suffers. This post shows why proactive table maintenance, across snapshot expiration, data compaction, and orphan file removal, is a functional necessity. This is especially true if you want to maintain performance, control costs, and scale Iceberg safely in production.

Building on our previous discussion of Iceberg’s layered architecture and concurrency, we now focus on the operational requirements. Without regular intervention, even the most efficient tables suffer from metadata accumulation, stale snapshots, and orphan files, leading to reduced performance and increased storage expense.

In this technical deep dive, we examine the end-to-end Iceberg maintenance lifecycle, offering proven methodologies for managing write concurrency and resolving the recurring FileNotFoundException. From infrastructure scaling to performance tuning, these operational insights and code implementations ensure your production environment remains resilient, cost-efficient, and high-performing.

Common Challenges in Iceberg Table Management

Iceberg solves many hard problems in data lakehouse design. However, it introduces its own operational complexity. Here are the challenges teams most commonly encounter in production:

• Dynamic Schema Evolution Management: Iceberg supports smooth schema changes, but frequent updates still require tight coordination. Without clear change processes, downstream consumers can silently fail or read inconsistent data structures.
• Cross-Engine Catalog Synchronization: Using multiple compute engines on the same tables can cause metadata to drift out of sync. Without automated alignment, outdated manifests and metadata often lead to occasional file access errors during queries.
• Snapshot Isolation and Commit Resiliency: In high-concurrency environments, many writers can compete for locks and sometimes leave metadata in a bad state. Fixing these issues often means digging into manifest logs to understand overlapping operations, which demands strong observability.
• Optimization of Storage and Compaction Overhead: At scale, the steady growth of many small files is a constant problem. As metadata grows, compaction becomes more expensive, so teams must balance aggressive cleanup against overall compute cost.
• Governance in Distributed Environments: Managing multi-tenant access while keeping a clear audit trail and lineage is difficult. Without centralized guardrails, tables are exposed to accidental overwrites and uncoordinated structural changes from different teams.

Why Iceberg Tables Management is Important

These production challenges highlight why disciplined Iceberg table management is critical. Iceberg’s write-heavy nature quickly increases metadata, so regular cleanup is essential to prevent performance degradation, rising storage costs, and reduced reliability.

Essential Management Drivers

• Data Integrity & Consistency: Iceberg leverages snapshot isolation to provide a stable read view, ensuring that concurrent write operations do not result in partial or corrupted data exposure during analytical processing.
• Metadata and Storage Optimization: While the framework provides native cleanup and compaction mechanisms, these procedures require deliberate scheduling to prevent metadata accumulation and file fragmentation from impacting table performance.
• Query Performance Tuning: By merging small files and executing effective partition pruning, Iceberg allows compute engines to skip unnecessary data, sustaining low-latency queries even as data volumes scale.
• Governance and Lineage Auditability: Every table modification is captured through versioned snapshots, creating an audit trail that simplifies compliance requirements and enables precise tracking of data evolution.
• Reproducibility and Disaster Recovery: Native time-travel support allows users to query historical snapshots, ensuring that data science experiments and production pipelines remain fully reproducible regardless of updates.

End-to-End Cleanup Workflow

To put these management principles into practice, the comprehensive Apache Iceberg metadata cleanup flow involves four sequential steps:

1. Expire Snapshots (expire_snapshots)

• Action: Removes old table history by deleting outdated snapshot metadata.
• Effect: Old data files become “orphan candidates” but are not immediately deleted.
• Benefit: Results in smaller metadata and fewer snapshots.

2. Rewrite Data Files (rewrite_data_files)

• Action: Compacts files by merging many small files into fewer, larger optimized files.
• Effect: Original small files become orphans.
• Benefit: Enables faster queries and reduces file fragmentation.

3. Rewrite Manifests (rewrite_manifests)

• Action: Optimizes metadata index files by merging multiple manifests into fewer.
• Effect: Reduces metadata overhead without modifying data files.
• Benefit: Enables faster query planning.

4. Remove Orphan Files (remove_orphan_files)

• Action: Deletes physical files no longer referenced by table metadata.
• Effect: Uses a safety buffer to prevent deleting in-use files.
• Benefit: Frees storage by cleaning up “garbage.”

A step-by-step breakdown of the end-to-end data lake cleanup workflow using standard Apache Iceberg table maintenance actions.

Case Study: AWS Glue FileNotFoundException

To ground these best practices in reality, the following case study illustrates how a Glue-based Iceberg maintenance job can fail with FileNotFoundException, and how to fix it.

Problem: An AWS Glue, a serverless data integration service, Iceberg maintenance job intermittently failed with FileNotFoundException during snapshot expiry, compaction, and orphan-file cleanup.

Cause: remove_orphan_files listed files in S3 (Simple Storage Service), then checked them later. During that gap, Spark or Iceberg could delete short-lived files created by retries, speculation, or aborted writes.

Why files vanish:

• Task retries: failed Spark tasks may leave temporary files before a retry succeeds.
• Speculative execution: Spark may run duplicate task attempts and keep only the winner.
• Write aborts: Iceberg deletes files from failed or losing attempts.

Race timeline:

1. t0: A task writes data/data01.
2. t1: remove_orphan_files lists and sees the file.
3. t2: The task fails or loses speculation, so Iceberg deletes the file.
4. t3: The cleanup job checks the file and fails with FileNotFoundException.

Key point: older_than controls deletion eligibility, not listing. Recent files are still scanned, so retries may only succeed by timing luck.

Fix: Avoid live recursive S3 walks. Build a file inventory once with boto3 and pass it to Iceberg using compareToFileList(). Treat orphan cleanup as best-effort and keep safety windows plus small retries.

For continuously written Iceberg tables, use a pre-built inventory snapshot instead of live remove_orphan_files directory walks.

Code snippet (Python):

# Use compareToFileList with a pre-built boto3 inventory instead of
# Iceberg's recursive EMRFS walk to avoid FileNotFoundException.
SparkActions.get().deleteOrphanFiles(iceberg_table) \
.compareToFileList(file_list_df._jdf) \
.olderThan(older_than_ms) \
.execute()

Alternative Mitigation: Ignore Missing Files

Spark provides a native configuration that Iceberg inherits when it distributes physical file listing across executors during large-scale directory walks:

spark.sql.files.ignoreMissingFiles = true

From the official Spark documentation:

*“If true, the Spark jobs will continue to run when encountering missing files and the contents of summary files will be ignored. Otherwise, the jobs will fail.” (Reference)*

With this flag enabled, a FileNotFoundException encountered mid-walk is downgraded from a fatal job-aborting error to a warning log. This allows the maintenance job to complete even when concurrent writes cause files to disappear temporarily.

Core Principles of the Iceberg Metadata Cleanup Strategy

After examining how cleanup can fail in practice, we can now distill a set of core principles to guide an Iceberg metadata cleanup strategy.

Planning and execution safety

• Catalog Selection & Ecosystem Alignment: Choose a catalog suited to your infrastructure (e.g., Glue, Nessie, Representational State Transfer or REST) to prevent metadata fragmentation and ensure multi-engine interoperability.
• Implement Safety Windows: Use safety windows (e.g., 2–6 hours for snapshots) to avoid race conditions with active writers.
• Follow Execution Order: Run cleanup in the correct order: expire_snapshots → rewrite_data_files → rewrite_manifests → remove_orphan_files.
• Account for Propagation: Add a 30–60 second delay between steps to allow metadata propagation and account for S3 consistency.
• Reduce Compaction Conflicts: Reduce conflicts during compaction by enabling partial-progress.enabled and use-starting-sequence-number.

Layout and scheduling

• File Layout Optimization: Arrange small files into optimized data structures and align partitioning with query predicates to maximize data skipping.
• Strategic Scheduling: Schedule cleanup during low-write windows when streaming and batch loads are complete.
• Partition-Aware Cleanup: Prioritize compaction on older, inactive partitions.

Resilience, governance, and observability

• Resilient Execution Logic: Implement retry and continue-on-failure logic with per-table timeouts and exponential backoff.
• Monitor Table Health Metrics: Track snapshot count, manifest count, small file count, and metadata file size.
• Balanced Cleanup Buffers: Avoid over-aggressive cleanup by not using immediate expiration or zero-hour buffers for orphan file removal.
• AWS Glue Governance: For AWS Glue Catalog, monitor version growth to avoid excessive metadata commits.
• Continuous Observability: Monitor indicators like scan durations and file volumes to allow for proactive tuning.

Ideal cleanup frequency:

• Daily for streaming tables
• Weekly for heavy batch tables
• Monthly for low-write tables

Rule: Never clean files that might still be referenced by active writers. Use time buffers, the correct execution order, compaction safeguards, and low-write maintenance windows to prevent concurrent write conflicts.

Summary

Apache Iceberg is a strong foundation for a modern lakehouse, but it only delivers when you actively manage it.

A disciplined maintenance lifecycle which includes expiring snapshots, compacting data, and cleaning up orphan files, while hardening Glue jobs against FileNotFoundException, keeps metadata debt in check, controls storage costs, and preserves query performance.

Committing to these practices lets your Iceberg tables scale with high performance and long-term reliability, and should encourage you to regularly reassess your table maintenance strategy through the lens of resilience, cost, and operational simplicity

To gain a deeper understanding of the architecture of the Guidewire Data Platform, we encourage you to explore our previous blog post: Introducing Guidewire Data Platform. This platform is engineered as an enterprise-grade, internet-scale, and cloud-native Big Data solution.

If you are interested in joining our Engineering teams to develop innovative cloud-distributed systems and large-scale data platforms that enable a wide range of AI/ML SaaS applications, apply at Guidewire Careers.