Observability for an Iceberg Lakehouse

Photo by Hubert Neufeld on Unsplash

Full project in github: https://github.com/vanikkars/iceberg-monitoring

Intro

In previous **article we have built a small streaming sourced lakehouse using Flink Kafka and Iceberg. This time I want to talk about something less glamorous but yet vital for production-ready lakehouse: observability. More specifically, we will focus solely on observability** for Iceberg tables.

More often than not, I see even experienced software engineers getting lost in data-related topics. I have seen people repeatedly make basic mistakes when working with Iceberg:

1. Ignoring the small-files problem One of the most common mistakes is continuously generating tiny Parquet files without proper compaction. This leads to poor query performance, high metadata overhead, increased S3 API calls, and expensive compute costs.
2. Not configuring snapshot expiration Teams often forget to expire old snapshots and orphan files. Over time, this causes metadata bloat, slower query planning, growing storage costs, and in extreme cases even unstable table operations.
3. Using poor partitioning strategies Over-partitioning tables (for example, partitioning by high-cardinality columns such as user_id) creates too many small partitions and makes efficient file compaction almost impossible.

I have seen production tables hundreds of GB in size with an average file size of only around 100 KB, the metadata.json bloated to 500MB in size and loaded into each query plan. The result was tremendously poor performance and skyrocketing infrastructure costs in live projects.

You do not see it until you measure it.

In order to mitigate such issues before they become critical, it is essential to have proper observability in place. Otherwise, you may only discover the problem once business teams start reaching out because a business-critical dashboard failed to load due to poor lakehouse performance. And trust me, that is not an experience you would appreciate.

So in this article I want to cover the key metrics you actually want to monitor on an Iceberg table.

As usual, to try it out:

git clone git@github.com:vanikkars/iceberg-monitoring.git
docker compose up --build

Recap of the Architecture

For this little project I have prepared two Kafka topics that produce the source events:

• users
• transactions

A Flink SQL job consumes both and writes them into Iceberg tables every 30 seconds (one Iceberg commit per Flink checkpoint). The catalog is Apache Polaris speaking the Iceberg REST protocol, and the query engine is Trino with one coordinator and one worker — the less, is more!

High level architecture

Iceberg Metadata Tables in Trino

Iceberg exposes its own internal metadata statistics as regular SQL tables. In Trino you reach them by appending a $<name> suffix to the table:

iceberg.demo."transactions$snapshots"

There are seven metadata tables that matter for day-to-day monitoring:

Fore a brief sanity check to verify if your data layout is “ok” you can check these two:

• $snapshots → commit cadence, records growth, file growth
• $files → small-file problem, distribution of file sizes

Let’s explore these metadata tables.

Snapshots metadata table

SELECT
  *
FROM iceberg.demo."transactions$snapshots"
ORDER BY committed_at DESC;

snapshots metadata table

As we can see, the last column is actually JSON and contains useful details about each snapshot, such as how many records were added or deleted, the total number of records, and more.

We can explore these details by simply extracting the relevant fields from the last column:

SELECT
    snapshot_id,
    operation,
    CAST(element_at(summary, 'added-records')         AS bigint) AS added_records,
    CAST(element_at(summary, 'deleted-records')       AS bigint) AS deleted_records,
    CAST(element_at(summary, 'added-data-files')      AS bigint) AS added_data_files,
    CAST(element_at(summary, 'removed-data-files')    AS bigint) AS removed_data_files,
    CAST(element_at(summary, 'total-records')         AS bigint) AS total_records,
    CAST(element_at(summary, 'total-data-files')      AS bigint) AS total_data_files
FROM iceberg.demo."transactions$snapshots"
ORDER BY committed_at DESC

snapshots-related summary metrics

As you can see above, snapshots are committed roughly every 30 seconds (committed_at), and each commit operation is an append.

We can also observe how records are distributed across commits. For example, the snapshot committed at 2026-05-22 18:29:32.650000+00:00 added only 3,600 records. This can create small-file issues if it is not handled properly through Iceberg maintenance jobs.

Files metadata table

SELECT
    file_format,
    count(*)                                            AS file_count,
    sum(record_count)                                   AS total_records,
    round(avg(record_count), 0)                         AS avg_records_per_file,
    min(record_count)                                   AS min_records,
    max(record_count)                                   AS max_records,
    sum(file_size_in_bytes)                             AS total_size_bytes,
    round(avg(file_size_in_bytes) / 1024.0 / 1024, 2)  AS avg_size_mb
FROM iceberg.demo."transactions$files"
GROUP BY file_format

Files table

This query provides a high-level overview of the average file size and the number of records stored in each Parquet file backing the transactions table.

As the output reveals, the average file size is only 1.85 MB, which is a strong indicator of a small-files problem.

These were just two examples demonstrating the power of Iceberg metadata tables. More details, along with the full list of exploration queries, are available in Jupyter notebook: https://github.com/vanikkars/iceberg-monitoring/blob/main/docker/jupyter/notebooks/trino_iceberg_metadata.ipynb, so feel free to check them out.

So far, we have explored these metadata queries in an ad-hoc manner. However, in production environments, you would typically want observability dashboards that expose these metrics through a clean UI

— so welcome to Grafana!

What Is Grafana?

For anyone who has not used it: Grafana is an open-source dashboarding tool. It supports wide range of sources including Prometheus, InfluxDB, SQL databases, or in our case Trino. Grafana renders the results as panels (time series, tables, gauges, stat tiles). It also has a templating system so you can parameterise dashboards with dropdown variables.

Dashboards

In our example Grafana dashboard we have high level metrics on top showing the aggregated totals per table:

• number of records
• number of commits
• avg file size
• time from the last commit

aggregated metrics per table

The Commit Activity section provides a historical overview of commits made to each table over time. In particular, it shows how the total number of rows grows over time and how frequently data is committed to the corresponding Iceberg table.

time retrospective view

Particularly interesting is the Files section, which shows aggregated file-level metrics (such as average file size), as well as a timeline illustrating how file sizes evolve over time.

File level metrics

Funny enough while I was finishing my writing the file size in transactions has decreased from 1.85 MB to 604 KB, stating even stronger the problem of small files :D

Summary

In this article, I tried to show which Iceberg table metrics are worth monitoring in your lakehouse in order to avoid future headaches related to small-file issues or snapshot bloat.

In the next article, I will dive deeper into the impact small files can have on your lakehouse and explore different strategies for solving this problem.

For more details and additional examples follow me on GitHub and Linkedin.

Thank you for reading all the way to the end!