Observability Is a Data Problem

How data-centric observability, distributed tracing, and table-level state tracking redefine reliability in analytical systems

Despite the sophistication of modern data stacks, debugging data pipelines in production still feels disturbingly primitive.

When something goes wrong, the questions are always the same:

• Why is this table incomplete?
• Which job introduced this corruption?
• Why did freshness degrade without any job failing?
• Can we prove this dataset is correct?

And the answers are usually unsatisfying:

• Airflow says the DAG succeeded
• Spark metrics look “mostly fine”
• Logs exist, somewhere
• The data, however, is wrong

This gap exists because most observability practices in data engineering are execution-centric, while data systems fail at the semantic and state level.

In this article, we will argue that: Observability in data pipelines must be modeled around data state transitions, not jobs.

We will explore how OpenTelemetry and Apache Iceberg, when combined intentionally, allow us to build deep, data-native observability, not dashboards, but explainability.

The Fundamental Mismatch: Jobs vs Data

Execution Success != Data Correctness

Schedulers, orchestrators and compute engines tell us whether code ran. They do not tell us whether the resulting data is:

• Complete
• Consistent
• Fresh
• Semantically valid
• Regressed compared to previous states

A Spark job can succeed and still:

• Write fewer rows than expected
• Overwrite valid partitions
• Introduce silent schema drift
• Corrupt late-arriving data

Execution observability is necessary, but structurally insufficient.

Observability Reframed: Data State as the Primary Signal

A modern data platform should be observable across three layers:

| Layer      | What It Answers                 |
| ---------- | ------------------------------- |
| Execution  | What ran and where did it fail? |
| Data State | What changed in the dataset?    |
| Semantics  | What does this change mean?     |

Most platforms stop at the first layer.

The combination of OpenTelemetry and Iceberg allows us to instrument all three — coherently.

The Key Design Decision: What Is a Trace?

Most data implementations make a fatal modeling choice:

One job = one trace

This creates isolated telemetry islands.

Instead, traces should represent logical data movements, such as:

• A batch ingestion window
• A streaming watermark interval
• A table snapshot lifecycle
• A business event aggregation cycle

Think of a trace as:

“Everything that happened to this slice of data.”

OpenTelemetry as a Causal Graph for Data Systems

Traces Are Not Just for Microservices

OpenTelemetry is often perceived as a microservices tool. That’s a mistake.

At its core, OpenTelemetry provides:

• Distributed causal graphs (traces)
• Context propagation
• High-cardinality metadata
• Vendor-neutral telemetry

These properties map perfectly to data systems — if we stop modeling traces as “requests”.

Modeling Data Pipelines as Distributed Transactions

Consider a daily batch pipeline:

1. Kafka ingestion
2. Spark transformation
3. Iceberg commit
4. dbt model refresh
5. Downstream materialization

This is not five unrelated jobs. It is a single distributed transaction on data state.

A Correct Trace Model

Trace: sales_pipeline_2025_01_15
 ├── kafka.consume (partition=12)
 ├── spark.transform (dataset=sales_raw)
 ├── iceberg.commit (snapshot=839201)
 ├── dbt.model (sales_daily)
 ├── freshness.check

Failures now have causal meaning, not just timestamps.

Apache Iceberg: Observability Hidden in Plain Sight

Iceberg is often introduced as:

• A table format
• A lakehouse enabler
• A solution to small files

That framing severely undersells it.

Iceberg is a transactional log of data state transitions.

Which makes it an observability goldmine.

Snapshots as First-Class Observability Events

Every Iceberg commit produces:

• A snapshot ID
• A parent snapshot
• A timestamp
• An operation type
• A manifest of data files

This is perfect observability material.

Yet most platforms ignore it.

Correlating Iceberg Commits with OpenTelemetry Spans

The moment where data actually changes is the Iceberg commit.

That commit should emit a span.

Example Span Attributes

{
  "span.name": "iceberg.commit",
  "trace_id": "data_batch_2025_01_15",
  "attributes": {
    "iceberg.table": "sales",
    "iceberg.snapshot_id": 839201,
    "iceberg.parent_snapshot_id": 839155,
    "iceberg.operation": "overwrite",
    "rows_added": 124039,
    "rows_deleted": 0
  }
}

Now you can answer questions like:

• Which execution produced this snapshot?
• What changed compared to the previous state?
• Which downstream models consumed it?

Time Travel as a Debugging Primitive

Iceberg’s time travel is often pitched for analytics.

Its real power is debugging.

Metrics That Describe Data, Not Machines

Stop Measuring Jobs. Start Measuring Datasets.

Traditional metrics:

spark_job_duration_seconds
cpu_usage
executor_failures

These are operationally useful, but semantically empty.

Data-centric metrics look like:

data_freshness_seconds{table="customers"}
rows_written{table="sales", pipeline="daily"}
snapshot_commit_latency{table="orders"}

These metrics can be derived directly from:

• Iceberg metadata tables
• OpenTelemetry spans
• Trace context

Logs Without Context Are Noise

Most data platforms still produce logs like this:

ERROR Failed to write partition 2025-01-15

This is operationally useless.

Logs Should Be Trace-Aware

Every log line should include:

• trace_id
• span_id
• dataset
• snapshot_id (if applicable)

Example:

{
  "level": "ERROR",
  "trace_id": "data_batch_2025_01_15",
  "span_id": "iceberg_commit",
  "snapshot_id": 839201,
  "message": "Row count mismatch: expected >= 120k, wrote 98k"
}

Now logs become navigable explanations, not forensic debris.

A Deterministic Debugging Workflow

1. Detect a data quality regression
2. Identify the first bad snapshot
3. Inspect snapshot metadata
4. Correlate snapshot → trace → job
5. Roll back safely
6. Reprocess deterministically

This turns data incidents from:

“Let’s rerun and hope” into “We know exactly what happened.”

Conclusions

The future of data platforms is not defined by faster engines or cheaper storage.

It is defined by:

• Explainability
• Determinism
• Trust

OpenTelemetry gives us the language of causality. Apache Iceberg gives us the memory of data state.

Together, they allow us to build data systems that can finally answer the hardest question in analytics:

“Why is this data the way it is?”

Common Achitecture

A production-grade setup typically includes:

• Instrumented compute (Spark, Flink, dbt)
• OpenTelemetry SDKs + Collector
• Trace backend (Tempo, Jaeger)
• Metrics backend (Prometheus)
• Iceberg catalog (REST / HMS / Glue)
• Metadata enrichment layer

Observability emerges from correlation, not tooling.

Common Anti-Patterns

• Instrumenting only Airflow → No visibility into data state
• Treating Iceberg as storage only → Lost lineage and causality
• Job-level SLAs instead of data SLAs → False confidence
• Low-cardinality metrics → No dataset-level insight

🙌 Found this helpful?

If you enjoyed the article or found it useful, feel free to give it a few claps, it helps more people discover it on Medium.

Also, feel free to share it with others who might benefit.

I’d love to hear your thoughts, any feedback, questions or suggestions are more than welcome in the comments!

Thanks for reading!