Data Quality Solutions

When Green Means Nothing

The pipeline said success. The logs said success. The dashboard said success. It took a customer and a spreadsheet to tell us the truth.

There is a particular kind of confidence that comes from watching a job complete without errors. The status turns green, the counts check out, and somewhere in the room, someone exhales. We’re good. It is also one of the most dangerous feelings in data engineering.

This is a story about the week we learned that the hard way — not from an alert, not from a dashboard, but from a customer who opened a spreadsheet, compared two columns of numbers, and quietly sent us an email.

· · ·

The job that always runs fine

Our analytics ETL pipeline had logging, alerting, and input-output record count validation. What it did not have was a sense of memory. It knew what came in today and what went out today. If those numbers matched, it declared victory.

It had no idea what last month looked like. And that, as it turned out, was everything.

The upstream process had developed a quiet problem — not a crash, not an exception. Something subtler. A gap in the data that our validators were completely blind to, because our validators only ever asked one question: does the output match the input? It did. Perfectly. The input was already incomplete, and we faithfully delivered every incomplete record downstream, wrapped in a green success status.

· · ·

The email nobody wants to receive

A week passed. Then came a formal, measured message from the customer — challenging the “completeness” of the data. Record counts appeared lower than prior periods suggested they should be.

The team’s first reaction was not panic. It was disbelief. The job had run clean. Surely there was an explanation on the customer’s side.

There wasn’t. Engineers pulled CloudWatch logs, inspected Kafka offsets, traced the transformation logic end to end. Multiple people, multiple angles, same conclusion: by every metric the system knew how to measure, everything had worked.

Then someone did what the customer had already done — pulled the current month’s counts and placed them next to the previous months. No dashboards, no anomaly detection. Just numbers in a column.

The dip was obvious.

“A customer with Excel had caught something that a purpose-built data pipeline, with all its logging and cloud-native infrastructure, had completely missed.”

Not because the customer was smarter than the system. Because the customer was asking a different question. The pipeline had been built to validate process. It had never been built to validate reality — to ask whether the output made sense in the context of the world it was supposed to represent.

That kind of judgment requires history, seasonality, and the intuition that comes from knowing what normal looks like. Those things live in the business. They do not live in log files.

· · ·

What changed

The technical fix was straightforward. What took longer — and proved far more valuable — was the conversation that followed.

The team went to the customer and asked a question that should have been asked much earlier: How do you know when something is wrong? The answers came back in business terms — counts that move with seasons, ratios that hold steady across months, trends that any analyst would flag after thirty seconds with a pivot table. None of that thinking had ever made it into the data quality framework.

The checks that followedMonth-over-month record count comparisons. Seasonality-aware thresholds. Business-defined ratio checks between related data points. Simple, interpretable, and almost embarrassingly obvious in hindsight — but none of them would have been discovered without the conversation.

The customer who sent that email became, without quite intending to, one of the most important contributors to the reliability of the pipeline. They compared two columns of numbers and trusted what they saw. It is a kind of rigor no monitoring tool ships with out of the box — you have to go looking for it, which means going looking for the people who carry it.

The next time a job completes clean, ask yourself what question your validation didn’t ask. That’s where the real monitoring gap lives.