Green Dashboards, Broken Systems: From Circuit Breakers to Silent Data Corruption

Resilience engineering has a blind spot. Most of what we build protects against loud failures — the kind that errors out, times out, or trips a circuit. The failures that hurt the most don’t make any noise at all.

A perfectly healthy system, quietly broken

A daily sync job had been running for six months. Every run completed. Every dashboard turned green. Record counts were within their expected variance. No errors, no timeouts, no anomalies in any of the metrics the team had set up to watch.

Then someone in finance ran a reconciliation against the source system and found that roughly three percent of records had been silently dropped each day. Different records each time, depending on which ones happened to hit the bug. Six months of accumulation meant a meaningful number of customers whose records in the analytics warehouse no longer matched their actual state with the company.

There was nothing to find in the logs. The job hadn’t crashed. It had completed — just incorrectly.

This kind of failure has a particular flavor. It’s not the kind that pages you. It’s the kind that surfaces in a meeting, where someone says “this number doesn’t match what we expected” and you realize you don’t know how long it hasn’t matched, why, or how much else might be wrong that nobody has thought to check.

The dashboards were green the whole time.

The familiar resilience playbook

When teams talk about building resilient systems, the conversation tends to center on a familiar set of patterns: circuit breakers, retries with backoff, timeouts, bulkheads, rate limiting, graceful degradation. These are good patterns. They’ve been refined over decades of distributed systems work, and the literature on them is mature.

The mental model behind them is something like: bad things will happen, components will fail, traffic will spike, downstreams will get sick, so we need ways to detect those conditions and respond gracefully. Fail fast. Shed load. Give the sick service room to recover. Hystrix, Resilience4j, Polly, Envoy’s outlier detection, Istio’s destination rules — most of the tooling exists to make this category of resilience easier to implement.

Anyone who has operated systems at scale has seen this work. A downstream service gets overwhelmed; circuit breakers correctly trip; the calling service is spared from collapse. Retries with jitter prevent thundering herds. Timeouts keep slow callers from hogging thread pools. The patterns deliver, and there’s a reasonable temptation to think this is what resilience means — we’ve covered the bases.

There’s a quiet assumption baked into all of these patterns, though. They assume the failures we’re guarding against are observable. They assume that when something is going wrong, the system will produce signals — errors, timeouts, latency, status codes — that the resilience layer can react to.

But what about the failures that don’t produce any signal at all?

What circuit breakers actually need to work

A circuit breaker is, mechanically, a feedback control system. It watches a stream of outcomes — success, failure, latency — and uses those observations to decide whether to allow more traffic through or short-circuit it. The math varies by implementation, but the loop is the same: observe outcomes, compute a failure rate, compare to a threshold, transition states.

The whole apparatus depends on outcomes being categorizable as success or failure. HTTP 500 is failure. Connection refused is failure. Timeout is failure. HTTP 200 with a normal response time is success. The breaker counts these categorized outcomes and acts on the count.

This works when failures are loud. A backend that has crashed returns connection errors. A backend that’s overloaded times out. A backend that’s been deployed broken throws 500s. These are exactly the conditions circuit breakers exist for.

But what happens when the backend returns HTTP 200 with normal response time, and the response is wrong?

The circuit breaker counts it as a success. It has no choice — by every signal it has access to, the call succeeded. The breaker stays closed, traffic flows, and wrong responses keep going to callers, who consume them as if they were correct.

This isn’t a defect in circuit breakers. They were never designed to verify semantic correctness. They were designed to detect operational failure, and they do that well. The problem is that “operationally healthy” and “doing the right thing” are not the same condition, and we sometimes mistake one for the other.

The territory beyond loud failures

Once you start looking, the category of failures that produce no error signal turns out to be surprisingly large. A few patterns recur:

Data drift in integrations. A daily sync pulls from an upstream system. The upstream changes a field’s semantics — what was an absolute count becomes a delta, or what was a string ID starts being numeric. The sync continues to “work”: records are pulled, written, no errors. The data is now subtly wrong, in ways that take weeks or months to surface.

Partial completion. A multi-step process where step one writes to the database, step two emits an event, step three triggers an email. Step one and three are healthy; step two is silently dropping events because of a misconfigured permission. From the outside, every order looks healthy in the database. Downstream systems that depend on the events are missing data they should have received. The breakage is invisible because no single component sees the whole flow.

Conditional logic blind spots. A confirmation email sends correctly for 99.7% of orders. The 0.3% that fails has a specific edge case — orders with promotional discounts above a certain threshold, say. The email service is healthy by every metric. The orders system is healthy. The bug lives in the if-statement that decides whether to send, and it’s only ever wrong for that small slice of cases. Customer support eventually surfaces it after enough complaints accumulate.

Timestamp and unit corruption. A code change starts returning timestamps in the wrong timezone, or amounts in cents instead of dollars. The data flows. The dashboards render. The numbers are wrong. Nothing alerts because nothing is broken in any way the system can recognize.

Authorization edge cases. A change to permission logic introduces a case where one tenant’s users can occasionally see another tenant’s data. The code returns successful responses. The audit logs show successful API calls. There’s no error to detect. The problem is that the content of those successful responses contains the wrong data.

Audit trail gaps. Logs flow into the centralized logging system. Volume looks normal. But a recent refactor changed a field name, so events that used to log with user_id are now logging with userId, and the alerting that filters on user_id no longer matches them. The logs are technically being collected. They're just no longer findable by the queries that depend on them.

In each case, the system passes every health check it has. The instruments report green. The circuit breakers stay closed. And the system is wrong.

Why silent failures are worse than loud ones

There’s a temptation to treat these silent failures as just another category of bug — annoying, but not categorically different from regular software defects. In practice, they have several properties that make them more dangerous than loud failures.

They compound. A loud failure stops accumulating damage the moment it’s detected. You find it quickly, fix it, lose maybe a day. A silent failure accumulates damage for as long as it goes undetected. Six months of subtly wrong data is a categorically different problem from a six-hour outage. Total damage scales with detection latency, and detection latency for silent failures is measured in weeks or months.

Recovery is harder. When a service crashes, you restart it. When data has been silently corrupted for six months, you have to figure out which records are affected, what the correct values should have been, whether downstream systems have made decisions based on the bad data, and how to repair the damage without making things worse. There’s often no clean rollback. The repair process can introduce new errors.

Trust degrades non-linearly. When a system goes down loudly and recovers, customers might be annoyed but the trust relationship is intact — the system had a problem, the team fixed it. When customers discover that the system has been silently wrong for months, the trust damage is qualitatively different. What else has been wrong that we haven’t noticed? That question, once asked, is hard to fully answer.

They’re often discovered by outsiders. Loud failures are usually discovered by the team running the system, who have the context to fix them. Silent failures are often discovered by auditors, regulators, customers, or finance teams running their own reconciliations. Losing narrative control — having someone else tell you that your system has been wrong — makes the situation harder to manage.

They erode the validity of all metrics. Once you discover that one set of numbers in your system was silently wrong, every other set of numbers becomes suspect. Reports that were taken as ground truth become provisional. Decisions made based on those reports come into question. The blast radius extends well beyond the immediate technical problem.

The asymmetry matters. A loud failure is a bounded incident. A silent failure is an unbounded, ongoing condition that you might still be in.

The shared root

Circuit breakers and silent corruption sound like unrelated topics. One is a resilience pattern; the other is a data integrity problem. They’re actually two faces of the same underlying question: how does the system know whether it’s doing the right thing?

Circuit breakers answer one slice of that question — the slice where “doing the right thing” can be inferred from operational signals like response codes and latencies. They cover the case where wrong looks like failed.

Silent corruption is what lives in the other slice — where “doing the right thing” can’t be inferred from operational signals, because the system can be wrong while operationally succeeding. Wrong looks like fine.

Both slices need defenses, but the defenses look different. Circuit breakers and their cousins (retries, timeouts, bulkheads) work for the first slice because they can react to the signals the system produces. The second slice requires something else: explicit verification of business correctness, separate from the system’s normal operation. Reconciliation. Invariant checking. Cross-source comparison. End-to-end auditing.

Most teams have built up substantial muscle on the first slice and very little on the second. The first has frameworks, libraries, conferences, and a vocabulary. The second has scattered practices that go by different names — data observability, reconciliation, audit, integrity monitoring — and rarely gets discussed as a coherent discipline.

The result is that resilience efforts disproportionately cover loud failures and leave silent ones largely undefended. A team can have meticulous circuit breaker configuration, perfect retry policies, comprehensive bulkhead isolation — and still discover, six months in, that their daily sync has been quietly losing 3% of records the whole time.

A more honest definition of resilience

Resilience isn’t just staying up. It’s staying correct. A system that’s up but producing wrong outputs isn’t resilient; it’s just operationally unfailed. The customer who gets a wrong account balance doesn’t care that your circuit breakers prevented a cascade — they care that your system told them something untrue.

This redefinition matters because it changes what counts as a complete answer to “is our system resilient?” Under the old definition, the answer is: do we have circuit breakers, retries, timeouts, autoscaling, redundancy? Under the new definition, those are necessary but insufficient. The additional question is: do we know whether the system is producing correct outputs, beyond the absence of error signals?

Most teams can’t answer that question with confidence. They have monitoring that would tell them if the system crashed. They don’t have monitoring that would tell them if the system was silently giving wrong answers.

This isn’t because they’re sloppy. Monitoring for correctness requires a kind of work that monitoring for availability doesn’t: you have to define, in domain terms, what “correct” means. You have to articulate the invariants the system is supposed to maintain. You have to build verification that’s independent of the system’s normal operation. None of this is hard in isolation, but it requires explicit effort that doesn’t get prioritized unless someone insists on it.

The tier-0 mindset

The temptation when confronted with the silent failure problem is to try to monitor everything. This doesn’t work, for the same reason that trying to test everything doesn’t work: the surface area is too large, the cost is too high, and the alert noise becomes counterproductive.

The more useful framing is to triage. Not all correctness is equal. Some things in your system, if they go silently wrong, would be catastrophic. Other things would be merely annoying. The discipline is identifying which is which, and investing accordingly.

For most systems, there are only a handful of truly critical invariants — things that cannot be silently wrong without serious consequences. For a financial system: money conservation. Every dollar accounted for, balances reconciling, no creation or destruction of value through software bugs. For a healthcare system: patient identity binding to clinical data. The right record attached to the right person, every time. For an authentication system: that revoked credentials are actually rejected. For an e-commerce system: that orders placed result in fulfillment and that inventory commitments match physical inventory.

These are the tier-0 invariants. They deserve overwhelming, redundant, paranoid monitoring. The investment in their verification should be disproportionate to the rest of your system, because the cost of a silent violation is also disproportionate.

Below tier-0 are second-tier invariants — important, worth checking, but not catastrophic if occasionally violated. Below those is the broad mass of system behaviors that are good to verify but not worth heavy investment.

The mistake teams make is not making this distinction. They either try to monitor uniformly (and end up with thin coverage everywhere) or they don’t think about correctness monitoring at all and rely on operational health checks. Both approaches leave the catastrophic failures undefended.

A practical framework

If this resonates and you want to do something concrete about it, here’s an exercise that takes maybe a half-day and produces unusual clarity.

Step 1: List the invariants. Get a few people in a room — engineering, product, and someone from the business side who understands the consequences of things going wrong. Ask: if this were silently violated for a year, how bad would it be? Write down everything that comes up. Don’t filter yet.

A finance system might produce a list like: every transaction is fully accounted for in the double-entry ledger; account balances at end of day match the sum of beginning balance and net transactions; no transaction is processed twice; external settlement totals match internal totals.

An e-commerce system might produce: every paid order results in a fulfillment record within 24 hours; inventory committed equals inventory shipped plus inventory in flight; every order generates exactly one confirmation email; no customer can access another customer’s order history.

The exercise of writing these down is itself surprisingly hard. If you struggle to articulate an invariant clearly, you can’t enforce or verify it either — and that’s diagnostic.

Step 2: Tier them. For each invariant, ask: if this were silently wrong for six months, what’s the worst-case business consequence? Sort into tiers:

• Tier 0: Catastrophic. Existential business risk, regulatory exposure, irrecoverable customer trust damage.
• Tier 1: Serious. Significant remediation cost, customer impact, but survivable.
• Tier 2: Important. Worth catching, manageable consequences.
• Tier 3: Nice to have.

For most systems, tier 0 will have between one and five invariants. That’s normal and intended. If you find yourself with twenty tier-0 invariants, you haven’t really triaged — almost nothing is actually catastrophic in the way that, say, “the bank’s books don’t balance” is catastrophic.

Step 3: Design verification for tier-0 invariants. For each tier-0 invariant, design at least one — ideally several — independent verification mechanisms. The principle is defense in depth: each layer catches what the others miss.

A typical layered approach for a financial invariant might look like: in-database constraints that prevent imbalanced transactions from being written; real-time queries that check system-wide totals continuously; daily reconciliation against external sources of truth; periodic shadow accounting using independent code paths; external audit on a regular cadence. Each layer has different blind spots, so collectively they catch much more than any single layer could.

For a less critical invariant, even a single daily reconciliation job with alerting is a major improvement over nothing.

Step 4: Make violations real incidents. When a tier-0 invariant violation is detected — even if no customer noticed, even if the impact was small — treat it as a P0 incident. Page someone, run the incident process, do a postmortem. The goal is to build organizational muscle around taking these violations seriously, because the cultural pattern of “data integrity issues are just data team problems” is exactly how silent failures get normalized.

Step 5: Track invariant coverage as part of your engineering process. When new features ship, ask: what invariants does this feature introduce or affect? Are they monitored? Make this part of definition-of-done, the same way tests are. Without this discipline, invariant coverage rots — every new feature potentially adds new ways to be silently wrong, and unless someone is explicitly thinking about it, the gaps grow.

This isn’t a heavy framework. A few hours of explicit thinking, followed by focused engineering work on the highest-priority invariants. It produces something most teams don’t have: a defensible answer to “how do we know our system is doing the right thing?”

Closing thought

The resilience patterns refined over the last fifteen years — circuit breakers, retries, bulkheads, all of it — are excellent tools for one half of the problem. They protect against failures that announce themselves. They keep your system standing when downstreams collapse, when traffic spikes, when networks misbehave.

The other half of the problem is quieter. It’s the failures that don’t announce themselves, that pass every operational health check, that compound in the background while dashboards stay green. These failures aren’t rarer than loud ones — if anything, they’re more common — they’re just harder to see.

A complete resilience practice covers both halves. Circuit breakers for when failures are loud. Invariant monitoring for when failures are silent. The first protects against the obvious disasters. The second protects against the disasters you don’t realize you’re already in.

The discipline isn’t really about adding more tools. It’s a shift in mindset: from “is the system running?” to “is the system correct?” Those are different questions, and most monitoring answers only the first. The second takes more work to answer, but answering it is what separates systems that occasionally surprise you with quiet six-month failures from systems that don’t.

It’s worth asking, of whatever you’re working on right now: if it were silently wrong, would you know?