Your 90% Code Coverage Is the Most Expensive Vanity Metric

The coverage report is not wrong. It is answering a question you never asked.

A sleek visual reminder that high code coverage can look green while hidden bugs quietly grow beneath the dashboard.

The coverage dashboard finally hit ninety percent.

Someone dropped a screenshot in the team channel. A long stretch of effort, every pull request gated on the number, every merge nudging it up a fraction. And over that same stretch, the production bug rate had been climbing.

Nobody connected the two at first. Why would they?

The Claim, Without the Hedging

Here is what I think, stated plainly: code coverage is the most expensive vanity metric most engineering teams pay for.

Not because coverage is meaningless. Because the moment a team turns coverage into a target, it stops measuring quality and starts measuring compliance. Those are different things, and the gap between them is exactly where bugs live.

This is Goodhart’s law wearing an engineering badge. When a measure becomes a target, it stops being a good measure. Coverage is unusually easy to game, which makes it unusually dangerous as a goal.

The rest of this piece is the part where I earn that claim and then argue against myself.

Coverage Measures Execution, Not Verification

A coverage tool answers one question. Which lines of code ran while the tests executed.

That is all it answers. It does not ask whether anything was checked when those lines ran. You can write a test that calls a function, runs every branch inside it, and asserts nothing of consequence. Coverage goes up. Confidence should not.

In many teams chasing a number, this is the first thing that quietly happens. Tests get written to touch code, not to pin down behaviour. The suite turns green. The report turns gold. And the defects that ship are in the assertions nobody wrote, on the lines the report already counted as covered.

Coverage tells you which lines ran. It tells you nothing about whether anything was checked when they did.

The Number Pulls Tests Toward Implementation Detail

Reaching the last stubborn branches to push a number from eighty to ninety is where the real cost hides.

Those branches are usually hard to reach on purpose. They are error paths, defensive guards, internal helpers. To cover them, engineers reach inward, testing private methods, mocking internal calls, asserting on how the code is structured rather than what it does. The tests pass. The number climbs.

Then someone refactors. The behaviour does not change at all, but half the suite turns red, because the tests were welded to the old shape of the code. The team learns, correctly, that a red suite usually means a refactor and not a regression. You have probably seen what comes next. People stop reading the failures. A brittle suite is a suite nobody trusts, and an untrusted suite is worse than a small one.

Green CI Can Be Hiding the Tests That Matter

The reported percentage hides its own distribution.

Ninety percent coverage can be ninety percent fast trivial unit tests with almost nothing exercising the risky seams where systems actually integrate. The number looks identical either way. It cannot tell you that the payment path or the auth boundary is the thin part.

There is a sharper version of this. When a test turns flaky, the common fix under deadline is to quarantine it, mark it skipped or expected to fail in continuous integration (CI), the automated pipeline that runs your tests on every change. The line still counts toward coverage in many setups. The assertion no longer runs. The dashboard stays green precisely because the test that would have caught the regression has been switched off. That is the shape of a climbing bug rate sitting underneath a healthy looking number.

Where the Believers Are Right

Now the honest part, because a coverage advocate is not a fool.

Coverage is a genuinely good signal when you read it as a floor rather than a ceiling. A module sitting at ten percent is a real warning, and you do not need a deeper metric to know it. Coverage delta on a pull request is one of the cheapest ways to catch new code that arrived with no tests at all. That alone earns its place in review.

And in safety critical work, coverage is not vanity, it is law. Standards like DO-178C, the certification standard for avionics software, mandate strict forms such as MC/DC (modified condition decision coverage), and meeting them is the job, not a vanity exercise. If you are shipping flight control or medical device firmware, ignore everything above and follow your standard.

If your context is a regulated floor or an early warning signal, then coverage as a metric makes complete sense. The failure is not the metric. The failure is the target.

What To Track Instead

Stop gating on a single coverage percentage above a low floor, and start watching two things that correlate with defects far better.

The first is test distribution, the test pyramid. A healthy suite is mostly fast focused tests at the base, fewer integration tests in the middle, and a thin layer of full system tests at the top. When that shape inverts, slow brittle tests on top and hollow units below, bug rate tends to rise no matter what the coverage number says. The shape predicts pain better than the percentage does.

The second, for your highest risk modules only, is mutation testing. A mutation tool deliberately introduces small faults into your code, flips a comparison, deletes a line, and checks whether your tests notice. If your tests still pass with a bug injected, those tests were theatre. Mutation score measures whether your assertions actually assert.

The trade-off is real. Mutation testing is slow and expensive to run across a whole codebase, so scope it to the parts where a defect costs the most. Pyramid discipline costs review attention that a percentage gate lets you skip. Neither is free. Both measure something true.

The Action Worth Taking

In your next review, pick one new test and ask out loud what behaviour breaks if you delete its strongest assertion. If the answer is nothing, you found a covered line that verifies nothing, and you found it before your customers did. The sentence to bring to your team is this one: “I do not care that this line is covered, I care what fails if it is wrong.” This works best when you also lower or remove the hard coverage gate, because the gate is what was rewarding the empty test in the first place. If you are in a regulated context, keep your mandated coverage and add the assertion question on top.

Back to the Dashboard

The screenshot in the channel said ninety percent, and it was true, and it was not the thing anyone actually wanted to know.

A team that makes this shift does not chase a smaller number. It stops treating one number as the answer. The dashboard still shows coverage, sitting there as a signal instead of a scoreboard, while the conversation moves to whether the tests would survive a real bug. The celebration gets quieter and the on call rotation gets calmer, which is the trade most teams would take if anyone framed it that way.

What is your suite actually proving on the day a real regression walks in?

Follow me for more such articles, where I take this same lens to the other numbers engineering teams trust without checking what they measure, velocity, uptime, story points, the dashboards that stay green while something quietly drifts underneath. If you would rather each piece reach you directly than hope the feed surfaces it, the email list is the better place for that.

Here is what I would leave you with. If your coverage number could read ninety percent and be lying to you the whole time, which of the other green dashboards on your wall has actually earned the trust you give it?