A Bot Scan Almost 5×’d Our Metrics Bill in Minutes. One Label Was the Cause

Our active series went from ~100K to over a million on a quiet long weekend, and the Grafana bill climbed right along with it. Here’s what happened — and the fix I reach for first now.

Photo by Nicholas Cappello on Unsplash

Over a long weekend — when almost nothing in our lab should have been running — our active metric series had jumped from around 100,000 to over a million in a matter of minutes, and the forecasted bill had climbed almost 5×. No deploy. No traffic anyone planned. Just a flat green line that turned into a cliff.

I’d never heard of metric cardinality until it showed up on an invoice. If you’re running Prometheus, Grafana Alloy, or any self-managed observability stack, this is the failure mode that bit us. Here’s how I now spot it, why it happens, and the two-layer fix that keeps it bounded even when everything else is misconfigured.

The Situation: A Green Line That Became a Cliff

This happened during my time at the Apple Developer Institute for DevOps. Instead of toy labs, my cohort runs real, production-shaped infrastructure — a full observability stack with Prometheus and Grafana, and Grafana Alloy shipping metrics, logs, and traces off our hosts. That realism is the whole point. It’s also why a real bot found us.

One of our internal apps exposed a standard /metrics endpoint. Nothing unusual about it. Then, over a long weekend, the billable-series graph went vertical: ~100K to 1M+, a clean 10× in minutes. The cost forecast jumped roughly 385%. We were on Grafana Cloud at the time, so nothing crashed — the explosion just quietly turned into money.

None of us had touched it. And it was a long weekend — no other teams were in the lab, and there was essentially no organic traffic. Whatever real usage existed would have been a rounding error next to what the graph was showing.

What We Tried (and Missed)

My first instinct was the comforting one: maybe the app was just getting more use. Operational growth. Except it was a dead weekend — in hindsight, “normal growth” explained nothing.

A shared credential muddied things further. One API key was used across the whole cohort, so when the numbers moved, we couldn’t instantly attribute the spike to a single service or team.

A few of us did flag the strange graph. It got judged as within expected bounds, and with most people away for the long weekend, fewer eyes meant slower follow-up. That combination cost about 24 hours of uncontrolled ingestion before anyone treated it as an incident.

And we did have a billing alert. The catch is that a rising bill is a ticking time bomb: the alert can only fire once the cost has already been climbing for hours. It tells you an incident has been underway, not that one is about to start — by the time it pings, the damage is already hours deep. (Or maybe we just configured it wrong; I’m not ruling that out.) Either way, it confirmed the problem long after the meter had run.

The quiet lesson hiding in those dead ends: a billing alert verifies a problem, it doesn’t prevent one. By the time ours could fire, we were already a day late.

The Turning Point: It Was the path Label

When we finally dug in, the cause was almost insultingly simple.

A bot had run a high-volume vulnerability scan against the app — thousands of unique URLs probing for secrets and misconfigurations (/.ssh/authorized_keys.save, /admin/credentials.tar, /api/docker-compose.env, and on and on). Our app dutifully recorded each unique URL as a distinct path label value on its request metrics (http_request_duration_seconds_bucket, _count, and friends).

Here’s the concept I wish I’d understood beforehand. In Prometheus-style metrics, every unique combination of label values becomes its own time series — its own line the backend stores, indexes, and bills for. The count of those unique series is its **cardinality**.

A label with a small, fixed set of values is safe — HTTP status has maybe ten possibilities. But a label that can take unbounded values — a raw URL path, a user ID, a request ID — is a landmine. An attacker, or just a buggy client, can mint brand-new series faster than anyone can refresh a dashboard.

So the math was brutal:

one metric × scanned paths × bucket variants = a flood of new series.

Then 100K became 1M, and the meter kept running.

What Actually Fixed It

Before any real fix, we had to stop the bleed. Our team revoked the Grafana API token, which cut the app off from Grafana entirely — no token, no ingestion, so the meter stopped climbing. It’s a blunt instrument: that one shared key also fed our logs and traces, so revoking it left us flying blind while it was off. But when money is hemorrhaging — or, on a self-hosted box, when you’re racing an out-of-memory crash — stopping the flow first is the right call. It buys you the calm to fix the cause properly instead of in a panic.

Then came the actual fix, in two layers, and the order matters.

Layer 1 — fix it at the source. We changed the app to record the registered route pattern instead of the raw URL. If you do the same, both /items/abc123 and /items/xyz789 collapse into a single /items/:id, and anything that doesn't match a known route maps to one bucket like unmatched.

// BAD: raw path recorded
path="/items/abc123"  → new series
path="/items/xyz789"  → new series

// GOOD: registered route pattern recorded
path="/items/:id"     → one series for all ids
path="unmatched"      → one series for every unknown route

That one change bounds cardinality to the number of routes you actually defined — no matter how creative a scanner gets. To check it after deploying, count by (path) (http_request_duration_seconds_count) should return a couple dozen values, never thousands.

Layer 2 — put a firewall in the collector. I no longer trust an app to always behave. In Grafana Alloy, a relabel stage can strip dangerous label names outright (user_id, request_id, trace_id, email, ip…) and collapse anything that doesn't look like a real route into unmatched. I think of that relabel block as a cardinality firewall — the last line of defense before metrics leave the box.

I also learned to watch the leading indicator, not the lagging one. A metric like scrape_series_added flags the instant a single target mints a suspicious number of new series. A billing alert only screams after the money is spent; this fires the moment series start multiplying.

What I Learned (the part you can steal)

1. I now treat every metric label as untrusted input. If a value can be unbounded or attacker-controlled, it doesn’t belong in a label — normalize it or drop it.

2. Defense in depth beat any single fix. App-level route patterns and a collector-level relabel firewall meant one mistake couldn’t sink us. Either layer alone would have bounded this.

3. I alert on causes now, not symptoms. A billing alert is a lagging signal — it fires after the spend, once the problem has been running for hours. Series-growth alerts fire the moment series multiply, and they’re basically free. I’d rather be told the cause is starting than told the symptom has arrived.

4. A strange graph flagged early is still a signal. We had the warning roughly a day before we acted, and calling it “normal growth” was the most expensive decision in the whole timeline.

5. Wide-open access turned a billing problem into a breach question. Permissive security groups and a shared credential meant we couldn’t cleanly rule out a compromise. Least privilege isn’t paranoia — it’s what lets you sleep after an incident.

What I’d Do Differently — and What’s Next

If I could rewind: per-service budget alerts wired from day one, the shared credential swapped for something scoped per team, the security group locked down, and — honestly — speaking up louder the moment the graph looked wrong instead of waiting for someone to confirm I should worry.

The biggest change since: we’ve moved off Grafana Cloud to self-hosted Grafana. It sounds like a downgrade, but it flips the failure mode in our favor. On a managed cloud, a runaway cardinality event is invisible until the invoice catches up — the meter just keeps spinning, and the only thing that “breaks” is the bill. On our own box, the same event hits a wall: the machine runs out of resources and falls over. A capped, self-inflicted outage is something I can see instantly and recover from. A silently inflating bill is not. If this happens again, I’d much rather the box tip over than the price.

The encouraging part is that this is the kind of mess you only meet by running real infrastructure that real attackers can reach. That’s the whole point of the Apple Developer Institute for DevOps program.

If you found this story helpful, please give it a clap so it can reach others who need it.