Master PromQL | Part 3 of 7

Why Your Prometheus Graphs Are Lying to You

Counters don’t measure rates. Gauges don’t need rate(). And irate() isn't as real-time as you think.

Your HTTP request counter just hit 847,293. Ten seconds later? 847,301.

Congrats, you just watched a counter do exactly what it’s supposed to do. Exciting, right?

Now, try to graph requests per second from that data.

• Plot the raw counter → a steadily climbing line that tells you… nothing.
• Divide by time naively → hello, spikes at every counter reset.
• Use irate() blindly → it looks responsive, but hides the truth during scrape gaps.

There is a right way to do this.

In this Part, I’ll show you exactly how and, more importantly, why it works.

Counters and Gauges: The Two Metric Types You Must Distinguish

Before you write rate(), you need to understand which metric types exist and why only one of them makes sense with rate functions.

What Exactly Is a Counter?

A counter is one of the simplest metric types you’ll encounter.

It has one job: go up.

Every time an event happens, the counter increments.

A request is served? +1.

An error occurs? +1.

A few more bytes are sent over the network? The counter increases again.

The value keeps growing forever — unless the process restarts, in which case it resets back to zero and starts counting again.

Some common examples:

• http_requests_total → Total requests served since the application started
• http_request_duration_seconds_count → Total number of recorded request durations
• node_network_receive_bytes_total → Total bytes received since the machine booted
• process_cpu_seconds_total → Total CPU time consumed by the process

So far, so good.

But here’s the catch:

A raw counter value is almost useless when you’re trying to understand what’s happening right now.

Imagine seeing this number on a dashboard:

847,301

What does it tell you?

Not much.

Is your service handling 10 requests per second?

1,000 requests per second?

10,000 requests per second?

You have no idea.

A counter only tells you how much has happened since the beginning.

To understand current activity, you need something more important:

How fast the counter is changing.

In other words, you need the rate.

Gauges: Metrics That Tell You What’s Happening Right Now

Unlike counters, gauges aren’t trying to keep a running total.

They’re taking a snapshot.

A gauge represents the current state of something, which means it can move in either direction.

Memory usage goes up.

Then it goes down.

The number of ready pods increases after a deployment.

Then drops when a node fails.

That’s exactly what gauges are designed to capture.

Some common examples:

• node_memory_MemAvailable_bytes → Current available system memory
• container_memory_working_set_bytes → Current memory usage of a container
• kube_deployment_status_replicas_ready → Number of replicas currently ready
• node_load1 → Current 1-minute load average

The key difference from counters is simple:

A gauge’s current value is already meaningful.

If a memory usage graph shows 8 GB, you immediately know how much memory is being used.

If a deployment shows 3 ready replicas, you know its current state.

No extra math required.

And that’s where many people make a costly mistake.

They learn about rate() for counters and start applying it everywhere.

Including gauges.

Almost every time, that’s wrong.

Why?

Because gauges already represent the current measurement.

Applying rate() doesn't tell you the current value but it tells you how quickly that value is changing, which is usually not what you're trying to measure.

For gauges, the value itself is often the answer.

Why You Can’t Just Divide a Counter by Time

It seems simple, right?

If you saw 1,000 requests in 5 minutes, you might think:

1,000 ÷ 300 seconds = 3.33 requests/sec

Easy. Problem solved.

Not so fast.

The Counter Reset Problem

Counters aren’t perfect. They reset to zero whenever a process restarts.

Imagine this:

• Counter at 847,000
• Process restarts → counter goes to 0
• Next scrape shows 1,000

Do you really think dividing by time will give the right rate?

Nope. You’ll see either:

• A negative rate
• A massive spike

Neither reflects reality.

The Sampling Problem

There’s more. Your data isn’t continuous but it comes in discrete scrape intervals.

Dividing by the time between two samples doesn’t account for:

• Scrape alignment
• Missed scrapes
• Uneven intervals

This makes naive division unreliable.

How Prometheus Fixes It

This is why Prometheus has the rate() function.

• Detects counter resets automatically
• Extrapolates a per-second rate across all samples in the window
• Gives you an accurate, smooth view of real traffic

Bottom line: rate() exists for a reason. Dividing counters by time won’t cut it.

rate()The Standard Tool for Counters

When you want to know how fast something is happening, rate() is your go-to.

It calculates the average per-second increase of a counter over a specific time window.

# Requests per second, averaged over the last 5 minutes
rate(http_requests_total[5m])

What Actually Happens Behind the Scenes

Prometheus doesn’t just divide numbers. It:

1. Gathers all the http_requests_total samples from the last 5 minutes
2. Detects and accounts for any counter resets
3. Extrapolates a per-second rate across the entire window

The result? A smooth, averaged rate.

Even if traffic spiked 30 seconds ago and dropped, rate(...[5m]) still reflects that spike—just dampened by the averaging window.

When to Use rate()

• Dashboards tracking traffic trends over time
• Alerts on sustained error rates (you want stable averages, not noise)
• Anywhere you need a representative, smoothed rate

Picking the Right Window

The [5m] in rate(...[5m]) defines how much history you average.

• Short windows ([1m]) → respond quickly, but noisy
• Long windows ([10m]) → smooth, but slower to react

Practical guideline from the community:

• Minimum: 2× your scrape interval (to ensure at least 2 samples)
• Better: 4× your scrape interval → more samples, smoother rate

Examples:

• 1-minute scrape → [4m] is a solid default
• 15-second scrape → [1m] works well

These aren’t hard rules — adjust based on how much smoothing your dashboards need.

irate() — The Instantaneous Rate

irate() calculates the instantaneous per-second rate using only the last two samples in the window.

# Requests per second, calculated from the last two scrapes only
irate(http_requests_total[5m])

Despite the [5m] range window, irate() only ever uses the two most recent data points within that window. The window's only purpose is to bound how far back it looks for those two points, it doesn't average across the window.

This gives you a very different result from rate():

• More responsive: reacts immediately to traffic spikes
• More volatile: a single unusual scrape interval produces a spike in the graph
• Degrades with scrape gaps: if one scrape is missed, irate() spans two scrape intervals instead of one; the calculated rate is still approximately correct (larger delta over larger interval), but you lose instantaneous resolution exactly the thing irate() is supposed to provide

When to use irate():

• High-resolution graphs where you need to see spikes clearly
• Short-lived metrics where averaging would mask the signal
• When you know your scrape interval is consistent and gaps are rare

When NOT to use irate():

• Alerting rules — a single missed scrape can cause a false positive or false negative
• Long-term trend dashboards — the volatility makes them unreadable
• Any environment with unreliable scrape timing

increase() When You Need the Actual Count

Sometimes you don’t care about requests per second.

You just want to know:

“How many requests did we receive?”

That’s exactly what increase() is for.

Instead of returning a rate, it returns the total increase of a counter over a time window.

# Total requests received in the last hour
increase(http_requests_total[1h])

The result is a count, not a rate.

If your service handled 250,000 requests during the last hour, increase() returns approximately 250,000.

Simple.

How Is It Different From rate()?

Under the hood, they’re closely related.

In fact, you can think of increase() as:

increase() ≈ rate() × time

For example:

increase(http_requests_total[1h])
≈
rate(http_requests_total[1h]) * 3600

Both functions:

• Handle counter resets automatically
• Work with the same underlying counter logic
• Use multiple samples within the selected range

The difference is in the output.

• rate() answers: "How fast?"
• increase() answers: "How many?"

When Should You Use increase()?

Whenever the total count matters more than the rate.

Common examples include:

• Total requests during the last hour
• Total errors during the last day
• SLO calculations over weeks or months
• Billing dashboards
• Audit and compliance reporting

If someone asks, “How many?”, increase() is usually the right tool.

The Gotcha Most People Miss

There’s one subtle detail that surprises a lot of engineers.

increase() doesn't simply subtract the first value from the last value.

It extrapolates.

That means Prometheus estimates what happened across the entire range window, even if it doesn’t have data covering every second of that window.

Imagine you run:

increase(http_requests_total[1h])

But Prometheus only has 45 minutes of samples available.

Instead of returning a count for those 45 minutes, it scales the result to estimate what the full hour would have looked like.

That’s usually what you want for monitoring.

But it also means the result is an estimate, not an exact accounting figure.

The Rule of Thumb

Use increase() for monitoring, reporting, and trend analysis.

But if you’re building systems where every event must be counted precisely billing, audits, financial reporting, or compliance — you’ll want a more authoritative source, such as:

• Recording rules that persist calculated values
• Long-term storage systems
• Event logs or transactional databases

For operational visibility, increase() is excellent.

For exact accounting, treat it as an approximation.

The Decision Framework

Use this as your mental checklist every time you need to query a counter:

Is this metric a counter (only goes up) or a gauge (can go up/down)?
│
├── Gauge → query the raw value directly, no rate function needed
│          e.g.: node_memory_MemAvailable_bytes
│
└── Counter → you need a rate function
           │
           ├── Do I need a smooth trend for dashboards or alerting?
           │   └── Use rate()    e.g.: rate(http_requests_total[5m])
           │
           ├── Do I need to see instantaneous spikes clearly?
           │   └── Use irate()   e.g.: irate(http_requests_total[5m])
           │
           └── Do I need a total count over a period?
               └── Use increase()  e.g.: increase(http_requests_total[1h])

The most important branch: don’t apply rate() or irate() to a gauge. It produces results that look numeric but are meaningless. Memory usage doesn't have a "rate of increase per second" in the same sense. what you want is the current value or maybe a delta, not a rate.

Counter Resets: What Actually Happens

When a Prometheus-instrumented process restarts, its in-process counters reset to zero. The next scrape will show a value much lower than the previous one.

Without reset detection, calculating the rate across a restart would give you a massive negative delta (or a nonsensical large rate). rate() and irate() both detect resets automatically and adjust for them, Prometheus documents this behaviour without specifying the internal algorithm.

Here’s what that looks like in practice:

Scrape 1: http_requests_total = 50,000
Scrape 2: http_requests_total = 51,200   ← normal, delta = 1,200
Scrape 3: http_requests_total = 52,400   ← normal, delta = 1,200
Scrape 4: http_requests_total = 320      ← RESET (process restarted)
Scrape 5: http_requests_total = 1,520    ← normal post-restart, delta = 1,200

When rate() encounters the drop from 52,400 to 320 between scrapes 3 and 4, it automatically adjusts for the reset. The result is a rate that correctly reflects post-restart traffic not a spike caused by the discontinuity.

Prometheus handles this automatically, you don’t need to write reset-detection logic yourself. The behaviour is documented; the exact internal mechanism isn’t something you need to reason about when writing queries.

This is precise enough for dashboards and most alerting. For exact accounting across restarts, you’d need a persistent counter mechanism outside of Prometheus.

Debug Angle: Spikes and Misleading Graphs

Spike at process restart

Symptom: A massive spike in your rate graph exactly when a service restarted. Cause: You’re using a function or expression that doesn’t handle counter resets — for example, dividing raw counter values directly. Fix: Use rate() instead. It handles resets correctly.

Unexpected flatness or coarse spikes with irate()

Symptom: Your irate() graph looks less responsive than expected, or shows coarser spikes than you'd see at the actual scrape interval. Cause: A scrape was missed. irate() spans two scrape intervals to find its two samples — the calculated rate is still approximately correct in value, but you've lost the instantaneous precision that makes irate() useful. Its entire advantage is the per-scrape-interval resolution; a missed scrape eliminates that. Fix: Switch to rate() for any graph where scrape reliability isn't guaranteed. Reserve irate() for high-frequency, well-scraped environments where you specifically need sub-window spike visibility.

Flat line at zero when you expect a rate

Symptom: rate(my_metric[5m]) returns a flat zero even though you know traffic is flowing. Cause: The metric might be a gauge, not a counter. A gauge that isn't changing returns zero rate. Fix: Check the metric's type. Run the raw metric name and look at the values over time — if they go up and down, it's a gauge. Query it directly. If they only go up, it's a counter and rate() should work.

The “rate of a rate” mistake

# Type error: rate() expects a range vector, not an instant vector
rate(rate(http_requests_total[5m])[5m])

This is a hard type error, Prometheus will reject it at parse time. The inner rate() returns an instant vector; a range selector [5m] cannot be applied to an instant vector. Beyond the type system, it wouldn't make conceptual sense either: rate() values go up and down (traffic can decrease), so they're not counters and have no meaningful "rate of change."

Your Five Rate Exercises

Run these in order each one builds on the previous:

# 1. Raw counter — watch it only go up
http_requests_total{job="<your-service>"}

# 2. Rate — now you can see per-second traffic
rate(http_requests_total{job="<your-service>"}[5m])

# 3. irate — compare side-by-side with rate() in Grafana
# Notice: more volatile, reacts faster to spikes
irate(http_requests_total{job="<your-service>"}[5m])

# 4. Increase — total requests in the last hour
increase(http_requests_total{job="<your-service>"}[1h])

# 5. Wrong: rate on a gauge — see what happens
rate(node_memory_MemAvailable_bytes[5m])

# Compare with the correct query:
node_memory_MemAvailable_bytes

Exercise 5 is the most instructive. Running rate() on a gauge metric returns a value — Prometheus won't error — but the number is meaningless noise. This is why knowing the metric type before choosing a function matters.

What You Now Know

Counters vs. gauges:

• Counter → only goes up; reset on restart. Use rate(), irate(), or increase().
• Gauge → can go up or down. Query the raw value directly.

The three rate functions:

• rate() → smooth average per-second rate across the window. Default choice.
• irate() → instantaneous rate from last two samples. Responsive but volatile.
• increase() → total count over the window. For absolute reporting, not rates.

Counter resets:

• Both rate() and irate() detect and handle resets automatically.
• A drop in counter value = restart; Prometheus doesn’t treat it as negative delta.

Window size matters:

• Too small → not enough samples, empty or unstable results.
• Too large → over-smoothed, slow to reflect real changes.
• Community guideline: 2× scrape interval is the minimum; 4× gives a stable result for most dashboards.

What’s Next: Aggregation

You now have per-second rates per time series. In Part4 we covers how to collapse those into meaningful summaries: total rate across a service, average across instances, or a breakdown by status code.

📚 Series: Master PromQL

• Part 1: ✅ The Prometheus Data Model
• Part 2: ✅ Selecting Data Correctly — Where Most Mistakes Start
• Part 3: ✅ Rates — Understanding Change Over Time ← You are here
• Part 4: ⏳ Aggregation — Turning Noise into Meaning
• Part 5: ⏳ Joins & Vector Matching — The Real Mastery Checkpoint
• Part 6: ⏳ Real-World Patterns — Apply Everything Together
• Part 7: ⏳ Debugging & Thinking Like a PromQL Expert

If this helped clarify how Prometheus metrics actually work, a small action goes a long way in supporting more deep-dive content like this.

👏 Clap 50 times if you found it useful — it helps surface this article to more engineers who are struggling with the same confusion. 🔔 Follow on Medium for more practical breakdowns of observability, systems, and real-world debugging patterns.