PromQL: The Query Language That Makes Your Metrics Talk

A practical guide to querying Prometheus — from your first metric to production-grade observability queries

Metrics without queries are just numbers sitting in a database. PromQL — the Prometheus Query Language — is what transforms those raw numbers into answers. Why is my service slow? Which endpoint has the highest error rate? Is my infrastructure about to run out of memory?

PromQL answers these questions. And once you understand its logic, it becomes one of the most powerful tools in an engineer’s observability toolkit.

What Is PromQL?

PromQL (Prometheus Query Language) is a functional, read-only query language built specifically for querying time-series data stored in Prometheus. It was designed from the ground up for one purpose: slicing, aggregating, and computing rates over metrics that change over time.

Unlike SQL, PromQL doesn’t have SELECT, FROM, or JOIN. Instead, it treats metrics as first-class citizens and gives you operators purpose-built for time-series math — things like rate(), histogram_quantile(), and topk() that would be awkward to express in SQL.

Every query you write in PromQL returns one of four result types:

Type Description Example Use Instant vector One value per time series at a point in time Current CPU usage Range vector Multiple values per time series over a time window CPU usage over last 5 min Scalar A single number A computed constant String A text value Rarely used directly.

The Building Blocks

1. Metrics and Labels

Every metric in Prometheus is identified by a name and a set of labels — key-value pairs that describe dimensions of that metric.

http_requests_total{job="api", method="POST", status="200"}

Here http_requests_total is the metric name, and job, method, status are labels. Labels are what make PromQL powerful — they let you filter, group, and aggregate with surgical precision.

Label matchers:

# Exact match
http_requests_total{status="200"}

# Not equal
http_requests_total{status!="200"}

# Regex match — all 5xx errors
http_requests_total{status=~"5.."}

# Regex not match — exclude health checks
http_requests_total{path!~"/health|/ping"}

2. Instant Vectors vs Range Vectors

An instant vector gives you the current value of a metric:

http_requests_total

A range vector gives you values over a time window — written with [duration]:

http_requests_total[5m]    # last 5 minutes of samples

Range vectors are almost always used as input to functions like rate() or increase() — you rarely use them raw.

Duration units:

s → seconds    m → minutes    h → hours
d → days       w → weeks      y → years

The Functions You’ll Use Every Day

rate() — The Most Important Function

rate() calculates the per-second average rate of increase of a counter over a time window. This is how you turn a monotonically increasing counter like http_requests_total into a meaningful "requests per second" metric.

# Requests per second over the last 5 minutes
rate(http_requests_total[5m])

Rule of thumb: Always use rate() with at least 4x the scrape interval. If Prometheus scrapes every 15s, use [1m] as the minimum window.

irate() — Instant Rate

Like rate() but uses only the last two data points. More responsive to sudden spikes, but noisier. Good for dashboards where you want sensitivity; avoid for alerting.

irate(http_requests_total[5m])

increase() — Total Increase Over a Window

Returns the total increase in a counter over the time range — not per second, but the raw delta.

# Total requests in the last hour
increase(http_requests_total[1h])

histogram_quantile() — Percentile Latency

This is the go-to function for calculating p50, p95, p99 latency from a Prometheus histogram metric.

# p99 request latency
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket[5m])
)

# p50, p95, p99 together — use in Grafana with legend {{quantile}}
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

delta() — Change in a Gauge

For gauges (metrics that go up and down), delta() gives you the change over a time window.

# How much did memory usage change in the last 10 minutes?
delta(node_memory_MemFree_bytes[10m])

predict_linear() — Forecasting

Projects a gauge’s value forward in time using linear regression. Invaluable for capacity planning alerts.

# Will this disk fill up in the next 4 hours?
predict_linear(node_filesystem_free_bytes[1h], 4 * 3600) < 0

Aggregation Operators

Aggregation collapses multiple time series into one (or fewer) by applying a mathematical operation across label dimensions.

# Sum across all instances
sum(rate(http_requests_total[5m]))

# Sum, grouped by service
sum(rate(http_requests_total[5m])) by (service)

# Sum, dropping the instance label (keep everything else)
sum(rate(http_requests_total[5m])) without (instance)

# Average CPU across all nodes
avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (node)

# Maximum memory usage across the fleet
max(node_memory_MemUsed_bytes) by (instance)

# Top 5 services by request rate
topk(5, sum(rate(http_requests_total[5m])) by (service))

# Count of running instances per job
count(up == 1) by (job)

The RED Method in PromQL

The RED method (Rate, Errors, Duration) is the standard framework for monitoring request-driven services. Here’s how to express it in PromQL:

Rate — Requests per second

sum(rate(http_requests_total[5m])) by (service)

Errors — Error rate percentage

sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
* 100

Duration — p99 latency in milliseconds

histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
) * 1000

These three queries, dropped into a Grafana dashboard, give you a complete health picture of any HTTP service.

Alerting with PromQL

PromQL is also the language of Prometheus alerting rules. Alert conditions are just PromQL expressions that return a non-empty result when the alert should fire.

# prometheus/alerts.yml

groups:
  - name: api_alerts
    rules:

      # Fire when error rate exceeds 5% for 5 minutes
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)
          > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: "Error rate is {{ $value | humanizePercentage }}"

      # Disk filling up in 4 hours
      - alert: DiskFillingSoon
        expr: predict_linear(node_filesystem_free_bytes[1h], 4 * 3600) < 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Disk on {{ $labels.instance }} predicted to fill in 4h"

Common Patterns & Recipes

Is my service up?

up{job="my-api"}
# Returns 1 (up) or 0 (down) per instance

CPU utilization percentage per node

100 - (
  avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) * 100
)

Memory usage percentage

(
  node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
) / node_memory_MemTotal_bytes * 100

Apdex score (satisfaction ratio)

(
  sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
  +
  sum(rate(http_request_duration_seconds_bucket{le="1.2"}[5m]))
) / 2
/
sum(rate(http_request_duration_seconds_count[5m]))

Active connections per service

sum(http_active_connections) by (service)

PromQL Pitfalls to Avoid

1. Using rate() on a gauge rate() is only for counters (ever-increasing values). For gauges, use delta() or deriv().

# ❌ Wrong — memory is a gauge
rate(node_memory_MemUsed_bytes[5m])

# ✅ Correct
delta(node_memory_MemUsed_bytes[5m])

2. Time window too short A [1m] window on a 30s scrape interval means only 2 data points. rate() needs at least 4 for statistical reliability. Use [2m] minimum.

3. Forgetting by in aggregations Without by, you collapse ALL labels and lose the ability to distinguish services, instances, or environments.

# ❌ Collapses everything — one number
sum(rate(http_requests_total[5m]))

# ✅ Grouped — one line per service
sum(rate(http_requests_total[5m])) by (service)

4. Comparing metrics with different label sets Binary operations between two metrics require matching labels. Use on() or ignoring() to control which labels are matched.

# Match only on 'instance', ignore other labels
metric_a / on(instance) metric_b

Quick Reference Card

# Rate of change (counters)
rate(metric[5m])

# Percentile latency
histogram_quantile(0.99, rate(metric_bucket[5m]))

# Error rate %
sum(rate(errors[5m])) / sum(rate(total[5m])) * 100

# Top N by value
topk(5, sum(metric) by (label))

# Predict exhaustion
predict_linear(gauge[1h], 3600 * 4) < 0

# Service up/down
up{job="service-name"}

# CPU %
100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100

# Memory %
(MemTotal - MemAvailable) / MemTotal * 100

The Bottom Line

PromQL has a learning curve. The concept of range vectors, the distinction between counters and gauges, the nuance of label matching — these take time to internalize. But the investment pays off quickly.

Once you can write histogram_quantile(0.99, rate(latency_bucket[5m])) from memory, you've unlocked the ability to answer any performance question your system throws at you — in seconds, not hours.

Start with the RED method. Wire up three panels in Grafana. From there, everything else is just patterns stacked on a solid foundation.

Metrics tell the story of your system. PromQL is how you read it.