PromQL: The Query Language That Makes Your Metrics Talk

A practical guide to querying Prometheus — what it is, how it works, and when to use it

What Is PromQL?

Prometheus Query Language — PromQL — is a functional, read-only query language built specifically for time-series data stored in Prometheus. If Prometheus is the engine that scrapes and stores your metrics, PromQL is the steering wheel that lets you ask questions of that data.

Unlike SQL, which operates on rows and tables, PromQL operates on time-series vectors — streams of timestamped float64 values identified by a metric name and a set of key-value labels.

http_requests_total{job="api-server", status="200"}

That single expression returns every time series that matches those label filters — no FROM, no JOIN, no WHERE. The data model and the query language are tightly coupled, which makes PromQL terse but extremely powerful once you understand the building blocks.

The Data Model in 60 Seconds

Before writing queries, you need to understand what you’re querying. Prometheus stores four metric types:

Every metric is tagged with labels — key-value pairs that add dimensions. Labels are what make PromQL’s filtering and aggregation so expressive.

The Four Building Blocks of PromQL

1. Instant Vectors

A snapshot of all matching time series at a single point in time.

node_cpu_seconds_total

Returns one sample per matching series — the current value. This is the most basic query form.

2. Range Vectors

All samples in a time window for each series — required as input for functions like rate().

node_cpu_seconds_total[5m]

The [5m] window says: give me the last 5 minutes of data. You can use s, m, h, d, w, y as duration suffixes.

3. Scalars

Plain numeric values with no time-series context.

1024 * 1024

Useful as multipliers or thresholds in expressions.

4. Strings

Used in some special functions, rarely in practice.

Label Filtering

Label matchers let you slice your data precisely. There are four operators:

# Equality
http_requests_total{status="200"}

# Inequality
http_requests_total{status!="500"}

# Regex match
http_requests_total{path=~"/api/.*"}

# Regex non-match
http_requests_total{path!~"/health|/metrics"}

Combining multiple matchers acts as a logical AND:

http_requests_total{job="api-server", status=~"5.."}

The Functions You’ll Use Daily

rate() — For Counters

Calculates the per-second average rate of increase over a range window, handling counter resets gracefully.

rate(http_requests_total[5m])

Rule of thumb: Always use rate() on counters, never raw values. Counters only go up — rate() gives you the meaningful derivative.

Use irate() instead when you want the instantaneous rate (last two samples only) — useful for fast-moving metrics where averages would smooth out spikes.

increase() — Total Change Over a Window

increase(http_requests_total[1h])

Returns the total number of requests in the last hour. Internally equivalent to rate() * duration.

sum(), avg(), min(), max() — Aggregations

Aggregate across label dimensions using the by or without clause:

# Total request rate across all instances
sum(rate(http_requests_total[5m]))

# Per-job breakdown
sum by (job) (rate(http_requests_total[5m]))

# Drop the instance label, keep everything else
sum without (instance) (http_requests_total)

histogram_quantile() — Latency Percentiles

The most powerful function for latency analysis:

histogram_quantile(0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

This computes the p99 latency across all request durations. The le label (less-than-or-equal) is the bucket boundary that Prometheus histograms use.

topk() / bottomk() — Finding the Extremes

topk(5, sum by (endpoint) (rate(http_requests_total[5m])))

Returns the 5 busiest endpoints by request rate. Indispensable for debugging traffic spikes.

How to Configure Prometheus for PromQL

PromQL is usable anywhere Prometheus exposes its HTTP API — but you need a properly configured Prometheus instance first.

Basic prometheus.yml

global:
  scrape_interval: 15s       # How often to scrape targets
  evaluation_interval: 15s   # How often to evaluate rules

scrape_configs:
  - job_name: "node-exporter"
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: "my-api"
    static_configs:
      - targets: ["api-server:8080"]
    metrics_path: /metrics
    scrape_interval: 10s     # Override global for this job

Recording Rules — Pre-compute Expensive Queries

For heavy queries you run frequently (dashboards, alerts), record them as new metrics:

# rules/recording.yml
groups:
  - name: api_metrics
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

Load in prometheus.yml:

rule_files:
  - "rules/*.yml"

Alerting Rules

groups:
  - name: availability
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m])
          /
          rate(http_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for {{ $labels.job }}"

Real-World Use Cases

1. Service Health Dashboard

# Request rate
sum(rate(http_requests_total[5m])) by (service)

# Error ratio
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/ sum(rate(http_requests_total[5m])) by (service)

# p95 latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

These three queries — rate, error ratio, latency percentile — are the RED method (Rate, Errors, Duration), the gold standard for service monitoring.

2. Infrastructure Capacity Planning

# CPU utilisation per node
1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
)

# Memory pressure
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# Disk space trend — will I run out in 4 hours?
predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600) < 0

predict_linear() does linear regression on a range vector and extrapolates — powerful for proactive alerting.

3. SLO Burn Rate Alerting

Multi-window burn rate is the modern approach to SLO alerting (from Google’s SRE workbook):

# 5-minute burn rate
rate(http_requests_total{status=~"5.."}[5m])
/ rate(http_requests_total[5m])

# 1-hour burn rate  
rate(http_requests_total{status=~"5.."}[1h])
/ rate(http_requests_total[1h])

# Alert when fast AND slow windows both exceed threshold
(burn_rate_5m > 14.4) and (burn_rate_1h > 14.4)

4. Kubernetes Pod Resource Monitoring

# CPU throttling ratio per pod
sum by (pod, namespace) (
  rate(container_cpu_cfs_throttled_seconds_total[5m])
)
/ sum by (pod, namespace) (
  rate(container_cpu_cfs_periods_total[5m])
)

# Memory usage vs limit
sum by (pod) (container_memory_working_set_bytes)
/ sum by (pod) (kube_pod_container_resource_limits{resource="memory"})

Accessing PromQL

Common Pitfalls

Using rate() on a gauge — rate() is only meaningful for counters. Use plain values or delta() for gauges.

Too-short range windows — rate(metric[1m]) with a 15s scrape interval gives only ~4 samples. Use at least 4× your scrape interval as the window.

Label cardinality explosion — Adding high-cardinality labels like user_id to metrics will bloat your TSDB. Keep label values bounded.

Forgetting by in aggregations — sum(metric) drops all labels. If you need a specific breakdown, always specify by (label1, label2).

The Mental Model

Think of PromQL as a pipeline:

raw time series
  → filter with label matchers
  → apply functions (rate, histogram_quantile)
  → aggregate across label dimensions (sum by, avg by)
  → compare or combine with operators (+, /, >, and)
  → single number or vector ready for alerting/dashboards

Once that pipeline clicks, even complex multi-step queries become readable.

Wrapping Up

PromQL has a learning curve — the vector model feels foreign at first — but it pays off fast. A handful of functions (rate, sum by, histogram_quantile, predict_linear) covers 90% of real-world monitoring needs. Start with the RED method for your services, layer in infrastructure metrics, and build alerting rules incrementally.

The best way to learn is the expression browser at http://your-prometheus:9090/graph. Type a metric name, add label filters, wrap in rate(), aggregate — within an hour you'll be writing queries that would have taken dozens of SQL lines.