Master PromQL | Part 1 of 7

You Don’t Understand PromQL Yet Until You Understand This One Concept (Part 1)

Master the Prometheus data model and stop guessing your way through PromQL forever

You’ve probably been using Prometheus for a while now.

You can copy a PromQL query from Grafana, tweak a label or two, and move on like everything makes sense.

But the moment something breaks: empty graphs, duplicate lines, or numbers that don’t add up then you’re suddenly guessing🤔

And that’s the real problem.

Not PromQL. Not Grafana. Not even your queries.

It’s the fact that most engineers start querying before they understand what they’re actually querying.

Because every confusing PromQL result traces back to one root cause:

You were never shown how Prometheus actually thinks about data.

Before we fix queries, dashboards, or rates, this is the part that has to click first.

Once it does, PromQL stops feeling like trial-and-error… and starts becoming predictable.

This is Day 1. We fix that today.

By the end of this article:

You’ll understand exactly what Prometheus stores, why labels are the most important concept in the entire system, and how to explain any query’s behavior from first principles.

Days 2 through 7 build directly on this, so treat it as your foundation.

What Is a Time Series, Actually?

Before you write a single PromQL query, you need to know what you’re querying.

A time series in Prometheus is a sequence of timestamped values, uniquely identified by a metric name and a set of key-value pairs called labels.

That definition sounds simple. It has enormous consequences.

Here’s how a single time series looks in memory:

http_requests_total{job="api", instance="10.0.0.1:8080", method="GET", status="200"}
  → [t=1000, value=1423]
  → [t=1060, value=1431]
  → [t=1120, value=1438]
  → [t=1180, value=1447]

That’s one time series: one metric name, one specific combination of labels, scraped every 60 seconds (Prometheus default).

Change any label, and you get a completely different time series:

http_requests_total{job="api", instance="10.0.0.1:8080", method="POST", status="200"}
http_requests_total{job="api", instance="10.0.0.2:8080", method="GET", status="200"}
http_requests_total{job="api", instance="10.0.0.1:8080", method="GET", status="500"}

Each of those is an independent time series. They share a metric name but nothing else. Prometheus stores them separately and tracks them independently.

This is the first thing to internalize: the metric name is just a namespace. Labels are the identity.

Before we continue

If you found this story insightful, …

👏 Clap 50 times (yes, you can, simply hold the button), it will help me a lot. Medium’s algorithm favors this, increasing visibility to others who then discover the article.

🔔 Follow me on Medium and subscribe to get my latest articles straight to your inbox.

Labels Are Identity, Not Metadata

In most systems, metadata is decorative.

You add tags to a resource to help humans search it, but the resource itself exists independent of its tags.

Prometheus works differently. Labels don’t describe a time series, they define it.

Two time series with the same metric name but different labels are as different as two completely separate metrics.

There is no “base” time series that you tag. There is only: name + full label set = a unique series.

This has a concrete implication when you write queries.

When you write:

http_requests_total

You’re not selecting one thing.

You’re selecting every time series that matches the metric name http_requests_total.

That means every job, every instance, every method, and every status code.

Do the math: 3 services × 5 instances × 4 methods × 3 status codes. That’s 180 separate time series returned by a single query.

This is why queries often “return multiple lines.”

Each line is one unique time series.

You haven’t done anything wrong. You just asked for everything.

Photo by Kristina Flour on Unsplash

Your First Five Queries (Watch the Labels)

Let’s make this concrete.

Imagine your API service:

• runs on 2 instances
• handles GET and POST requests
• returns 200 and 500 status codes

Prometheus scrapes all of it.

Now watch what happens.

Query 1: Select Everything

http_requests_total

This returns 8 separate time series.

Why?

Because you have:

• 2 instances
• × 2 methods
• × 2 status codes

That’s 8 unique label combinations.

So Grafana draws 8 lines.

Nothing is broken.

You asked for all of them.

Query 2: Filter by Job

http_requests_total{job="api"}

Still 8 series.

Why?

Because every series already belongs to job="api".

The filter matched everything.

This is an important PromQL lesson:

Just because you added a filter doesn’t mean you reduced the result set.

Query 3: Filter by Status Code

http_requests_total{status="500"}

Now you get 4 series.

You removed every series that doesn’t have status="500".

That’s your first real narrowing operation.

Query 4: Add Another Filter

http_requests_total{status="500", method="POST"}

Now you’re down to 2 series.

One per instance.

Every additional label filter shrinks the matching set.

PromQL is fundamentally doing set filtering on labels.

That’s it.

No magic.

Query 5: Fully Specify the Identity

http_requests_total{
  status="500",
  method="POST",
  instance="10.0.0.1:8080"
}

Now you get exactly one time series.

At this point, you’ve fully identified the series.

Metric name + full label set = one unique series.

That formula matters more than it looks.

The Data Model Visualized

Here’s how to picture the full structure:

One metric name. Many time series beneath it. Each with its own independent sequence of values.

When you write a PromQL query, you’re selecting one or more of those boxes, either by name alone (get all), or by name + label filters (get the specific ones you want).

How Prometheus Actually Stores This

You don’t need to be a storage engineer to use PromQL well, but understanding the storage model at a high level prevents a class of mistakes.

Prometheus uses a custom time-series database called TSDB (Time Series Database). Here’s what it does:

• Every time series gets a numeric ID internally (more efficient than storing the full label set repeatedly)
• Incoming samples are first written to a WAL (write-ahead log) and held in memory, then flushed into blocks on disk, each block covering ~2 hours of wall-clock time
• Each block contains a chunks/ directory of compressed sample data, an index, and a metadata file
• Blocks are periodically compacted into larger blocks to improve query performance (default retention: 15 days)
• An in-memory index maps label sets → series IDs for fast lookup

When you run a query, Prometheus:

1. Looks up matching series IDs from the label index
2. Reads the relevant chunks from disk or memory
3. Applies your PromQL expression to the retrieved samples

This explains a few things you’ll eventually bump into:

• Queries against recent data are fast (in-memory). Queries against old data hit disk.
• Very high cardinality (many unique label combinations) inflates the index and slows lookups.
• Prometheus isn’t designed for long-term storage, that’s what remote write targets like Thanos or Cortex are for.

Cardinality: The Double-Edged Sword

Cardinality is one of the most important Prometheus concepts.

And one of the most dangerous.

Here’s the simple definition:

Cardinality = the number of unique time series created by your labels.

Some labels are safe.

Low cardinality: A label with a small, bounded set of values.

# Good: status has ~5 possible values (200, 201, 400, 404, 500)
http_requests_total{status="200"}

# Good: environment has 3 values (dev, staging, prod)
http_requests_total{environment="production"}

High cardinality: A label with an unbounded or very large set of values.

# Dangerous: user_id has millions of unique values
http_requests_total{user_id="user-8472918"}

# Dangerous: request_id is unique per request
http_requests_total{request_id="req-a4f8c2e1"}

Here’s what goes wrong with high-cardinality labels:

Every unique label value creates a new time series.

A service with 1 million users, each tagged with user_id, creates 1 million separate time series for that one metric.

Prometheus wasn't designed for this. You'll see:

• Memory usage exploding (all active series must fit in RAM for recent data)
• Slow queries (the label index becomes enormous)
• Scrape timeouts (too many series to serialize in one scrape)
• Eventually: OOM crashes

The rule is simple but important: labels should have a small, predictable number of possible values.

Status codes, environments, HTTP methods, service names — these are good labels.

User IDs, request IDs, IP addresses, email addresses — these are cardinality bombs.

We’ll come back to cardinality throughout this series. It influences aggregation, joins , and performance.

The Scrape Model: Where Data Comes From

Prometheus does not wait for applications to send data.

It pulls the data itself.

This is called scraping.

Every service exposes a /metrics endpoint.

Prometheus periodically calls that endpoint and stores the returned values.

Usually every:

• 15 seconds
• 30 seconds
• or 1 minute

This detail matters more than most people realize.

Because Prometheus is not storing continuous streams.

It’s storing snapshots taken at intervals.

That means:

• missing scrapes create gaps
• rate calculations depend on scrape timing
• spikes can disappear between scrapes

Every graph you see is built from sampled points in time.

Not continuous reality.

That distinction explains a surprising number of weird graphs.

Debug Angle: Why Is My Query Returning Multiple Lines?

This is the most common first confusion in PromQL.

You run a query and get 12 lines instead of 1. You expected a single number.

Now you know why: you selected a metric that has 12 distinct time series. Each line is one unique label combination.

Here’s how to debug it systematically:

Step 1: Run the raw query in the Prometheus UI (not Grafana)

http_requests_total

Look at the table view. Count the rows. Read the label sets. You now know exactly what data exists.

Step 2: Identify which labels are varying

If you see rows differing by instance you have multiple scrape targets. If they differ by statusyou have multiple status codes being tracked.

This is expected behavior, not a bug.

Step 3: Decide what you actually want

• Do you want the total across all instances? → You need aggregation ( sum())
• Do you want only one specific combination? → Add more label filters
• Do you want one per service regardless of instance? → sum by (job)(...)

The multiple lines are information, not errors. The question is: which subset do you want, and how do you want to combine them?

What You Now Know (And Why It Matters)

Let’s be explicit about what you’ve learned and how it connects forward:

The data model: a metric name plus a label set identifies exactly one time series. Change any label value, and you have a different series.

Labels are identity: every label in the label set contributes to uniqueness. Labels with many unique values create many time series (cardinality).

Queries return sets : a bare metric name returns all matching series. Filters narrow the set. Aggregation collapses it.

Storage is sampled — data exists only at scrape points. Rate functions handle the math between samples.

Everything in the remaining 6 parts connects back to this. When a rate function surprises(Part 3) you, it’s because of how series are defined. When aggregation drops labels (Part 4), it’s because labels are identity, not decoration. When joins return nothing (Part 5), it’s because label sets don’t match.

The data model isn’t a prerequisite you get through to reach the “real” content. It is the real content.

Try It Yourself

If you have a Prometheus instance available (local kube-prometheus-stack or a test cluster), run these in sequence and observe the results:

# 1. Select all series for a metric — how many do you get?
up

# 2. How many distinct jobs exist?
count by (job)(up)

# 3. What's the cardinality of each label in http_requests_total?
# (count unique values of the 'method' label)
count(count by (method)(http_requests_total))

# 4. Find your highest-cardinality label
# (look for labels with many unique values in the table view)
http_requests_total

# 5. Narrow to exactly one series using full label specification
up{job="<your-job-name>", instance="<your-instance>"}

No Prometheus available? PromLens (by Julius Volz, co-creator of Prometheus) provides a query builder with a live demo endpoint.

Alternatively, spin up a local Prometheus with Docker in under a minute:

docker run -p 9090:9090 prom/prometheus

it exposes its own metrics at localhost:9090

📚 Series: Master PromQL

• Part 1: ✅ The Prometheus Data Model ← You are here
• Part 2: ✅ Selecting Data Correctly — Where Most Mistakes Start
• Part 3: ✅ Rates — Understanding Change Over Time
• Part 4: ⏳ Aggregation — Turning Noise into Meaning
• Part 5: ⏳ Joins & Vector Matching — The Real Mastery Checkpoint
• Part 6: ⏳ Real-World Patterns — Apply Everything Together
• Part 7: ⏳ Debugging & Thinking Like a PromQL Expert