Grafana for Everyone — Part 6: Loki — Adding Logs to Your Grafana Stack

Metrics tell you a service is slow. Logs tell you why. This part adds Grafana Loki to the stack — and shows how to correlate logs with metrics in the same dashboard.

Why Loki?

Before Loki, the standard log stack was Elasticsearch + Kibana (the ELK stack). It works, but it’s heavy: Elasticsearch indexes every field in every log line, which is powerful but expensive — both in compute and operational complexity.

Loki’s philosophy is different: it indexes only metadata (labels), not the log content itself. Log lines are stored compressed. Querying scans and filters the content at read time.

The tradeoff:

• ✅ Much cheaper to store and operate
• ✅ Label model matches Prometheus — same labels, unified querying
• ✅ Native Grafana integration — logs appear in the same dashboard as metrics
• ❌ Full-text search is slower than Elasticsearch for massive volumes
• ❌ No field extraction at ingest time (unless you use pipelines)

For most teams running Kubernetes or Docker workloads, Loki hits the sweet spot. For compliance-heavy log analytics at petabyte scale, Elasticsearch may still win.

Architecture

Your Application / Containers
         │ (stdout/stderr)
         ▼
    Promtail / Alloy
    (log shipping agent)
         │ (push)
         ▼
    Loki (storage + query)
         │
         ▼
    Grafana (visualization)

Promtail (or the newer Grafana Alloy) runs as a DaemonSet on each Kubernetes node, tails container logs, attaches labels, and ships to Loki. Loki stores them. Grafana queries Loki and renders them.

Quick Start with Docker Compose

Add Loki and Promtail to your existing stack:

version: "3"
services:
  loki:
    image: grafana/loki:latest
    ports: ["3100:3100"]
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - loki-data:/loki

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yaml:/etc/promtail/config.yaml
    command: -config.file=/etc/promtail/config.yaml

  grafana:
    image: grafana/grafana-oss:latest
    ports: ["3000:3000"]

volumes:
  loki-data:

promtail-config.yaml:

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container
      - source_labels: [__meta_docker_container_image]
        target_label: image

Add Loki as a Data Source

Connections → Data sources → Add data source → Loki

Field Value Name loki-local URL [http://loki:3100](http://loki:3100)

Save & test. You should see “Data source connected and labels found.”

LogQL — The Query Language

LogQL is Loki’s query language. It’s modeled after PromQL but operates on log streams.

Log Stream Selector (required)

Every LogQL query starts with a stream selector — a set of label matchers that identify which log streams to query:

{app="api-server"}
{app="api-server", env="production"}
{app=~"api-.*"}

This is how Loki avoids scanning all logs — it looks up only the matching streams.

Pipeline Stages

After the stream selector, pipe through filters and parsers:

{app="api-server"}
  |= "error"                          # line contains "error"
  != "health check"                   # exclude health check lines
  | json                              # parse JSON log fields
  | status_code >= 500                # filter on parsed field
  | line_format "{{.method}} {{.path}} → {{.status_code}}"

Filter operators:

Operator Meaning |= "string" Line contains string != "string" Line does not contain string |~ "regex" Line matches regex !~ "regex" Line does not match regex

Parsers:

• | json — parse entire line as JSON, extract fields
• | logfmt — parse key=value format
• | pattern "<ip> <method> <path> <status>" — positional pattern extraction
• | regexp "(?P<status>\\d{3})" — named capture groups

Metric Queries

LogQL can derive metrics from log streams — count log lines, extract numeric fields, and compute rates:

# Count error log lines per minute
sum(rate({app="api-server"} |= "error" [1m])) by (service)

# Extract HTTP status codes and compute error rate
sum(rate({app="nginx"} | json | status_code >= 500 [5m]))
/
sum(rate({app="nginx"} | json [5m]))

These LogQL metric queries can be used in Grafana panels just like PromQL — including in alert rules.

Building a Logs Dashboard

Panel 1: Log Stream (Logs visualization)

{app=~"$service"} |= "$search_term"

• Visualization: Logs
• Panel options: Enable Deduplication, Wrap lines
• Time: Newest first

Add a search_term text box variable (Part 3) to let users filter logs in real time. Type "timeout" in the box → only timeout log lines appear.

Panel 2: Error Rate from Logs

sum by (app) (
  rate({app=~"$service"} |= "ERROR" [5m])
)

• Visualization: Time series
• Unit: logs/sec

Panel 3: Log Volume by Level

sum by (level) (
  rate({app=~"$service"} | json | level != "" [5m])
)

• Visualization: Bar chart (stacked)
• Shows the distribution of INFO / WARN / ERROR over time

Correlating Logs and Metrics in One Dashboard

This is Grafana’s killer feature for observability.

Layout

Row: "Service Health"
├── [Time series] Error rate (PromQL)
├── [Stat] p95 latency (PromQL)
└── [Time series] Request rate (PromQL)

Row: "Logs"
├── [Logs panel] Error log stream (LogQL)
└── [Time series] Log error rate (LogQL metric)

When you zoom into a time range on any panel, all panels update — both metric and log panels honor the same time range. Spot a latency spike at 14:32 on the Prometheus panel → scroll down → the Loki logs panel shows exactly what was logged at 14:32.

Data Links: Metrics → Logs

Add a data link on your metric panels to jump to a pre-filtered log view:

Title: View logs
URL: /explore?orgId=1&left={"datasource":"loki-local","queries":[{"expr":"{app=\"${__series.name}\"}","refId":"A"}],"range":{"from":"${__from}","to":"${__to}"}}

Click a series on the error rate graph → opens Grafana Explore with logs for that service in the same time window. This is the “click to drill down” pattern that separates great observability platforms from mediocre ones.

Loki in Kubernetes

For Kubernetes, use the official Helm chart:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install Loki stack (Loki + Promtail)
helm install loki grafana/loki-stack \
  --namespace monitoring \
  --set grafana.enabled=false \
  --set prometheus.enabled=false

Promtail auto-discovers pods and attaches labels from Kubernetes metadata:

namespace="production"
pod="api-server-7d4f8b-xkq2p"
container="api"
node="ip-10-0-1-45"

Those labels flow through to Loki and appear in LogQL queries — same label model as your Prometheus metrics.

LogQL Best Practices

Always start with a specific stream selector — {app="api-server"} before filters. Never query {} (all streams) — it's the equivalent of SELECT * FROM with no WHERE clause on a billion-row table.

Use structured logging — If your app outputs JSON logs, | json unlocks field-level filtering. Unstructured text logs are harder to filter precisely.

Index carefully chosen labels — Loki indexes labels at ingest. High-cardinality labels (request IDs, user IDs) should stay in log content, not labels. Good labels: app, env, namespace, level.

Use \|= "error" before | json — Filter on raw line content first to reduce the set of lines being parsed. Parsing is expensive; pre-filter when possible.

Up Next

Part 7: Provisioning and Dashboard-as-Code

We’ll manage dashboards, data sources, and alert rules as version-controlled YAML and JSON — so your entire Grafana setup is reproducible, reviewable, and deployable via CI/CD.