The ELK Stack: Your Complete Guide to Centralized Logging

What it is, why every engineering team needs it, and how to get it running from scratch

Picture this. Your application is throwing errors in production. Users are complaining. You SSH into server one — nothing obvious. Server two — same. Server three — logs rotated, gone. Meanwhile the error keeps happening and you have no idea where to look.

This is the problem every engineering team hits as they scale. Logs scattered across dozens of servers, containers, and services — no way to search them together, no way to see patterns, no way to correlate what happened on service A with what broke on service B two seconds later.

The ELK Stack was built to solve exactly this.

What Is the ELK Stack?

ELK is an acronym for three open-source tools that work together to form a complete centralized log management and analytics platform:

E → Elasticsearch   (stores and indexes logs)
L → Logstash        (collects, transforms, and ships logs)
K → Kibana          (visualizes and explores logs)

Together they form a pipeline: logs flow in from your applications and infrastructure, get processed and enriched, land in a searchable index, and surface through dashboards and queries — all in near real-time.

In modern deployments you’ll also hear about the Elastic Stack or ELK + Beats — where lightweight agents called Beats (Filebeat, Metricbeat, Packetbeat) replace or complement Logstash at the collection layer. But the core trio remains the foundation.

Your Apps & Servers
        ↓
   Beats / Logstash       ← collect and transform
        ↓
   Elasticsearch          ← store and index
        ↓
      Kibana              ← visualize and explore

Why ELK? The Problem It Solves

1. Logs Are Everywhere — ELK Centralizes Them

Modern applications run across multiple servers, containers, microservices, and cloud regions. Each produces its own logs. Without centralization, debugging means SSH-ing into individual machines and running grep — slow, manual, and incomplete.

ELK pulls all logs into one place. One search box. Every service. Every server. Every container. Instantly.

2. Speed at Log Scale

When you have millions of log lines per day, you can’t read them — you need to search them. Elasticsearch’s inverted index makes full-text search across billions of log lines respond in milliseconds. grep on a flat file at that scale would take minutes.

3. Structure from Chaos

Raw logs are unstructured strings. ELK’s processing layer (Logstash or Beats) parses them into structured JSON fields — extracting timestamps, log levels, service names, user IDs, error codes — so you can filter, sort, and aggregate with precision.

4. Visibility for the Whole Team

Kibana puts log search and dashboards in a browser. Engineers, QA, product managers, and support teams can explore logs without SSH access or command-line skills. Observability becomes a team capability, not an individual one.

5. Alerting and Anomaly Detection

Elasticsearch’s alerting rules can fire notifications when error rates spike, specific patterns appear, or log volume drops unexpectedly — catching issues before users report them.

The Three Components in Depth

Elasticsearch — The Heart

Elasticsearch stores every log as a JSON document in an index (think: a database table, but distributed and schema-flexible). It uses an inverted index to make every word in every log line instantly searchable.

Key concepts:

• Index — a collection of documents (e.g., logs-2024-04-18)
• Document — a single log entry as JSON
• Shard — a horizontal slice of an index, enabling distribution across nodes
• Mapping — the schema defining field types (text, keyword, date, number)

// A log document stored in Elasticsearch
{
  "@timestamp": "2024-04-18T10:23:01.456Z",
  "level":      "ERROR",
  "service":    "payment-api",
  "message":    "Connection timeout after 30s",
  "host":       "prod-server-03",
  "trace_id":   "abc-123-xyz",
  "duration_ms": 30042
}

Logstash — The Pipeline

Logstash is a data processing pipeline with three stages:

INPUT → FILTER → OUTPUT

Inputs — where logs come from (files, Kafka, Beats, syslog, HTTP) Filters — how logs are transformed (parse, enrich, drop, rename) Outputs — where processed logs go (Elasticsearch, S3, stdout)

# logstash.conf

input {
  beats {
    port => 5044           # receive from Filebeat agents
  }
}

filter {
  grok {
    # Parse unstructured log line into structured fields
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:log_message}"
    }
  }

  date {
    match => ["timestamp", "ISO8601"]
    target => "@timestamp"
  }

  mutate {
    remove_field => ["message"]    # remove raw after parsing
    add_field    => { "environment" => "production" }
  }

  # Drop noisy health check logs
  if [path] =~ "/health" {
    drop { }
  }
}

output {
  elasticsearch {
    hosts     => ["http://elasticsearch:9200"]
    index     => "logs-%{+YYYY.MM.dd}"    # daily indices
    user      => "logstash_writer"
    password  => "${LOGSTASH_PASSWORD}"
  }
}

Kibana — The Interface

Kibana is the browser-based UI for everything Elasticsearch. Its core features:

• Discover — search and browse raw log documents
• Dashboard — build and share visual panels (charts, tables, stats)
• Visualize — create individual charts from aggregations
• Alerting — set threshold rules that notify via Slack, email, PagerDuty
• Canvas — pixel-perfect infographic-style dashboards
• APM — application performance monitoring (with Elastic APM agents)

How to Configure ELK — Step by Step

Step 1 — Run ELK with Docker Compose

The fastest way to get a local ELK stack running:

# docker-compose.yml
version: "3.8"

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false      # disable auth for local dev
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    ports:
      - "9200:9200"
    volumes:
      - esdata:/usr/share/elasticsearch/data

  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

volumes:
  esdata:

docker-compose up -d

# Verify Elasticsearch is up
curl http://localhost:9200

# Open Kibana
open http://localhost:5601

Step 2 — Install and Configure Filebeat

Filebeat is a lightweight agent that tails log files and ships them to Logstash or Elasticsearch directly. Install it on every server or container that produces logs.

# filebeat.yml

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
      - /var/log/nginx/*.log
    fields:
      service: my-api
      environment: production
    fields_under_root: true

  - type: container             # for Docker/Kubernetes
    paths:
      - /var/lib/docker/containers/*/*.log

output.logstash:
  hosts: ["logstash:5044"]

# OR send directly to Elasticsearch (skip Logstash)
# output.elasticsearch:
#   hosts: ["elasticsearch:9200"]
#   index: "filebeat-%{+yyyy.MM.dd}"

# Start Filebeat
filebeat -e -c filebeat.yml

Step 3 — Create Index Pattern in Kibana

1. Open Kibana → Stack Management → Index Patterns
2. Click Create index pattern
3. Enter logs-* (matches all daily indices)
4. Set @timestamp as the time field
5. Click Create

Now go to Discover — your logs are searchable.

Step 4 — Build Your First Dashboard

Discover — search for errors:

level: "ERROR" AND service: "payment-api"

Visualize — create a chart:

• Type: Bar chart
• Y-axis: Count
• X-axis: Date histogram on @timestamp
• Split series: level.keyword

Dashboard — combine panels:

• Error count over time (bar chart)
• Top services by error count (pie chart)
• Latest ERROR logs (data table)
• p99 response time trend (line chart)

Save and share the URL with your team.

Step 5 — Set Up an Alert

In Kibana → Stack Management → Watcher or Alerting:

{
  "trigger": {
    "schedule": { "interval": "1m" }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs-*"],
        "body": {
          "query": {
            "bool": {
              "filter": [
                { "term":  { "level": "ERROR" }},
                { "range": { "@timestamp": { "gte": "now-5m" }}}
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": { "ctx.payload.hits.total": { "gt": 100 }}
  },
  "actions": {
    "notify_slack": {
      "webhook": {
        "url": "https://hooks.slack.com/services/YOUR/WEBHOOK",
        "body": "🚨 More than 100 errors in the last 5 minutes"
      }
    }
  }
}

ELK in Production — Best Practices

Index Lifecycle Management (ILM)

Logs grow fast. Use ILM to automatically roll over, shrink, and delete old indices:

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_size": "50gb", "max_age": "1d" }}},
      "warm":   { "min_age": "7d",  "actions": { "shrink": { "number_of_shards": 1 }}},
      "delete": { "min_age": "30d", "actions": { "delete": {}}}
    }
  }
}

Structured Logging from Day One

The more structure your logs have when they enter ELK, the more powerful your queries become. Log JSON from your application directly:

# Python — structured logging
import structlog
logger = structlog.get_logger()
logger.error("Payment failed",
  service="payment-api",
  user_id="USR-123",
  amount=99.99,
  error_code="TIMEOUT"
)

Use Data Streams for Logs

Modern Elasticsearch recommends data streams over manual index management — they handle rollover and ILM automatically:

# Create a data stream template
PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "data_stream": {},
  "template": {
    "settings": { "number_of_shards": 1 }
  }
}

ELK vs The Alternatives

The Full Picture — ELK in One Diagram

Application Servers / Containers
         ↓
    Filebeat (agent)           ← tails log files, lightweight
         ↓
     Logstash                  ← parse, enrich, filter, route
         ↓
   Elasticsearch               ← distributed storage + full-text index
         ↓
      Kibana                   ← search, dashboards, alerts
         ↓
   Your Team                   ← engineers, QA, support, product

The Bottom Line

The ELK Stack solves one of software engineering’s most persistent pain points: making logs useful at scale. It transforms scattered, unreadable text files into a searchable, visualizable, alertable source of truth about everything your system is doing.

The setup investment — a few hours to get Docker Compose running, Filebeat deployed, and a first dashboard built — pays back every time you debug a production issue in minutes instead of hours, or catch a problem before your users do.

Start small. One service. One index. One dashboard. Then watch the rest of your team start asking “can we add our service to ELK too?”

That’s how you know it’s working.

Logs are the nervous system of your application. ELK is how you listen to them.