Building Our Own PaaS Infrastructure (Part 6) — Part 2: Centralized Log Management (Loki, Promtail & Grafana)

In the previous article of our series (Part 1), we started monitoring our server resources with Beszel and set up our early warning system. Now, we are moving on to the second major pillar of our monitoring architecture: Centralized Log Management.

Dozens of different Docker containers (databases, APIs, web interfaces, proxies) are running in our own PaaS infrastructure. Whenever an error occurs in any application, connecting to the server via SSH and searching for logs one by one with the docker logs <container_name> command becomes unsustainable as the system grows. We need a lightweight structure that will aggregate logs in a central location, label them, and allow us to search through them without exhausting the server.

Instead of resource-heavy traditional logging solutions (e.g., ElasticSearch), we will use the trio of Loki, which incredibly optimizes disk and memory usage by indexing only labels; Promtail, which collects logs on the server and forwards them to Loki; and Grafana, our visualization layer where we will read this data.

Security in Architecture: Authelia OIDC Integration

In this part, we continue to apply our “Zero Trust” philosophy. To ensure secure access to the Grafana panel, we connect Grafana directly to Authelia as an OIDC (OpenID Connect) client.

Thanks to this modern approach, we do not need to use an extra middleware on the Traefik side; Grafana itself directly redirects the user who wants to log in to Authelia’s secure login screen. Moreover, based on the group information returned from Authelia, we can automatically assign “Admin” or “Viewer” privileges to the user within Grafana.

Step 1: Preparation of Configuration Files

Before proceeding with the installation, we must create two basic configuration files that Promtail and Loki will need on our server (for example, in the /home/user/grafana/ directory).

1. Promtail Configuration (promtail-config.yaml)

This is the necessary configuration file for Promtail to read the logs of Docker containers, label them, and send them to Loki. By preventing the monitoring tools from sending their own logs, we prevent log pollution and resolve possible label errors.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

2. Loki Configuration (loki-config.yaml)

This is the file where we determine where Loki will save the logs and their retention periods.

In upcoming parts, when we install MinIO (S3 Object Storage) in our infrastructure, we will move this setting to MinIO storage in accordance with S3 standards. For now, we are configuring it to keep the logs on the server’s own disk (filesystem).

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: filesystem
  retention_enabled: true

# Performance and Limit Settings
limits_config:
  retention_period: 720h # Logs are kept for 30 days
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  allow_structured_metadata: true
  max_query_parallelism: 32
  split_queries_by_interval: 15m
  max_line_size: 512KB

query_scheduler:
  max_outstanding_requests_per_tenant: 2048

frontend:
  compress_responses: true

3. Creating the Grafana Client on the Authelia Side

As a final preparation, we must add Grafana to the configuration.yml file so that Authelia can accept Grafana's OIDC requests.

1. Hash the password you set for Grafana on your server (e.g., SecretPassword123):

docker run authelia/authelia:latest authelia crypto hash generate pbkdf2 --password 'SecretPassword123'

1. Copy the resulting hash value and add this block to the identity_providers -> oidc -> clients list in your Authelia configuration file:

identity_providers:
  oidc:
    hmac_secret: hmac-secret
    clients:
      - client_id: grafana
        client_name: Grafana
        client_secret: "$pbkdf2-sha512$310000$..." # The hash you generated will go here
        public: false
        authorization_policy: one_factor
        require_pkce: true
        pkce_challenge_method: S256
        redirect_uris:
          # Adjust according to your own domain
          - https://grafana.yourdomain.com/login/generic_oauth
        scopes:
          - openid
          - profile
          - email
          - groups
        response_types:
          - code
        grant_types:
          - authorization_code

Save the changes and restart the Authelia container. The infrastructure is now ready to accommodate Grafana.

Step 2: Docker Compose Installation

Now that our configuration files are ready, we can spin up the entire structure via Dokploy with a single docker-compose.yml. Pay attention to Grafana's environment variables here. We are handling the Authelia integration and role mappings entirely from here.

services:
  loki:
    image: grafana/loki:2.9.2
    container_name: loki
    labels:
      - "app_name=loki"
    restart: unless-stopped
    expose:
      - "3100"
    command: 
      -config.file=/etc/loki/local-config.yaml
    volumes:
      - /home/user/grafana/loki-config.yaml:/etc/loki/local-config.yaml
    networks:
      - dokploy-network
    healthcheck:
      test: ["CMD-SHELL", "wget -q --spider http://localhost:3100/ready || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s

  promtail:
    image: grafana/promtail:3.0.0
    container_name: promtail
    labels:
      - "app_name=promtail"
    restart: unless-stopped
    volumes:
      - /home/user/grafana/promtail-config.yaml:/etc/promtail/config.yaml
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: 
      -config.file=/etc/promtail/config.yaml
    networks:
      - dokploy-network
    healthcheck:
      test: ["CMD-SHELL", "test -e /etc/promtail/config.yaml || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 5s

  grafana:
    image: grafana/grafana:latest
    container_name: grafana-logs
    restart: always 
    environment:
      - GF_SERVER_DOMAIN=grafana.yourdomain.com
      - GF_SERVER_ROOT_URL=https://grafana.yourdomain.com/

      # Security: Disables the default login form, redirecting directly to OIDC
      - GF_AUTH_DISABLE_LOGIN_FORM=true
      - GF_AUTH_OAUTH_AUTO_LOGIN=true
      # OIDC (Authelia) Settings
      - GF_AUTH_GENERIC_OAUTH_ENABLED=true
      - GF_AUTH_GENERIC_OAUTH_USE_PKCE=true
      - GF_AUTH_GENERIC_OAUTH_NAME=Authelia
      - GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana
      - GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=YourSecretPassword # The plain text password we will give to Authelia
      - GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email groups
      - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.yourdomain.com/api/oidc/authorization
      - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.yourdomain.com/api/oidc/token
      - GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.yourdomain.com/api/oidc/userinfo
      - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true # Required for new users to be created
      - GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=preferred_username
      - GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=name
      - GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
      - GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH='Admin' # I am assigning the default admin role (can be managed via ldap groups)
    volumes:
      - grafana-storage:/var/lib/grafana
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.grafana.rule=Host(`grafana.yourdomain.com`)"
      - "traefik.http.routers.grafana.entrypoints=websecure"
      - "traefik.http.routers.grafana.tls.certresolver=letsencrypt"
      - "traefik.http.routers.grafana.service=grafana"
      - "traefik.http.services.grafana.loadbalancer.server.port=3000"
    networks:
      - dokploy-network
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:3000/api/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
volumes:
  grafana-storage:
networks:
  dokploy-network:
    external: true

Step 3: Adding Loki to Grafana and Monitoring Logs

When you go to your Grafana address (https://grafana.yourdomain.com), the Authelia login screen will directly welcome you. After logging in, our last step is to show Grafana where to pull the logs from.

1. Go to Connections > Data Sources from the left menu in Grafana.
2. Click the Add data source button and select Loki from the list.
3. In the URL section, directly type http://loki:3100 since the applications are located in the same Docker network.
4. Confirm the connection by clicking the Save & Test button at the bottom of the page.

Graph View Tip (Scattered Dots Solution): While examining Loki data in the Explore tab in Grafana or when you create a custom timeline panel showing log volume, you might notice that the graphs appear as scattered, disconnected dots instead of a continuous line.

To fix this, simply change the Line interpolation option under Graph styles in the settings menu on the right side of your panel to Linear or Smooth, and set the Connect null values option to Always. Your graph will instantly turn into a continuous and readable flow.

I am leaving some links below that I think might be useful regarding Grafana:

• https://github.com/grafana/grafana
• https://play.grafana.org/dashboards/f/examples/examples

We are now monitoring our system resources with Beszel and can securely query the logs of all our Docker containers from a single center via Loki and Grafana. We have built the observability layer of our own PaaS infrastructure to professional standards.

However, as our applications run and user data accumulates, the security of our databases — the most valuable and sensitive part of the system — becomes our number one priority. Leaving production databases in the same network clutter with other applications or relying on manual backups is a huge risk.

In the next article of the series (Part 7: Database Isolation and Backup), we will discuss how to isolate our databases (PostgreSQL, MySQL, Redis, etc.), which are the heart of our applications, in our infrastructure and how to configure automated cloud backup strategies (storing backups encrypted, S3/MinIO integration, etc.) to reduce data loss to zero.