From Blind Spots to Bulletproof Monitoring: Building an Uptime Dashboard with Prometheus, Blackbox Exporter and Grafana

It is Tuesday at 2:00 PM. The team just deployed a major feature to production. Everything looks stable, so you grab a coffee. Ten minutes later, your phone blows up. Users are seeing 500 Internal Server Errors, customer support is flooded, and you have no idea when or why the service dropped. Because you lack proactive monitoring, you are completely blind to failures from the user’s perspective.

During my internship, I was handed the task of fixing this exact vulnerability by creating a centralized, internal health check and uptime monitoring dashboard.

Understanding the Core Architecture

To build a reliable uptime system, firstly need to understand how Prometheus collects data.

Prometheus Core Ecosystem Architecture. Source: Prometheus

Prometheus relies on a pull based system. It actively reaches out to targets to pull metrics at regular intervals and saves them in its internal Time Series Database. However, standard internal metrics only tell you if an application is running internally. They do not tell you if a public endpoint is accessible across the open internet, if the SSL certificate is valid, or if network routing is broken.

Core Architectural Components

• Prometheus Server: The brain of the entire operation. It handles three major jobs. First, it uses a retrieval engine to actively pull (scrape) metrics from your applications. Second, it saves that data inside its custom Time Series Database (TSDB). Third, it runs PromQL queries to calculate data trends and check if any alert thresholds are crossed.
• Thanos: The enterprise scaling layer that sits on top of our production Prometheus instances. Since local Prometheus storage is ephemeral and restricted to a short retention window, Thanos Sidecar automatically ships these metrics to long term cloud storage like Google Cloud Storage. It also utilizes Thanos Querier to provide a global, unified view of historical data across all our production clusters.
• Exporters: These act as translation proxies for systems that do not speak Prometheus natively. Since tools like Linux databases, hardware routers, or third-party APIs do not expose metrics in a format Prometheus understands, an exporter sits next to them, collects their internal stats, and translates them into a clean HTTP endpoint that Prometheus can scrape.
• Pushgateway: A buffer components for short lived jobs. Because Prometheus relies on a pull model, it needs an endpoint to stay alive long enough to scrape it. If you have a batch script or a cron job that runs for only ten seconds and shuts down, Prometheus will likely miss it. The script pushes its metrics to the Pushgateway instead, which holds onto them so Prometheus can scrape them later.
• Alertmanager: The notification router for your system. The Prometheus server does not actually send emails or ping your phone; it simply triggers an alert state and hands it off. Alertmanager takes that alert, deduplicates it so your inbox does not flood, groups similar errors together, and routes them to communication channels like PagerDuty or Telegram.
• Service Discovery: The automatic detective of your infrastructure. In a dynamic cloud environment like Kubernetes, pods and services are constantly being created, destroyed, or scaled up. You cannot hardcode target IP addresses into your configuration. Service Discovery automatically talks to the underlying cluster API to find new endpoints the exact moment they launch.
• Grafana: The visual interface of the stack. While Prometheus acts as the storage and querying database, Grafana connects directly to it as a frontend data source. It uses PromQL expressions to pull raw time-series data and format it into readable graphs, tables, and uptime timelines.

Why Choose Blackbox Exporter Over Alternatives?

When evaluating external uptime solutions, standalone tools like Uptime Kuma or SaaS platforms are popular due to their instant graphical setups. However, for a production grade Kubernetes infrastructure, chose the Blackbox Exporter configuration.

Standalone systems create isolated data silos, meaning you have to manage a separate UI, an independent database, and an isolated alerting engine. If you want to correlate an infrastructure spike with an API slowdown, you are forced to switch between completely different tools.

The Blackbox Exporter integrates directly into Prometheus. It acts as an endpoint probing proxy. When Prometheus wants to check an external site, it queries the Blackbox Exporter. The exporter hits the target URL over HTTP, HTTPS, TCP, DNS, or ICMP, records the network response, and sends the resulting metrics right back into Prometheus. This keeps all of your telemetry data under a single, unified enterprise roof.

Blackbox Exporter End To End Probing Flow. Source: DevOps Cube

Note: All the code snippets, YAML configurations, and URLs provided in this guide are sample templates. Please ensure you replace placeholder values (such as your-service-api.example.com, project names, tokens, and IDs) with your actual infrastructure details and credentials before deploying them to your cluster.

Deploying Prometheus and Blackbox Exporter via Helm

The most efficient way to manage this stack inside Kubernetes is using the Prometheus Community Helm chart, which deploys the kube-prometheus-stack bundle. This installs Prometheus, Alertmanager, and Grafana together.

First, add the official repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Next, configure the custom properties for the Blackbox Exporter using a values.yaml file to define how endpoints are probed.

prometheus-blackbox-exporter:
  enabled: true
  config:
    modules:
      http_2xx:
        prober: http
        timeout: 5s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
          valid_status_codes: [200, 201, 202]
          method: GET
          follow_redirects: true
          tls_config:
            insecure_skip_verify: false

Deploy the stack into your cluster using the following command:

helm install prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring --create-namespace -f blackbox-values.yaml

Configuring Targets via ServiceMonitor

The ServiceMonitoracts as a simple bridge that tells Prometheus exactly which websites you want to test and routes them through the Blackbox Exporter. Because Prometheus cannot check external URLs on its own, you use the ServiceMonitor to provide a list of your target endpoints and point Prometheus toward the exporter. It automatically instructs Prometheus to send those URLs to the Blackbox Exporter as test requests, collect the performance data that comes back, and save it in your monitoring dashboard without you ever needing to configure complicated routing rules manually.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: internal-uptime-monitor
  namespace: monitoring
  labels:
    release: prometheus-stack
spec:
  jobLabel: app
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus-blackbox-exporter
  endpoints:
  - port: http
    interval: 30s
    scrapeTimeout: 5s
    path: /probe
    params:
      module: [http_2xx]
      target:
      - https://your-service-api.example.com/healthz
      - https://your-frontend-app.example.com/login
    relabelings:
    - sourceLabels: [__address__]
      targetLabel: __param_target
    - sourceLabels: [__param_target]
      targetLabel: instance
    - targetLabel: __address__
      replacement: prometheus-stack-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115

Building the Grafana Uptime Dashboard

Once the metrics start flowing into the Prometheus Time Series Database, Grafana is used to visualize them. Grafana serves as the frontend dashboard layer, querying Prometheus using PromQL (Prometheus Query Language) to build panels that represent the actual user experience.

Setting Up the Data Source and Variables

Before building panels, you need to link Grafana to your data and make the dashboard dynamic so you can filter by specific environments.

1. Connect the Data Source: Inside Grafana, navigate to Connections then Data Sources, click Add data source, and select Prometheus. In the HTTP URL field, enter your internal cluster address [http://prometheus-stack-prometheus.monitoring.svc.cluster.local:9090](http://prometheus-stack-prometheus.monitoring.svc.cluster.local:9090) and click save and test

Note: In our actual production environment, this data source was integrated with thanos to handle global query scaling and long-term history across multiple GKE clusters, but the Grafana setup steps remain exactly the same.

1. Define Dashboard Variables: To keep the dashboard dynamic without creating separate views for every service, added dropdown variables at the top of the page. By navigating to Dashboard Settings then Variables and creating a new Query variable, Grafana automatically pulls in lists of your endpoints, GKE projects, or cluster names directly from data source labels. Can apply a custom Regex pattern right inside the variable settings to clean up the names or filter out unwanted endpoints. Once saved, you can reference these variables in your panels to ensure entire dashboard filters itself automatically based on the specific environment you select from the dropdown menu.

Here is how the key panels on the dashboard were built:

High Level Aggregates (Stat Panels)

These panels sit at the top of the dashboard to provide an immediate summary of global health.

• Total Services: Counts the distinct targets being monitored.
• Services Up vs Down: Checks the probe_success metric, where 1 means healthy and 0 means down.
• Average Response Time: Displays the global network latency across endpoints.

# Average probe duration in milliseconds
avg(probe_duration_seconds{job="internal-uptime-monitor"}) * 1000

Success and Failure Ratios

To see availability trends over time beyond an instantaneous up or down status, added dedicated ratio metrics using PromQL rate windows.

• Success Ratio: Calculates the percentage of successful probes over the last hour.

sum(rate(probe_success{job="internal-uptime-monitor"}[1h])) / count(probe_success{job="internal-uptime-monitor"}) * 100

• Failure Ratio: Identifies problematic endpoints by looking at the percentage of failed probes over the same window.

sum(rate(probe_success{job="internal-uptime-monitor"}[1h] == 0)) / count(probe_success{job="internal-uptime-monitor"}) * 100

Instance Overview Table

A grid breakdown mapping each monitored endpoint to its real time metrics. This panel displays the current HTTP status code, specific response time, target GKE cluster name, and the respective GKE project name, making it easy to see if an outage is isolated to a specific environment.

# Querying current HTTP status codes per instance
probe_http_status_code{job="internal-uptime-monitor"}

Uptime State Timeline

This visualization provides a clear view of historical stability, mapping changes in target behavior over time to show exactly when an instance dropped and for how long.

By setting the Grafana visualization type to State Timeline and tracking probe_success, Grafana automatically draws green bars when the uptime is 99%+ and red bars when it drops to below 98%.

Probe Status Code Graph

A time series graph tracking probe_http_status_code over time. This makes it simple to distinguish whether a service went down due to a network timeout or if the application server explicitly started throwing 502 Bad Gateway or 500 internal application errors.

Actionable Alerting via Alertmanager

A dashboard is only effective if your team knows when to look at it. True observability requires a mechanism that wakes someone up when a system fails. This is handled by Alertmanager.

Alertmanager is the dedicated component in the Prometheus ecosystem that deduplicates, groups, and routes alerts sent by the core Prometheus engine to external platforms like PagerDuty or Telegram.

To set this up, I utilized designing the alert thresholds inside the Grafana UI or it can also be configured declaratively as code via Kubernetes YAML manifests.

The Production Advantage with Thanos: In production setup, while local alerts handle immediate cluster-level failures, relying on Thanos Ruler to evaluate these alerting rules globally is helpful. Because Thanos has access to telemetry data across all GKE clusters and long-term storage, it ensures that if an entire cluster drops, alerting pipeline doesn’t go silent. It gives a centralized, cluster-agnostic alerting engine.

1. Setting Up Alerts via the Grafana UI

Using the Grafana UI allows you to visually test alert thresholds against historical data before activating them.

Inside the panel editor, navigate to the Alerting tab and click Create alert rule from this panel. You select your expression (such as tracking endpoint failures using probe_success == 0, or catching bad responses when the HTTP status code is not successful with probe_http_status_code != 200), define the evaluation interval (evaluate every 1 minute for a duration of 1 minute), and configure the contact points. Within the UI settings, you can append labels and drop direct link variables into the annotations so that the resulting message contains a clickable path back to the exact dashboard panel.

2. Setting Up Alerts via Kubernetes Declarative YAML

For reliable infrastructure management, these exact same rules should be saved as code using a PrometheusRule resource. This ensures your alert configuration is version controlled and matches what you see in the UI.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: uptime-alert-rules
  namespace: monitoring
  labels:
    release: prometheus-stack
spec:
  groups:
  - name: uptime.rules
    rules:
    - alert: EndpointDown
      expr: probe_success{job="internal-uptime-monitor"} == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "Endpoint Down: {{ $labels.instance }}"
        description: "The instance {{ $labels.instance }} has been unreachable for more than 1 minute."
        dashboard_url: "https://grafana.example.com/d/uptime-dash?var-instance={{ $labels.instance }}"

    - alert: InvalidHTTPStatusCode
      expr: probe_http_status_code{job="internal-uptime-monitor"} >= 400
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "Bad HTTP Status Code on {{ $labels.instance }}"
        description: "The instance {{ $labels.instance }} returned an HTTP status code of {{ $value }}."
        dashboard_url: "https://grafana.example.com/d/uptime-dash?var-instance={{ $labels.instance }}"

Alertmanager Routing Configuration

Once Prometheus triggers these rules, Alertmanager processes them according to your routing definitions, sending critical errors to PagerDuty and warnings to Telegram.

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'telegram-alerts'
  routes:
  - match:
      severity: critical
    receiver: 'pagerduty-high-priority'

receivers:
- name: 'telegram-alerts'
  telegram_configs:
  - bot_token: '123456789:ABCdefGhIJKlmNoPQRsTUVwxyZ'
    chat_id: -987654321
    message: "Alert: {{ .CommonAnnotations.summary }}\nDescription: {{ .CommonAnnotations.description }}\nDashboard: {{ .CommonAnnotations.dashboard_url }}"

- name: 'pagerduty-high-priority'
  pagerduty_configs:
  - routing_key: 'pd-integration-key-placeholder'
    client: 'Prometheus Alertmanager'
    details:
      instance: '{{ .CommonLabels.instance }}'

Personal Takeaways

Building this uptime monitoring system from the ground up highlighted how valuable proactive engineering truly is. Moving away from a reactive strategy, where you only find out about a crash when a customer complains to an automated alert system changes how you manage production infrastructure. By binding Prometheus, the Blackbox Exporter, and Grafana into a single workflow, the team gains the exact visibility needed to catch and fix issues before they ever impact the end user.