Read the Thread, Merge the PR: Building a First-Responder AIOps Bot with Grafana, Openab, and Claude Code

Photo by Lucas van Oort on Unsplash

Everyone is using AI to write code faster, but writing code isn’t the bottleneck in production engineering. Debugging is.

When an incident occurs, we waste massive amounts of cognitive load just gathering state — running the same repetitive triage scripts over and over. We don’t need AI to replace the engineer; we need AI to do the mechanical grunt work so the engineer can focus entirely on judgment.

We built a first-responder bot that does exactly that. By pairing Grafana alerts with Claude Code, our on-call engineers are greeted with a fully triaged incident thread instead of a blank terminal. The entire interaction — from reading findings to requesting the fix — fits in a messaging app.

This post breaks down the exact pipeline that turned a complex, multi-week debugging nightmare into a 5-minute PR review. This is not a demo. It runs in our production clusters today, gated by read-only IAM, Kubernetes RBAC, and human approval.

The High-Level Architecture

The goal was to build a non-disruptive, asynchronous helper. When an alert fires, it forks into a standard notification path and an automated investigation path simultaneously, keeping human intervention completely opt-in.

Grafana alert fires
       │
       │  Routing rule: auto_investigate=true
       ▼
Webhook contact point
       │
       │  POST /alert-openab-notifier
       │  Authorization: Bearer <WEBHOOK_SECRET>
       ▼
WAF (NAT-IP allowlist)
       │  Only cluster NAT gateway IPs reach this path
       ▼
API Gateway  →  Lambda (Alert Bridge)
       │         Parses Grafana payload
       │         Groups firing alerts
       │         Posts to Slack with @openab in top-level text
       ▼
Slack #alerts channel
       │
       │  message.channels event → openab
       │  bot_id validated against trusted_bot_ids
       ▼
openab pod (Kubernetes)
       │
       │  Spawns Claude Code agent per thread
       │  Agent reads SKILL.md (cluster context, runbooks)
       │  Agent calls MCP tools + kubectl + AWS CLI
       ▼
Investigation posted in alert thread
       │
       │  Human reads findings, asks follow-up, or:
       ▼
"@openab create a PR to fix this"
       │
       ▼
MR created → requires N human approvals → merge → deploy

Opting an alert into the pipeline requires a single label on any Grafana alert rule:

labels:
  auto_investigate: "true"

The notification policy routes these flagged alerts to an internal webhook bridge in parallel with normal team routing. Existing workflows are completely untouched.

Phase 1: Overcoming the Slack Notification Trap (Grafana → Lambda)

You might wonder why we didn’t just use Grafana’s native Slack contact point. We tried.

The Gotcha: Grafana’s native Slack integration places @mentions inside visual attachment blocks. Slack does not fire an app_mention webhook event for mentions hidden inside attachments. The bot stayed sound asleep.

To fix this, we dropped a lightweight, schema-agnostic AWS Lambda in the middle to handle payload normalisation and post clean, top-level text strings that Slack’s event listener actually detects:

payload = {
    "channel": SLACK_CHANNEL_ID,
    "text": f"<@{OPENAB_USER_ID}> 🚨 {len(alerts)} alert(s) firing..."
}

Writing a Zero-Maintenance Alert Bridge

We didn’t want to update our Lambda every time someone added a new metric alert. To keep it completely generic, the Lambda uses a dynamic annotation renderer. It sanitises and strips internal routing labels (__name__, grafana_folder), transforms URL values into clickable Slack links, and explicitly escapes raw inputs:

def _slack_escape(text):
    return str(text).replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")

Why this matters: label values pulled from Kubernetes nodes can contain characters like < or >. Without escaping, a malformed label could inject rogue Slack commands or fake links into a highly trusted infrastructure channel.

Security: Two Independent Layers

Layer 1 — WAF NAT-IP allowlist: a WAF rule blocks all traffic to the alert endpoint that does not originate from the EKS cluster’s NAT gateway IPs. Requests from any other source are dropped before reaching the Lambda.

Layer 2 — WEBHOOK_SECRET: the Lambda validates Authorization: Bearer <secret> on every request. The secret lives in Secrets Manager.

Production Gotcha #2: API Gateway v1 (REST API) passes headers with their original casing (Authorization). HTTP API v2 automatically lowercases all headers (authorization). If you change your API Gateway flavour without normalising your lookup dictionary, your webhook auth will silently fail with 401 Unauthorized:

# Safe header normalisation pattern
headers = {k.lower(): v for k, v in (event.get("headers") or {}).items()}
auth_header = headers.get("authorization", "")

Phase 2: Connecting Chat to the Cluster via openab

Once Slack fires the message event, it needs to talk to the cluster. We use openab — a lightweight, open-source Rust broker that bridges Slack and Discord to any ACP (Agent Client Protocol) compatible CLI — Claude Code, Kiro, Codex, Gemini, and others. It speaks Slack Socket Mode on one side and ACP stdio JSON-RPC on the other.

┌──────────────┐  Socket Mode  ┌──────────────┐  ACP stdio  ┌──────────────────┐
│    Slack     │◄─────────────►│    openab    │────────────►│   Claude Code    │
│   #alerts    │               │    (Rust)    │◄─ JSON-RPC ─│   (acp mode)     │
└──────────────┘               └──────────────┘             └──────────────────┘

openab’s design philosophy is deliberate: it is a thin transportation layer. It does not manage agent memory, does not inject system prompts, does not orchestrate multi-agent workflows. The orchestration layer sits above openab and belongs to the operator.

To allow the Lambda bridge to trigger the bot automatically, we configured openab with two critical lines:

slack:
  allowBotMessages: "mentions"   # Listens to automated bot alerts
  trustedBotIds:
    - "B0XXXXXXXXX"              # The specific ID of our Lambda bridge

The ID Trap: trustedBotIds requires the B-prefixed bot ID, not the user-facing U-prefixed user ID. Slack event payloads pass the raw bot_id. If you use the user ID here, openab will quietly ignore the alerts. Turn on debug logging to extract the exact B0... identifier from the rejected payload:

DEBUG openab::slack: bot not in trusted_bot_ids, ignoring app_mention
  event_bot_id="B0XXXXXXXXX"

Phase 3: The Security Boundary (How to Sleep at Night)

Giving an LLM agent access to kubectl commands understandably makes security teams anxious. We handle this by strictly separating the Read Surface from the Write Surface.

openab pod
  │
  ├── AWS IRSA role (read-only, IMDS disabled)
  │     CloudWatch: GetMetricData, ListMetrics, FilterLogEvents
  │     EKS:        DescribeCluster, ListClusters
  │     EC2:        Describe*
  │     STS:        AssumeRole → cross-account-readonly only
  │     AWS_EC2_METADATA_DISABLED=true  ← no fallback to node role
  │
  ├── Kubernetes RBAC: platform-readonly ClusterRoleBinding
  │     get, list, watch on all resources
  │     NO create / update / delete / patch
  │
  ├── Grafana MCP: Viewer SA token (read-only)
  │     PromQL, LogQL, dashboards — no writes
  │
  └── GITLAB_TOKEN (write surface — GitLab only)
        Can:    create branch, commit, open MR
        Cannot: merge without N human approvals
        Cannot: touch AWS, Kubernetes, or Grafana

The write surface is deliberately narrow: GitLab MR creation only, still gated by human review. The bot cannot apply its own fixes.

AWS_EC2_METADATA_DISABLED=true is a critical explicit control — without it, the pod could fall back to the EC2 node's IAM role, which typically has broader permissions.

The runtime environment uses aws eks get-token inside its kubeconfig plugin setup rather than static credentials. Tokens are ephemeral, scoped specifically to the pod's IRSA role, and fetched dynamically at the instant a tool runs.

Phase 4: Injecting Tribal Knowledge via SKILL.md

A generic AI model dropped into a complex infrastructure ecosystem is useless. It doesn’t know which Grafana organisation maps to which cluster, which Helm chart defines memory limits, or what your naming conventions imply.

Instead of writing custom code or brittle agent chains, we manage institutional knowledge as code via a version-controlled SKILL.md markdown file. An init container pulls this file fresh from git whenever the openab pod boots up.

The same init container also registers the agent’s full tool set — Grafana MCP servers for each cluster org, Atlassian MCP for Jira and Confluence, plus two knowledge MCPs that let the agent look things up rather than hallucinate:

• context7 — library and framework documentation (Kubernetes API, Helm, Terraform)
• aws-knowledge — official AWS service documentation and regional availability

When the bot encounters an unfamiliar AWS error or needs to verify a Kubernetes field, it queries these MCPs directly rather than guessing. This is what makes SKILL.md’s Unknown Checkpoint actually enforceable — the agent has the tools to resolve unknowns before it gives up.

Inside SKILL.md, we define strict operating behaviours:

The Unknown Checkpoint: before drafting an incident wrap-up, the agent must explicitly answer six guardrail questions (e.g., Do I know exactly when this behaviour started? Is the blast radius isolated?). If any answer is missing, it must continue gathering metrics.

Output Sanitisation: the agent is banned from ever dumping base64 blobs, raw CA certificates, or dense CVE lists directly into Slack, protecting channel log history from credential leakage.

The Self-Improving /skill-review Loop

Infrastructure evolves, and so should the agent. When an engineer resolves an interesting or unusual support ticket, they run a single command in the thread: /skill-review INC-1234.

Support session completes
       │
       ▼
Engineer runs: /skill-review INC-1234
       │
       ▼
Bot reads ticket + conversation + current SKILL.md
Self-challenges each proposed gap against 4 criteria
Writes exact proposed markdown changes to a new file
Creates branch → commits → opens PR for human review
       │
       ▼
Human reviews and merges → bot gets smarter on next pod spin-up

The agent reads the entire troubleshooting history, cross-references it with the existing SKILL.md, and drafts a highly specific patch outlining the missing context. Once a human engineer reviews and merges the PR, the bot automatically gets smarter on the next pod restart.

Real Production Demos

Example 1: The Standard OOMKill (Resolved in <2 minutes)

When an out-of-memory error hits our tracing stack, the bot responds within seconds inside the Slack thread, proactively mapping out all adjacent containers:

[8:10 PM] openab:
✅ kubectl get pod tempo-metrics-generator-1 -n tempo -o wide
✅ kubectl describe pod tempo-metrics-generator-1 -n tempo | tail -50
✅ kubectl logs --previous --tail=200 | grep -E "(OOMKill|killed|exit|signal)"

Root Cause: OOMKilled

| Pod                 | Memory             | Restarts | Last Killed          |
|---------------------|--------------------|----------|----------------------|
| metrics-generator-0 | 1860Mi / 2Gi (93%) | 2        | OOMKilled ~01:07 UTC |
| metrics-generator-1 | 1212Mi / 2Gi       | 12       | OOMKilled ~01:08 UTC |

Both pods are hitting the 2Gi ceiling on WAL reloads during high ingest spikes.

Recommended fix: increase metricsGenerator requests to 3Gi, limits to 4Gi
in Helm values.

⚠️ metrics-generator-0 is at 93% and will likely crash again before a fix
deploys. Run kubectl rollout restart statefulset/tempo-metrics-generator
-n tempo to buy time.

When the on-call engineer replies @openab create a PR to fix this OOM thanks, the bot uses its GitLab write token to locate the exact Helm file in the IaC repository, cuts a branch, commits the change, and links the merge request in the thread.

That sentence was typed on a phone. No laptop, no terminal, no VPN.

Example 2: The Complex, Non-Obvious Failure Class

This example shows why raw automation beats static runbooks. We had an internal synchronisation agent experiencing a CrashLoopBackOff.

Instead of hallucinating or guessing, the agent analysed the exit behaviours:

[2:36 PM] openab:
The previous container logged a clean shutdown with exit code 0, not a crash.
Kubernetes is restarting it because it finishes running, not because it panics.

Let me dig into the watch logic...

Findings: cluster-sync-agent on platform-prod-cluster

Root cause: The container exits cleanly every ~30 minutes due to one
unhandled KeyError exception:
  {"error": "'resourceVersion'", "kind": "ConfigMap", "level": "error"}

The Python code attempts to access resource['resourceVersion'] on a ConfigMap
deletion tombstone event where the field is absent. This single error triggers
a graceful shutdown, which the K8s deployment immediately restarts.

Pattern: This loop has been cycling silently for 21 days (1,048 total restarts)
with minimal user impact due to fast container startup times.
This is a chronic bug, not an acute outage.

The on-call engineer, realising a teammate might have looked at this earlier, asked: @openab we had a fix from a teammate — MR !13. Does it make sense?

The bot parsed the live GitLab merge request diff and responded:

[2:47 PM] openab:
The fix in MR !13 is architecturally sound. It introduces an _on_watch_error
callback that catches HTTP 410 'too old resource version' events cleanly and
re-opens the iterator rather than terminating the container.

However, the production cluster is still running the old image digest built
3 weeks ago. The automated pipeline built the new image with the fix today,
but the deployment itself hasn't been bumped.

To resolve: trigger your deployment stack to push today's updated digest
to production.

Instead of a human wasting hours searching logs and checking git history to see if code made it to production, the bot correlated the live Kubernetes container image SHA directly with the GitLab registry in seconds.

Bounding the Machine

AIOps works beautifully when it operates like an eager, highly disciplined junior engineer with instant reading speed. By keeping the runtime boundaries strictly read-only on the infrastructure side and forcing every code adjustment through standard GitOps human review pipelines, we get all the benefits of instant triage without passing operational control to a model.

ActionWhoValidate bot’s diagnosisHumanMR approvalHuman (N approvals required)Merge to mainHumanRollout timingHumanReviewing skill improvement proposalsHuman

The human engineer is still the ultimate authority — they just start troubleshooting at the answer instead of the query prompt.

What’s Next?

If you are looking to build or optimise a chat-driven operational bot like this, the path forward usually involves expanding your agent’s visibility or refining your continuous tuning pipeline.

Where would you like to take this architecture next?

• Explore writing custom MCP servers for infrastructure metrics
• Deep dive into structuring an effective SKILL.md system file

openab is open source: github.com/openabdev/openab