From Alert to Root Cause in 60 Seconds: Building an AI On-Call Agent

A POC that receives alerts from any monitoring tool, autonomously investigates using AWS APIs, and posts a root cause diagnosis to Teams/Slack — before a human even looks at it.

The 3am Problem

An alert fires. You wake up, check CloudWatch, scroll through logs, cross-reference with recent deployments, and 20 minutes later realise it was a transient spike that resolved itself.

The investigation process is almost always the same:

1. What’s the current state of the service?
2. What do the metrics/logs show?
3. Did anything change recently? (usually a deployment)
4. Is there a runbook for this?

That’s a workflow an AI agent can follow. So I built one.

The Architecture

The bot sits between your monitoring stack and your team. It doesn’t replace CloudWatch or Grafana or Datadog — it consumes their webhooks and adds an investigation layer before notifying humans.

Stack: Python 3.12 · FastAPI · Strands Agents SDK · Claude Sonnet · boto3 · ChromaDB · ECS Fargate

How It Works: The Agent Loop

This is the core concept. It’s a single Strands agent with 13 tools. Claude receives the alert, then iterates — calling tools, observing results, deciding what to check next — until it has enough evidence to notify the team.

Here’s a real investigation flow:

And here’s what’s happening under the hood:

1. Claude receives: "ECS order-service CPU spike, 3 unhealthy targets"
2. Thinks: "ECS alert — check service health"
   → calls describe_ecs_service() → "2/5 tasks running, 3 crashed OOM"
3. Thinks: "OOM — check logs for memory errors"
   → calls get_cloudwatch_logs(filter="OutOfMemory") → "47 events since 14:23"
4. Thinks: "Started at 14:23 — was there a deployment?"
   → calls list_recent_cloudtrail_events() → "UpdateService at 14:20"
5. Thinks: "Root cause found. Check runbooks, then notify."
   → calls search_runbooks("ECS OOM after deployment") → resolution steps
   → calls send_teams_alert(root_cause="Deploy at 14:20 increased memory...")

This is not a hardcoded sequence. If step 2 showed connection timeouts instead of OOM, Claude would pivot to checking the ALB or RDS. It reasons about each result before choosing the next action.

The system prompt constrains it to 3–4 tool calls — fast enough to feel real-time (~30–60 seconds), thorough enough to be useful.

The 13 Tools

investigation_agent = Agent(
    model=AnthropicModel(model_id="claude-sonnet-4-6"),
    tools=[
        # AWS investigation (read-only)
        get_cloudwatch_metrics,         # CPU, memory, error trends
        get_cloudwatch_logs,            # Error pattern search
        describe_ecs_service,           # Task counts, deployments, events
        describe_rds_instance,          # DB status, storage, connections
        describe_ec2_instance,          # State, status checks
        get_alb_target_health,          # Healthy/unhealthy targets + reason
        run_ssm_command,                # OS diagnostics (whitelisted)
        list_recent_cloudtrail_events,  # What changed in the last hour?
        describe_lambda_function,       # Config, runtime, state
        # Knowledge
        search_runbooks,                # ChromaDB vector search
        # Notification (Claude decides when)
        send_teams_alert,
        send_slack_alert,
        page_oncall,                    # PagerDuty — critical only
    ],
    system_prompt=SYSTEM_PROMPT,
)

All AWS tools are read-only. The SSM tool (which runs commands on EC2 instances) has a whitelist — only ps, df, free, journalctl, docker ps, etc. Nothing that modifies state.

The Normalizer: 8 Sources, One Format

Every monitoring tool sends a different payload. CloudWatch wraps alerts in SNS. PagerDuty nests incidents three levels deep. Datadog puts metadata in tags that are sometimes a list, sometimes a string.

The normalizer converts all of them into one unified structure:

@dataclass
class NormalizedAlert:
    source: str          # "CloudWatch", "Grafana", "Datadog", etc.
    severity: str        # critical | high | medium | low
    title: str
    service: str
    account_id: str
    region: str
    resource_id: str
    raw: dict            # Original payload for context

It also gates investigation — resolved alerts, OK states, and recovery notifications get acknowledged and skipped. No point spending $0.10 on a Claude call for an alert that already resolved.

Supported sources: CloudWatch, Grafana, New Relic, Site24x7, Datadog, Splunk, PagerDuty, OpenSearch.

The System Prompt: Encoding On-Call Expertise

The prompt isn’t “you are helpful.” It encodes specific on-call patterns:

• Service-specific paths: “For ECS alerts: describe_ecs_service first. For RDS:describe_rds_instance first.”
• CloudTrail is mandatory: “Always check what changed in the last hour. A deployment 20 minutes before an alert is almost always the root cause.”
• Notification quality: “Be specific: ‘3 of 5 ECS tasks crashed — OOM after 14:23 deploy.’ Not: ‘there appears to be an issue.’”
• Confidence levels: High (clear evidence), Medium (likely cause), Low (unclear)
• Paging rules: “Only page for CRITICAL + customer-facing + no resolution path. Never for medium/low or self-resolving.”

Fire-and-Forget Async

Monitoring tools timeout after 5–30 seconds. Investigations take 30–60 seconds. If you block, the tool retries and creates duplicates.

Solution: return 200 immediately, investigate in the background.

@app.post("/webhooks/{source}")
async def handle_webhook(source: str, request: Request):
    alert = normalize(source, payload)
    if not should_investigate(alert):
        return {"status": "acknowledged"}
    asyncio.create_task(_run_investigation(alert))
    return {"status": "investigation_started"}

Runbook Search: ChromaDB Vector Store

The bot doesn’t just investigate — it searches for known procedures. Runbooks from Confluence are loaded into ChromaDB as vector embeddings. When Claude identifies “ECS OOM,” it searches for similar past incidents and includes documented resolution steps in the notification.

The engine also stores every investigation result. Over time, it builds a corpus of past incidents — making future searches more accurate.

Why Strands, Not Hardcoded Logic

The first version used if-else chains:

# v1: Can't adapt mid-investigation
if "ECS" in alert.title:
    check_ecs() → check_metrics() → check_logs() → notify()

Problems: couldn’t pivot when findings pointed somewhere unexpected. Every new alert type needed new code. No reasoning.

Strands gives Claude tools and lets it decide. Same 13 tools handle any alert type — ECS, RDS, Lambda, EC2 — because Claude reasons about what’s relevant rather than following a fixed path.

What I Learned

1. CloudTrail correlation is the killer feature. ~60% of alerts in testing correlated with a recent deployment or config change.

2. The normalizer is 40% of the work. Parsing 8 different webhook formats with their quirks takes more time than the AI integration.

3. “3–4 tool calls” keeps it fast. Without this limit, Claude investigates exhaustively. The constraint keeps it under 60 seconds.

4. Confidence levels change behaviour. Without them, the team doesn’t know whether to trust the diagnosis or investigate themselves.

5. Read-only is non-negotiable. Investigation-only is safe to deploy immediately. The moment an AI can modify infrastructure, you need approval workflows and rollbacks.

What’s Not Production-Ready

• Alert storm protection — a noisy period could trigger 50 parallel investigations
• Deduplication — same alert firing 3 times shouldn’t trigger 3 investigations
• Feedback loop — no way to tell the bot “that diagnosis was wrong”
• Cost circuit breaker — need a daily spend cap on Claude API calls
• Multi-account IAM — production needs cross-account assume role

But as a POC, it proved the concept: an AI agent can do the first 80% of incident investigation — the part that currently wakes someone up at 3am to confirm “yes, it was the deployment.”

Building AI-powered ops tooling? I’d love to hear what patterns are working for you.