IncidentOps AI Agent: Intelligent AI Summarization

By Gururaj Nagaraja Rao, Site Reliability Engineering Technical Leader and Sumit Sengupta, Site Reliability Engineer, at Cisco ThousandEyes

Cisco ThousandEyes delivers comprehensive network visibility and AI-powered assurance to help organizations proactively monitor and optimize digital experiences across complex, distributed cloud and internet environments. Site Reliability Engineers (SREs) and developers ensure ThousandEyes remains available and operates as intended for our customers. We maintain a 24/7 on-call rotation to proactively address operational alerts.

At ThousandEyes, we use Prometheus to write alerting rules and Alertmanager to route alerts to PagerDuty, which integrates with Slack channels. The operational alert Slack threads contain key information on analysis, mitigation, root cause, and next steps. This blog explains how we built an in-house AI agent, IncidentOps, to capture key information from Slack threads for efficient debugging during on-call.

The Alert Context Problem

During on-call shifts, SREs face challenges that slow alert resolution — especially when they need to understand how similar alerts were handled in the past. PagerDuty provides a resolution note feature where On-call engineers can update the Key information about how an alert was resolved, the related root-cause and any long-term action items. PagerDuty integrates with Slack threads to send resolution notes back to them.

However, a significant number of alerts that are not auto resolved lack updated resolution notes in PagerDuty. Each Slack thread can contain anywhere between 20 to 100 messages. On-call engineers often miss updating the resolution note because manually parsing and summarizing these messages is challenging. Hence, this crucial resolution information often remains only in the Slack threads.

How we Leveraged AI to resolve this Problem?

With advancement in Generative AI capabilities, we leveraged AI to resolve this problem. We developed an in-house AI agent, IncidentOps, that ingests data from operational alert Slack threads, preprocesses and filters it, summarizes using AI, and posts resolution notes to PagerDuty. The agent leverages the Agno Agent framework. A brief overview of what this Agent performs is represented in the below diagram:

Figure 1: IncidentOps AI agent — Overview

Step-by-Step workflow of IncidentOps Agent

Figure 2: IncidentOps AI agent — Workflow

Ingestion. Thread data never goes through ad-hoc Slack API code in the workflow. We connect to a Slack MCP server over streamable HTTP, using SLACK_MCP_SERVER_URL as the single configuration knob. The MCP layer handles authentication and protocol details; the workflow focuses on what to fetch (a channel, a time window, or one thread_ts).

Preprocessing and filtering. Raw channels are noisy. We intentionally narrow the problem: threads must be rooted in PagerDuty’s Slack integration (messages from pagerduty_slack_bot) and must show human follow-up. Filtering out bot-to-bot threads allows the model to prioritize tokens for meaningful diagnostic and mitigation content.

Summarization. The filtered threads are passed by the IncidentOps agent to Amazon Bedrock. Amazon Bedrock uses the Claude Sonnet 4.5 Model for AI summarization, tuned to produce a single paragraph under 100 words covering root cause, key actions, and resolution. The AI summary is written to PagerDuty Resolution notes.

The agent runs as a scheduled cron job every 24 hours in production, performing the above steps. Running it daily ensures sufficient information on key actions are captured for the AI summarization

Security

IncidentOps is an autonomous AI agent that writes AI-summarized information to PagerDuty resolution notes without human intervention. We designed the agent to be secure by default and avoid elevated privileges. The design considerations include:

• Read-only access to Slack: The agent has read-only access to the Slack MCP server, preventing unintentional writes to Slack threads.
• Scoped access to PagerDuty: The agent’s permissions limit it to specific functions for reading and writing PagerDuty incidents, implemented via Scoped OAuth.
• PII restriction in PagerDuty writes: The agent excludes engineer names and personal data when writing AI summaries to PagerDuty

Token Costs: Measuring and Optimizing LLM Spend

Once an AI agent runs every day in production, the next question is no longer “does it work?” — it’s “what is it costing us?” Each Slack thread sent to the LLM consumes input and output tokens, and without visibility, those tokens silently turn into a monthly bill.

To answer this, we instrumented IncidentOps with Langfuse, an open-source LLM observability platform. Using OpenTelemetry under the hood, every workflow run is captured as a single trace, with each high-level step — initialize, fetch & filter Slack threads, summarize, and report — appearing as a span. Bedrock LLM calls and Agno agent runs (including any PagerDuty tool calls) show up as nested child spans with the prompt, response, model ID, and token counts attached automatically — no extra instrumentation in the business logic.

This gave us per-run token accounting broken down by thread and Bedrock call, daily and weekly cost trends that surface anomalies, and prompt-level debugging side-by-side with the resulting PagerDuty note.

The visibility itself was the win: it let us right-size the model choice, tighten the prompt, and improve agent instructions — all decisions that materially reduced tokens per run while keeping summary quality intact.

Figure 3: Tracing with Langfuse

Tips for Reliable AI Agent Instructions

An AI agent is only as good as its instructions. Here are some tips that worked for the IncidentOps agent

• Bounded output contract: Provide one paragraph under 100 words covering the root cause, actions, and resolution
• Few-shot over Prose: Provide an example of a realistic Slack-thread summary, as models mimic structure more quickly than they follow rules
• Tool call: Explicitly mention the tool to call (PagerDuty in this case) when a specific condition, such as a valid Incident ID, is found.
• Explicit negative prompting: Specify the noise to ignore — avoid chatter like debates on on-call escalation policy and focus only on key incident information

PagerDuty Notes in your IDE via MCP

On-call engineers already live in their IDE during a page. The moment they need incident context, though, they must leave the editor for the PagerDuty UI. A **PagerDuty MCP server** fixes that once — any MCP-aware client like Cursor gets the same typed tools for incidents, schedules, and notes, instead of every AI client reinventing its own integration.

The catch: the community PagerDuty MCP server (as of 0.9.x version) can add notes to incident but can’t read notes back — which breaks the loop for IncidentOps. So, we shipped Custom PagerDuty MCP Server, a thin fork that adds a list_incident_notes tool to fetch resolution notes. This runs as a Docker and exposes streamable HTTP through our internal MCP gateway (with stdio for local dev). Streamable HTTP addition in the Custom MCP server enables scalable, remote, multi-client, and secure API-based connections.

The security stance is stronger than scoped tokens: the server registers no write tools at all — no add_note, no manageincidents, no `create`*. Even if a model asked, the tool isn’t there. Access is VPN-gated, and the PagerDuty token stays server-side.

The result for on-calls: a one-click “Add to Cursor” install (for Cursor IDEs), then “show incident details and resolution for PagerDuty incident INC-12345” in the IDE chat panel returns the IncidentOps summary inline — without ever leaving the editor.

Figure 4: IncidentOps Summary retrieved via Cursor IDE

Business Value

After rolling out the IncidentOps summarization, we observed these metrics over one week:

• Slack threads that had AI summarization: > 50

• of messages across all Slack threads: > 1000

The metrics show AI summarization reduced manual alert fatigue by removing the need to review each Slack thread and extract key information from about 1,000 messages. Following is an example of a summarization in Staging environment:

Figure 5: IncidentOps Summarization added to Pagerduty Resolution note

On-call engineers gave positive feedback, saying the single summary improved their understanding of incidents and served as a reference to speed mitigation of related incidents.

Conclusion

IncidentOps streamlines on call by extracting signals from sprawling Slack threads and publishing concise PagerDuty resolution notes. Built on a deterministic Agno workflow, it remains secure by default while limiting LLM to summarization. Early results reduced manual review and improved understanding of incidents. We expect this agent to interact with other Incident AI agents to accelerate triage, strengthen security and auditability, and enable our teams to resolve incidents faster with greater confidence.

Want to be a part of our team? ThousandEyes is hiring! Please see our Careers page for open roles.