Operational Excellence on a Data Platform — Two AI Systems, One Story

By Uday Pani, AI/ML Engineering Technical Leader at Cisco ThousandEyes

Operating a modern data platform means juggling EMR, orchestration, ML workloads, tickets, runbooks, and chat — often within the same incident. At NADP (Network Assurance Data Platform) within Cisco ThousandEyes, we invested in AI-driven automation not to replace engineers, but to shorten time-to-answer, standardize investigations, and surface risks before they become outages.

This post describes two complementary systems: the NADP AI Supervisor — a delegated operations copilot grounded in real tools — and the NADP Capacity Planning application, which transforms metrics and forecasting into repeatable, scheduled capacity intelligence for EMR and Airflow workloads.

Business problem

At Cisco ThousandEyes, we operate the Network Assurance Data Platform (NADP) — comprising EMR clusters and Airflow environments that power network assurance applications at scale.

We faced several recurring challenges:

• Reactive scaling: Queueing and saturation occur during peak demand
• Over-provisioning: “Just add more nodes” leads to unnecessary cost
• Manual reviews: Capacity checks depend on human intervention and timing

We wanted to move from reactive troubleshooting to proactive guidance: from “what went wrong?” to “what should we do next?”

Operational Challenges

• Operational complexity and slow time-to-answer:

Requests arrive through multiple channels (forums, chat, tickets), and assembling a useful response takes too long due to fragmented workflows.

• Fragmented tooling:

EMR clusters, Airflow workflows, and SageMaker resources live across AWS consoles, Airflow UI, and CloudWatch. Runbooks reside in Confluence and Jira, while coordination happens in Slack.

A single question — “Why did EMR cluster X fail?” or “What’s the status of workflow Y in environment Y?” — requires navigating multiple systems instead of a unified view.

• Expertise barrier:

Good diagnosis depends on knowing where to look and how to interpret results. This expertise does not scale easily across a growing or rotating team.

• Toil and inconsistency:

The same investigation pattern repeats: locate the cluster → inspect steps → cross-check runbooks → correlate with tickets. This manual process is error-prone and inconsistent.

Strategic impact

• Unified access: One entry point for operational questions — instead of hunting across consoles, wikis, and chat. Users ask in natural language; routing sends the work to the right supervisor and specialists.
• Composability: A single thread can combine domains, e.g. inspect an EMR cluster, pull the related Jira context, and surface the Confluence runbook — without forcing the user to stitch that flow manually. Extending coverage is mostly additive: new MCP tools or agents/teams behind the router, not a rewrite of the whole stack.
• Scalability: Expand the NADP MCP with new tools, add or adjust agents (or teams) behind the router, and the same integration patterns apply. The architecture favours horizontal growth over big-bang releases.
• Accuracy and trust: Langfuse and related tracing make runs inspectable, which tools ran, with what inputs and outputs, and where something failed. That supports debugging, evaluations, and continuous improvement of prompts and tool design — so the system gets better under real operational load.

NADP AI Supervisor Agent

• The NADP AI Supervisor follows a supervisor-style design: a coordinating system that delegates tasks to specialized components rather than attempting to centralize all logic in one place.
• A standardized interface exposes domain operations — EMR, Airflow, SageMaker, and environments — in a way that can be reliably invoked. Additional integrations bring in context from systems such as Kubernetes, GitHub, Jira, Confluence, and Slack.
• This allows a single question to combine live platform data with relevant operational context, reducing the need for engineers to manually stitch together information from multiple systems.

Customized MCP

NADP MCP is the standardised interface between your NADP domain (EMR, Airflow, SageMaker, multi-account/multi-region environments) and any agent that speaks with the MCP.

It turns “platform knowledge” into repeatable tools: list environments, find/investigate clusters, workflows, runs, SageMaker models/training — aligned with how operators ask questions.

“We iterated from many granular tools to a smaller, clearer set — same coverage, less agent confusion.”

Design principles we care about for the custom NADP MCP:

• Read-only, tool-grounded — Answers follow MCP and agent outputs; we avoid fabrication when data isn’t there.
• Minimal delegation — Route to the smallest set of agents needed for the question.
• Observable runs — Tracing and logging help us debug wrong tool choice and improve prompts and tools over time.

Engineers get one conversational entry point (e.g. via an internal AI interface and routing layer) for “what’s wrong with this cluster?” or “what’s the status of this workflow in a given environment?” — without tab-hopping as the first step.

NADP Capacity Planning Agent

While the Supervisor helps react in the moment, NADP Capacity Planning helps the team act before pain shows up in incidents.

The workflow has three main steps; the heavy lifting happens in the middle step, which runs the same pipeline for every resource.

Design Choices We’re Happy With

1. No static cluster list — Discovery is metric-driven; new clusters/environments show up automatically.
2. One pipeline for all workload types — EMR and Airflow (and future types) share the same steps; each has a “provider” for discovery, queries, capacity score, and signals.
3. Structured AI output — Recommendations are parsed into actions, priority, confidence, and cost impact so we can reliably build tables and filters.
4. One report per run — One notifier per week keeps the channel useful instead of noisy.
5. Extensibility — New workload = new provider + config; workflow and agents stay the same.

Deployed on a weekly cadence, it turns capacity planning from an ad hoc spreadsheet exercise into a repeatable operational process — aligned with operational excellence: predictable review, shared baselines, and documented outputs.

Bringing It Together

Together, the Supervisor and Capacity Planning systems support both reactive investigation and proactive optimization on the same platform.

Key Takeaways

• Operational AI is most effective when combining supervision and delegation with well-designed platform interfaces
• Start with operator workflows, then refine tools until each one has a clear purpose
• Move from reactive to proactive operations through scheduled, data-driven insights
• Apply AI selectively using deterministic systems for forecasting and AI for interpretation and recommendations

Architecture Approach

We use the Agno agent and workflow framework for orchestration, supporting multi-step workflows and stateful execution. Deployment is managed through GitOps practices using Argo CD.

This enables a structured model of agents, coordinated workflows, and service-based integration, allowing capabilities to evolve incrementally over time.

Conclusion

AI automation in NADP is not about replacing operators. It is about reducing repetitive work, improving consistency, and making expertise easier to apply.

By combining real-time assistance with proactive planning, we create a more predictable, efficient, and scalable operational model.

If you are building similar systems — whether supervisor-based automation, platform interfaces, or forecasting workflows — we welcome the opportunity to exchange ideas and learn from each other.

Want to be a part of our team? ThousandEyes is hiring! Please see our Careers page for open roles.