Distributed Tracing for LLM Agents: When MCP Makes Tool Calls Observable

One trace ID across the model loop and the tool subprocess — using OpenTelemetry, CrewAI, and Jaeger.

Production failures in LLM systems are often misattributed to the model. In practice, many incidents live in the action layer: a downstream API that time out, a tool that returns a business error inside a successful RPC, a subprocess the host spawned but never joined to the same trace. Standard logs capture completions; they rarely preserve the causal chain decision → tool invocation → observation → next decision.

This article is about that gap. It compares classic APM to agent telemetry, explains how the Model Context Protocol (MCP) gives observability a stable integration point, and points to a minimal reference stack (OpenTelemetry, optional Logfire, Jaeger) where host and tool server share one trace_id.

Code: github.com/ekb-dev-ai/mcp-trace-demo

LLM telemetry vs classic APM — and what MCP transfers

Classic APM assumes a largely deterministic call graph: a request enters a service, fans out to databases and queues, each hop becomes a span with stable identity, latency, and error semantics. The unit of analysis is the request boundary. OpenTelemetry succeeded because that graph is finite and repeatable across deployments.

Agent systems change the shape of work. Execution is a loop, not a handler: the runtime may call a language model, parse a structured action, invoke an external capability, append the result to context, and repeat. The expensive and risky steps are inference (variable latency, token cost) and side effects (tools, APIs, subprocesses). A trace that only wraps “the agent” or only logs final text cannot answer operational questions such as: which tool ran, with what arguments, how long it took, and whether the failure was transport-level or semantic.

Two partial fixes are common and both are incomplete:

1. Completion logging — auditable, but detached from tool causality.
2. A single root span around the agent — one box in Jaeger, no visibility into the tool subprocess or remote server.

What is needed is the same abstraction microservices already use: distributed tracing across process boundaries, with LLM-specific spans nested inside.

MCP does not replace OpenTelemetry; it defines where the tool boundary sits. The protocol specifies discovery, typed tool schemas, and invocation (tools/call). SEP-414 allows W3C trace context in params._meta, so propagation can cross stdio pipes or HTTP the way it crosses service meshes. An MCP server—whether a local subprocess or a remote host—is observability-wise a peer service: its own service.name, its own spans, joinable to the host via trace_id. The host records orchestration and model rounds; the server records tool execution; exporters merge them into one waterfall.

In short: HTTP gave APM a stable wire format for service-to-service calls; MCP gives APM a stable wire format for model-to-tool calls. The agent loop stays stochastic; the act step becomes inspectable.

What a useful agent trace contains

Without tying this to any particular domain, a minimal useful trace for one agent run typically includes:

[embed]

The operational payoff is attribution: distinguish “bad reasoning” from “slow dependency” from “tool returned an error payload the model misread.”

Architecture pattern (host + MCP server)

A common deployment pattern:

• Host process — agent framework, model client, MCP client. Exports spans as e.g. agent-host.
• Tool process — MCP server exposing one or more tools. Exports spans as e.g. mcp-tool-server.
• Transport — often stdio (subprocess with JSON-RPC on stdin/stdout) or HTTP for remote servers.
• Backend — OTLP to Jaeger, Grafana Tempo, Logfire, etc.

┌──────────────┐     MCP (stdio or HTTP)      ┌─────────────────┐
│  agent-host  │ ─── traceparent in _meta ──► │ mcp-tool-server │
│  LLM + client│                              │  tool handlers  │
└──────┬───────┘                              └────────┬────────┘
       │ OTLP                                          │ OTLP
       └────────────────────► trace backend ◄───────────┘

Propagation requirement: both sides instrument MCP and share a single TracerProvider (or compatible OTLP pipeline). On the host, framework and model instrumentors must attach to that provider — not install a second global one, or spans fragment.

Video: build + Jaeger walkthrough

[embed]

The recording covers: shared otel setup, stdio_safe on the MCP server, logfire.instrument_mcp(), running order #1842, and reading one trace from crewai.workflow through tools/call to the red check_inventory span.

Constraint: stdout is the protocol

On stdio, stdout is JSON-RPC only. Logfire console output on the same stream produces Failed to parse JSONRPC message.

The MCP server configures telemetry with stdio_safe=True:

# mcp_server/incident_server.py
configure_telemetry(service_name="mcp-incident-server", stdio_safe=True)

That sets console=False in logfire.configure for the child. Export to Jaeger is unchanged; only terminal printing to stdout is disabled. The agent process uses stdio_safe=False — it may log traces locally; nothing reads that pipe for MCP.

Shared telemetry (both processes)

# otel/setup.py (abbreviated)
def configure_telemetry(*, service_name: str, instrument_crewai: bool = False, stdio_safe: bool = False):
    logfire.configure(
        service_name=service_name,
        send_to_logfire=bool(token),
        console=False if stdio_safe else None,
    )
    logfire.instrument_mcp()
    if instrument_crewai:
        tracer_provider = logfire.DEFAULT_LOGFIRE_INSTANCE.config.get_tracer_provider()
        CrewAIInstrumentor().instrument(tracer_provider=tracer_provider)

Without LOGFIRE_TOKEN, spans go to **http://localhost:4318/v1/traces (Jaeger). The CrewAI instrumentor must use Logfire’s existing TracerProvider; otherwise you may see MCP spans only** and miss crewai.workflow / LLM spans.

Agent entry:

configure_telemetry(service_name="crew-incident-agent", instrument_crewai=True)
os.environ.setdefault("CREWAI_TRACING_ENABLED", "false")  # OTel, not CrewAI AMP prompts

Instrumented failure (server)

@app.tool()
def check_inventory(sku: str) -> dict[str, Any]:
    with logfire.span("check_inventory", sku=sku, slow_seconds=SLOW_SECONDS):
        time.sleep(SLOW_SECONDS)
        if sku == FAIL_SKU:
            logfire.error("inventory backorder blocks fulfillment", sku=sku)
            return {"found": True, "isError": True, "status": "backorder", ...}

isError: True in the payload is a business failure inside a successful tool call — not necessarily a transport error.

Reading one trace in Jaeger

1. Open **http://localhost:16686** → service crew-incident-agent → latest trace.
2. Expand: crewai.workflow → task → agent → *.llm spans (Ollama rounds).
3. Between LLM spans: MCP client tools/call for get_order, then check_inventory.
4. Switch service to **mcp-incident-server, same trace_id**: server handle → nested get_order → wide/erroring check_inventory.

Takeaway: the model may be fine; the inventory span explains the incident.

Reproduce

git clone https://github.com/ekb-dev-ai/mcp-trace-demo.git && cd mcp-trace-demo
docker compose up -d
ollama pull llama3.2:latest
poetry install && ./scripts/demo.sh

No Ollama (trace rehearsal only): ./scripts/quick_trace_demo.sh

Limits

• Demo uses in-memory orders/inventory, not production APIs.
• Span volume (CrewAI + LiteLLM + MCP + manual spans) can be high; use TRACELOOP_TRACE_CONTENT=false to redact prompts.
• stdio propagation is the clearest teaching case; HTTP MCP adds TLS, auth, and ops concerns.
• Tracing shows what happened; it does not replace evals or SLOs on tool success rates.

Summary

Application observability matured around request-scoped, deterministic graphs. LLM agents introduce stochastic loops with external actions. MCP standardizes those actions as protocol-level calls across services, and SEP-414 carries trace context across that boundary so existing OpenTelemetry pipelines apply. The engineering work is mostly wiring: one telemetry setup per process, MCP instrumentation, correct stdio discipline, and a backend that can display host and server spans under one id.

Code and video: mcp-trace-demo. Comments on production patterns for MCP tracing welcome.