The Agent That Lied: I Traced an AI Agent with SigNoz🕵

Healthy on the outside. Failing 53% on the inside. I called in a detective named SigNoz.

Here’s a confession: I trusted my AI agent.

From the outside, it looked healthy. It answered questions, called its tools, returned results. But an AI agent is a black box wearing a confident smile. It chains LLM calls, fires off tools, makes its own decisions, and when something breaks, it doesn’t raise its hand. It just quietly lies.

So before the Agents of SigNoz hackathon kicked off, I ran an experiment. I built an AI agent, then interrogated it.

My detective? SigNoz. Open-source, OpenTelemetry-native, and it sees everything your agent does: every trace, log, metric, and exception in one place. You self-host it, point your agent at it, and the black box gets glass walls.

So I hired the detective, wired up the suspect, and gave it nowhere to hide. Here’s what it confessed.

Meet the suspect

My agent had a simple job: take a question, decide which tool to use, call it, and answer. Three steps, one small brain.

Under the hood, Groq (running Llama 3.1) was the decision-maker, with two tools it could reach for: a calculator and a weather lookup. Every step, the LLM call, the tool call, the final answer, got wrapped in an OpenTelemetry span so SigNoz could watch it work.

I fed it a mix of honest questions and a few troublemakers designed to trip it up: math, weather, and the occasional nonsense query.

I ran it thirty times and let SigNoz record everything.

On my screen, it looked fine. Answers scrolled past. Job done. But SigNoz was watching a very different story unfold.

Hiring the detective

I didn’t want a hosted service watching over my shoulder. I wanted my own detective, on my own turf. So I set up the stakeout myself.

A few minutes of setup, and SigNoz was live and watching. No fancy office needed. Just a detective, a dashboard, and an agent that had no idea it was being observed.

The detective, reporting for duty.

I pointed my agent at it and let the interrogation begin.

The first clue

The detective didn’t waste time. It opened with APM, the Services view, the agent’s vitals, built automatically from the traffic.

And the truth was ugly.

Nearly five seconds a request, and more than half of them failing. On my screen the answers had scrolled by so fast that I never noticed the wreckage underneath. The agent looked calm. The numbers said otherwise.

Then I leaned in closer, and found the detail that cracked the case open.

Thirty requests, more than half of them failing. But the detail told a sharper story. The LLM calls never errored, not once. The failures were all in the calculator, choking 72% of the time on the messy input the model kept feeding it. The brain was fine. The hands were the problem. I’d never have found that by reading my code.

One look, and the suspect already had nowhere left to hide.

The smoking gun

Knowing the agent failed wasn’t enough. I needed to see inside a single request. So I opened Distributed Tracing, and every step the agent took was right there, timed to the millisecond.

Then I clicked into one request and opened the waterfall. This is where SigNoz stops being a dashboard and becomes an x-ray.

There it was, laid out like a confession. A single request took nearly five seconds, and the two LLM calls ate 2.21 and 2.28 seconds of it. Together, the model was responsible for about 93% of the entire request. The tool I’d been suspicious of took a harmless 329 milliseconds.

I’d been ready to blame the tools, the network, my own code. The waterfall pointed straight past all of them to the real time thief: the model calls themselves.

And it went deeper. Click any span and the side panel spilled everything, the exact query, the agent’s decision, the tokens used.

user.query: “Weather in Tokyo?” agent.decision: “WEATHER:Tokyo.”

I wasn’t just seeing that a step took two seconds. I was seeing what the agent was thinking during those two seconds. That’s the difference between watching a black box and reading its mind.

The waterfall didn’t just show me the crime. It showed me the motive.

The confession

I had the timing. Now I wanted the failures. So I told the detective to show me only the requests that broke.

The pattern was immediate. The failing spans were all tool.calculator, dying in fractions of a millisecond. The calculator wasn't slow. It was choking. But on what?

So I opened one of the broken traces and let SigNoz read me the confession.

user.query: “Calculate banana * 3” — exception.message: “invalid syntax”

There was the whole crime in one screen. The user asked to “calculate banana 3.” The LLM, trying to be helpful, passed `banana 3to the calculator as if it were real math. The calculator raneval()` on it, and Python did exactly what you'd expect: it threw a SyntaxError.

And SigNoz didn’t just say “it failed.” It handed me the exact exception message, the full stack trace, the precise line in my code, and the query that triggered it. Everything I needed to fix it, without adding a single line of error-tracking myself.

This is what makes Exceptions in SigNoz different. It doesn’t bury failures in a wall of logs. It groups them, counts them, and pins each one to the exact moment it happened. My thirty-request run had produced a tidy little rap sheet: a pile of SyntaxErrors from the calculator, and a couple of stray NameErrors where the model had invented the word "undefined."

The agent hadn’t been mysteriously broken. It had been feeding garbage to its own tools, over and over, and smiling while it did it.

Case closed on the how. But there was still more it hadn’t told me.

The wiretap

The traces told me what happened and when. But I wanted to hear the agent think out loud. So I opened the Logs.

The agent’s thoughts, line by line: “handling: Calculate banana * 3” then “agent failed: invalid syntax.” And buried below, something I never asked about.

Every decision was there in plain text, tied to the exact request that produced it. I could read the agent’s reasoning like a transcript. But two lines stopped me cold:

HTTP 429 Too Many Requests. Retrying request in 2.0 seconds.

My model provider had been rate-limiting the agent, and the client was quietly retrying in the background. I never saw it. It never threw an error I noticed. The agent had been silently waiting and retrying, quietly padding my request times, and I’d have blamed the model for being “slow.”

That’s what a wiretap catches: the conversations you didn’t know were happening.

The Logs view didn’t just store text. It tied every line back to its trace, so I could jump from a slow request straight to the log that explained it. One filter turned a wall of noise into a clean confession.

The agent had been keeping secrets at every layer. SigNoz kept finding them.

Following the money

Every investigation follows the money. For an AI agent, the money is tokens. Every call to the model costs them, and tokens turn into real dollars fast. So I wired my agent to report its own usage, and opened Metrics.

Two custom metrics, tracked automatically: agent.requests and agent.tokens.used.

These weren’t built-in numbers. They were custom counters I emitted from the agent, and SigNoz picked them up instantly, no config, no dashboards required. It even tagged them by model, so I could see exactly which model was burning what.

But a table is just numbers. I wanted to see the spend. So I dropped agent.tokens.used onto a Dashboard panel using the Query Builder.

Token usage over time, climbing to nearly 4000. The agent’s appetite, drawn in one line.

There it was: my agent’s token consumption, rising in real time. In one panel I could watch exactly when the agent got expensive and how fast. String a few of these together, latency, tokens, error count, and you’ve built a cockpit for your agent. One screen, whole story.

This is the part people skip until the bill arrives. With SigNoz, I could see the cost climbing while it happened, not a month later on an invoice.

The stakeout

Case solved. But a good detective doesn’t just close the file and walk away. The agent failed once because nobody was watching. So the detective stayed on the case, waiting for it to slip again.

I opened Alerts and set the trap: watch the token usage, and the moment it crosses the line, sound the alarm.

The trap is set. “High Token Usage.” That red line is the trigger. Cross it, and SigNoz makes the call.

That red dashed line is the boundary. The second my agent crosses it, SigNoz fires a notification, to Slack, a webhook, wherever I’m watching. I don’t have to sit and stare at a dashboard. The detective keeps the stakeout so I don’t have to.

Point it at anything: latency creeping up, errors spiking, tokens running hot. It’s the difference between finding out your agent broke because you went looking, and finding out because SigNoz told you, the moment it happened.

The agent lied to me once because I had no way to hear it. Now the detective never blinks. If it slips again, I’ll be the first to know.

The detective’s other tricks

I’d cracked the case with the basics. But going through SigNoz, I realized my detective was carrying tools I’d barely touched. Any one of them could break open a bigger case.

The interrogation bot (MCP). SigNoz has an MCP server, which means you can hand your telemetry straight to an AI assistant and just ask it questions. “Which span was slowest?” “What failed in the last hour?” Instead of hunting through dashboards, you let an AI read the evidence for you. For an agent hackathon, this is the dream: an AI that debugs your AI.

The connection map (Service Map). As your agent grows into many services, an LLM router, a vector database, a dozen tools, the Service Map draws the whole web automatically and lights up the connection that’s failing. One glance shows you where the trouble lives.

The perimeter watch (Infrastructure Monitoring). My agent was software, but software runs on machines. SigNoz also watches the ground it stands on: hosts, containers, CPU, memory, Kubernetes, all in the same place as your traces and logs. When an agent slows down, it isn’t always the code. Sometimes the machine underneath is choking, and this is where you’d catch it.

The drop-off tracker (Trace Funnels). Funnels follow a request through a sequence of steps and show you exactly where it leaks. For a multi-step agent, that means seeing which stage silently loses requests, before your users do.

The case file bookmarks (Saved Views). Every filtered view I built, errors only, slow requests, token spikes, I could save and reopen in one click. No rebuilding the same query twice. The detective keeps its files organized.

The evidence board (Correlation). This is the trick that ties it all together. From a single slow trace, I could jump straight to its logs, then to the metric spike behind it, without ever switching tools or losing my place. Traces, logs, and metrics stop being three separate piles of clues and become one connected board with red string running between them. For debugging an agent, that’s everything: you follow the thread from symptom to cause in seconds.

I only needed a few of these to catch my agent. The rest are waiting for a bigger case. And next week, at the real hackathon, they’ll get one.

Case closed. For now.

I started this as a warmup. Interrogate one agent, learn the detective’s tools, before the real hackathon began. What I got was a lesson I didn’t expect.

My agent was never “broken” in the dramatic way I imagined. It was worse. It was quietly wrong, over and over, wearing a confident smile the whole time. Failing more than half its requests. Burning twice the model calls I asked for. Getting rate-limited in silence. And on my screen, none of it showed. It looked fine.

That is the real danger with AI agents. They don’t crash loudly. They lie softly. And you only find out when a user does, or when the bill arrives.

Self-hosting SigNoz changed that in an afternoon. Traces, logs, metrics, exceptions, alerts, all in one place, watching everything my agent did. The black box got glass walls.

If you ask me which tool did the most damage to my agent’s lies, it’s the distributed tracing waterfall. It was the one feature that turned “something is wrong” into “here is exactly where, and why.” Watching 93% of a request vanish into model calls, in one glance, is the moment observability stopped being a buzzword for me. Everything else told me the agent failed. The waterfall showed me the whole crime, span by span.

The warmup is over. The detective is sharp, the tools are loaded, and I know exactly how to see inside an agent now.

Next week, at the Agents of SigNoz hackathon, I’m not building blind. I’m bringing the detective.🕵