Why Is Precise Tracking of LLM Usage a Challenge?

Disclaimer: I’m not a deeply technical person. This article is driven by curiosity, hands-on experience, and many conversations with people who professionally build AI systems.

Over the past year, I’ve worked extensively with AI tools — writing text, creating presentations, building web pages, generating content, and automating communication. Like many users, I was struck by how affordable and powerful these tools had become.

For a modest monthly fee, you can now access capabilities that previously required trained professionals.

But over time, I noticed something unsettling.

The same AI subscription that easily handled my workload earlier in the year suddenly felt constrained. Tasks that once fit comfortably within usage limits now consume resources much faster. Even more puzzling, nearly identical prompts could result in drastically different token usage.

Because I’ve spent years working with payments and billing infrastructure, this inconsistency stood out immediately.

A single user action in an AI product often expands into a graph of chained prompts, retries, tool calls, and background jobs. While LLM providers bill linearly by tokens, real execution paths are branching, asynchronous, and shared — making precise per-user or per-task cost attribution inherently difficult.

The uncomfortable realization

After discussions with experts and extensive reading, I realized this wasn’t a bug or pricing trick — it was structural.

LLM cost behavior depends on many variables:

• system architecture,
• prompt context,
• agent workflows,
• retries and failures,
• shared state and caching.

All of this introduces entropy into the cost of every user request.

And this led me to a practical question:

If I want to build AI agents and run them as a real business, how do I track costs per user and per task — and know when the unit economics break?

Despite plenty of advice on cost optimization, I couldn’t find a simple answer to a more basic question:

Why is detailed LLM usage tracking so difficult in the first place?

Where the difficulty comes from

Modern LLM systems are:

• multi-layered and distributed,
• billed at the token level rather than business level,
• agentic and multi-step by design,
• partially asynchronous and shared across users,
• constrained by privacy and compliance,
• probabilistic rather than deterministic.

A single user action can trigger dozens of model calls across prompt chains, background jobs, retries, and tool invocations. Meanwhile, costs are often shared, amortized, or delayed — making precise attribution approximate by design.

At some point, increasing tracking accuracy also increases system complexity, latency, and cost. Many teams deliberately stop short of perfect precision.

What this means for AI products

The difficulty of tracking LLM usage isn’t just a missing feature in existing tools. It’s a reflection of how AI systems actually work.

That doesn’t mean the problem is unsolvable — but it does mean we need:

• better abstractions for AI billing,
• product-aware cost analytics,
• and more realistic expectations around precision.

Improving this layer could significantly impact not just AI profitability, but also the broader adoption of AI in real-world businesses.

Final thought

Precise LLM tracking is hard because AI systems are no longer simple request–response machines. They are living systems — distributed, adaptive, and probabilistic.

Understanding that may be the first step toward building better AI products.

Do you feel the space of possibilities?