Why your “$0.002 per 1K tokens” mental model is lying to you

Image generated with AI

LLMs charge per token, prompts in, responses out, and output tokens are usually 2–5x more expensive than input.

On paper, that looks cheap: a few tenths of a cent per call. At a small scale, it is. At workflow scale, it isn’t.

Once you wire models into agents, tools, RAG, and guards, you’re no longer paying for “one call.” You’re paying for the conversation between services that happens under the hood.

Think of it like roaming charges in the early days of mobile. You don’t feel it on a single call. You feel it on your bill.

What actually happens in a “simple” agent workflow

A single “answer this question” interaction can easily look like this:

• Call 1 — Policy/safety model (small, cheap)
• Call 2 — Routing model: “Which tool or specialist model should handle this?”
• Call 3 — Retrieval: one or more LLM calls to build the search query, then vector DB, then expansion
• Call 4 — Main reasoning model (bigger, more expensive)
• Call 5 — Verification model to critique/refine the answer
• Call 6+ — Observability & evaluation calls for quality scoring, guardrails, logging

SemiAnalysis and others have been pointing out that every extra reasoning step, retrieval hop, and tool call multiplies token usage, even if you’re using the same base model

Your user sees one response. Your cloud bill sees six to ten model invocations.

Here’s where it gets counterintuitive:

Cheap models might be 10–50x less expensive per token than frontier models

But if you need three cheap calls plus retries, plus a separate guard model, plus a RAG helper, that “cheap” path may end up costing the same (or more) than one well‑prompted call to a higher‑quality model.

SemiAnalysis has shown similar dynamics on the consumer side: flat‑price ChatGPT plans often drive usage that would cost tens of thousands at raw-API pricing.

The same logic flips for you as the buyer of APIs: structure beats sticker price.

Per‑interaction economics: the only number that really matters

If you’re serious about AI as a product capability, the unit isn’t “tokens per month.”

It’s the cost per successful outcome.

A few examples of “per interaction” units that actually make sense:

• Cost per reconciled invoice
• Cost per resolved support ticket
• Cost per sales email that passes QA
• Cost per KYC file fully reviewed

Practitioners are starting to do exactly this: trace tokens, models, and infra back to a cost per completed workflow, not just a monthly spend line.

Once you do that, patterns jump out:

• 10–20% of workflows drive 70–80% of your spend
• A single over‑sized model or verification loop silently doubles per‑interaction cost
• Some “fancy” multi‑agent workflows add complexity without changing the outcome rate

That’s your hidden cost shock, quantified.

Where the money actually leaks

Over‑orchestration

Multiple agents talking to each other when one stateful agent (or even plain code) would do.

“Just in case” verification passes that run every time instead of on sampled traffic.

One‑size‑fits‑all models

Using your biggest model for every step, including things like formatting, extraction, or light classification that a small model, or deterministic code, could handle.

Retries and fallbacks hidden in the platform

Silent retries on timeouts or low confidence, each one a full extra call, and fallback chains that walk down three providers if the first one fails.

Chatty prompts and context windows

Over‑stuffed RAG prompts that drag in 20 chunks “just to be safe” and logging full prompts and responses for observability; debug mode accidentally left on in prod

Together, they’re the reason a workflow you thought would cost $0.005 per interaction is showing up as $0.05–0.10.

Designing for “good enough” rather than “max everything”

The quickest way to regain control is to separate where quality truly matters from where “good enough” is fine.

A pragmatic pattern that’s emerging:

Use a small, fast model (or plain code) for:

• Intent detection
• Schema validation, extraction, and routing
• Simple transformations and formatting

Use your best model only for:

• Non‑routine reasoning
• User‑visible responses in high‑value flows
• Edge cases you’ve explicitly identified

Use evaluation and guardrails selectively:

• Always‑on for sensitive/regulated actions
• Sampled or on‑demand for low‑risk flows

That’s how to shift from “multi‑model chaos” to a layered model strategy instead of just stacking calls until something works.

A minimal cost‑aware checklist for builders

If you want the ah‑ha compressed into something you can actually use tomorrow, here’s my short list:

For every workflow, ask:

• “How many model calls does a single user interaction really trigger?”
• “Which of those steps could be handled by a smaller model or pure code?”
• “Where are we doing work ‘just in case’ rather than because of a real risk?”

Instrument two numbers per flow:

• Tokens (and $) per interaction
• Cost per successful business outcome
• Default design choices:

Start with single‑agent workflows and add agents only when you can show measurable uplift.

• Keep prompts and context windows intentionally small; prove you need the extra tokens.
• Make retries, fallbacks, and verification configurable, not hard‑coded.

Do this for a month, and you’ll likely discover that your biggest savings have nothing to do with negotiating a better price per million tokens, and everything to do with how you architect each interaction.