You’re Burning Money on Tokens and Getting Worse Results. Here’s Why

Photo from Financial Time

Last quarter, our ML platform team got an alert that our OpenAI spend had jumped 340% month-over-month. No new features. No traffic spike. Same number of users.

We dug in. Turns out one engineer had refactored our summarization pipeline to include the full conversation history on every single API call “for better context”. The intention was not wrong. He was wrong about the cost. We were sending 18,000 tokens per request for a task that needed 800. The model wasn’t getting smarter. We were just paying more for the same answer.

That incident is what made me start taking token efficiency seriously as an engineering discipline, not a billing footnote.

The Real Cost Isn’t the Invoice

Most teams look at token costs as a line item. They shouldn’t. The real cost is threefold: money, latency, and quality — and the third one surprises people.

Here’s the counterintuitive part: more tokens in does not mean better output. Past a certain threshold, large context windows actively hurt model performance. This is called the lost-in-the-middle problem, documented in a 2023 Stanford paper. When you stuff 80K tokens into a context window and your relevant content is buried at position 40K, the model’s retrieval accuracy on that content degrades significantly.

We saw this firsthand when we were building a RAG pipeline for our internal documentation search. Increasing chunk retrieval from top-5 to top-20 documents reduced answer quality on specific factual questions by roughly 15%, even though we were technically giving the model more information. The noise drowned the signal.

The mental model to fix in your head: a context window is not a bucket you fill up. It’s a spotlight. You’re directing attention, not uploading data.

The Seven Things Teams Actually Get Wrong

I want to be specific here, because most advice on this topic is generic to the point of useless.

1. Repeating the system prompt logic in every user message

I see this constantly. Teams write a 2,000-token system prompt that establishes persona, output format, and constraints — and then rephrase half of it in the user message “just to be safe.” The model doesn’t need reminding. You’re paying twice for the same instruction, and often creating contradictions that degrade output consistency.

Trust your system prompt. Test it. If you don’t trust it, fix it — don’t patch it with redundant user-turn instructions.

2. Sending raw, unprocessed documents

A PDF gets uploaded. Someone passes the full extracted text — headers, footers, page numbers, repeated boilerplate, OCR artifacts — directly into the context. A 40-page contract becomes 22,000 tokens of noise with 6,000 tokens of actual content buried inside it.

Pre-process aggressively before you touch an API. Strip boilerplate. Remove repeated headers. Extract only the sections relevant to the task.

We built a preprocessing step in our document pipeline that reduced average input tokens by 61% with zero impact on output quality.

# Before: naive extraction
text = extract_all_text(pdf)  # 22,000 tokens

# After: targeted extraction
sections = extract_sections(pdf, target=["obligations", "termination", "SLA"])
text = "\n\n".join(sections)  # 5,800 tokens

3. Using a frontier model for a task that doesn’t need one

GPT-4o costs roughly 20x more per token than GPT-3.5-turbo or Claude Haiku. For classification tasks, reformatting, simple extraction, and short-form generation, the smaller models are within 5% of quality and a fraction of the cost.

We run a tiered routing system: simple intent classification and slot-filling goes to Haiku. Complex reasoning, nuanced generation, and multi-step tasks go to Sonnet or GPT-4o. That single decision cut our monthly inference bill by roughly $4,200 without a drop in user satisfaction scores.

Route by complexity. Don’t send a straightforward JSON extraction task to a frontier model because it’s the default.

4. No output length control

If you don’t tell the model how long to be, it’ll decide for you. And it’s not optimizing for your token budget.

Set max_tokens explicitly on every call. Better — tell the model in the system prompt what length is expected. "Respond in 3 sentences or fewer" isn't style guidance; it's cost control.

For structured outputs, use JSON mode or function calling. A clean JSON object beats a narrative response that you then parse anyway.

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=300,  # explicit ceiling
    system="Extract the key action items. Return as JSON array. Max 5 items.",
    messages=[{"role": "user", "content": document_text}]
)

5. Conversation history that grows unbounded

Multi-turn chat applications have a death spiral: every turn, you append the full history. By turn 15, you’re sending 12,000 tokens of context to get a response to a question that needs 400. By turn 30, you’re near the limit and the model has started ignoring the early turns anyway.

The fix isn’t removing history — it’s summarizing it.

After every N turns, run a cheap summarization pass with a small model, compress the earlier turns into a paragraph, and replace them.

Anthropic actually describes a variation of this in their documentation. We implemented it with a rolling window of 6 turns plus a running summary, and our average conversation cost dropped by 58%.

6. Prompt templates with dead weight

Most prompt templates accumulate cruft. They start lean, someone adds a clarifying sentence after an edge case, then a long list of edge case prompts that fire 2% of the time and consume tokens 100% of the time.

Audit your system prompts quarterly. Token-count them. Ask: is every sentence here load-bearing for the common case?

Edge cases belong in conditional logic at the application layer, not baked into a prompt that runs on every request.

7. No caching strategy

Anthropic, OpenAI, and Google all support prompt caching now. If your system prompt is 4,000 tokens and you’re firing 50,000 requests a day, you’re paying for those 4,000 tokens 50,000 times — unless you’re using cache.

With prompt caching enabled, repeated prefix tokens cost a fraction of uncached tokens (roughly 10%).

This is free money. Enable it. The implementation is usually two lines.

The Principle Behind All of It

Every one of these mistakes comes from the same root assumption: that more context always helps.

It doesn’t. The model performs best when you give it exactly what it needs, nothing more. That discipline — knowing what to include and what to cut — is a skill. It’s the same skill as writing a good function signature, or a tight SQL query. You’re defining an interface between your application and the model. Garbage in, garbage out applies here just as much as it does anywhere else in engineering.

Think of your prompt like a database query. You don’t SELECT * and filter in application code. You specify exactly the columns you need. Same principle.

Effective Token Architecture:

System prompt     → stable, cached, load-bearing only
Retrieved context → preprocessed, scored by relevance, top-K only
Conversation      → rolling window + compressed summary
User input        → unchanged
Output            → max_tokens set, format specified

What to Do Before Your Next Sprint Ends

Pick one API call in your codebase that fires frequently — your most common user-facing prompt. Token-count it. Break it down: system prompt, injected context, conversation history, user input. Find the biggest component. Ask whether it could be 40% smaller without changing the output.

Run the leaner version against your eval set if you have one. If you don’t have an eval set, that’s a separate problem — but you can still eyeball 20 examples.

I’ll bet you find at least one of the seven mistakes above. Fix that one. Then look at the next call.

Token efficiency is boring until your bill arrives. Make it a habit before that moment, not after.