Reducing LLM costs: personal hacks and production architecture

TL;DR: every token you send to an LLM has a cost. The strategies to reduce that cost look completely different depending on whether you’re running an agent for personal use or a multi-tenant production system. This post covers both ends of the spectrum: from talking like a caveman to lazy data fetching via MCP.

The problem scales with usage

When you’re experimenting, LLM costs are noise. When you’re running agents in production or even just a personal assistant that runs all day they’re not. The interesting thing is that the right strategy depends almost entirely on who controls the output and who reads it.

For a personal assistant or a coding agent, you control the full loop, you are the end user and can be aggressive using compression.

For a user facing application, serving multiple users, the output goes to people who expect readable and coherent response. In this case compression could break the product quality.

Part 1: the personal assistant approach

Caveman: compress the output

The idea behind caveman is simple: strip everything that is grammatically predictable from the model’s responses and keep only the words that carry actual meaning.

I used a skill inspired by this approach, available on OpenClaw: clawhub.ai/skaravind/caveman. Not the original tool but a skill that implements the same principle

To activate it: “caveman mode”. To deactivate: “stop caveman”.

A simple example. Query: “weather pozzallo this week”.

RTK: compress before the call

RTK is a CLI proxy that reduces token consumption by 60–90% on common dev commands. It works by intercepting terminal command outputs — git log, ls, catand stripping the parts that don't carry information before they reach the LLM

When this approach works

Especially for caveman the output is never user-facing. That makes aggressive compression acceptable. The tradeoff is legibility for cost a tradeoff you can make for yourself but not for your users.

Part 2: production approach

You can’t instruct a customer-facing agent to talk like a caveman. The strategies here are structural: you can change what you send, when you send it and whether you send it at all.

Don’t send what you don’t need: the MCP lazy fetch pattern

Most MCP tools are thin wrappers around API calls: fetch everything, return everything. For a spreadsheet with 500 rows or a Gmail inbox with 200 emails, most of what enters the context is noise the model will never use.

Workpiece built a different approach around the “4KB rule”: any tool result over 4KB goes to Redis, not to the LLM. The data is fetched once, the LLM receives a summary with a reference ID, and queries the cache with precise filters when it needs something specific.

tool result > 4KB → store in Redis → LLM gets summary + payload_id
tool result < 4KB → LLM gets the data directly

The full dataset never enters the context. The API is called once, the cache is queried many times

Semantic caching: don’t pay for the same answer twice

If the lazy fetch pattern is about not sending data you don’t need, semantic caching is about not paying for a response you’ve already generated.

The idea: convert incoming queries to vector embeddings, measure cosine similarity against cached queries, and return the cached response when similarity exceeds a threshold . “what’s your return policy?” and “how do I return something?” are the same question. Traditional caching: two API calls. Semantic caching: one.

LiteLLM proxy supports this natively with a config change

litellm_settings:
  cache: true
  cache_params:
    type: qdrant-semantic
    qdrant_api_base: os.environ/QDRANT_API_BASE
    qdrant_api_key: os.environ/QDRANT_API_KEY
    qdrant_collection_name: llm_cache
    similarity_threshold: 0.85
    ttl: 3600
    qdrant_semantic_cache_embedding_model: text-embedding-ada-002
    qdrant_semantic_cache_vector_size: 1536

tools

**JuliusBrussee/caveman** — Claude Code skill, 75% token reduction on agentic sessions. personal/dev use only.

**rtk-ai/rtk** — CLI proxy, 60–90% reduction on dev commands. Rust, zero dependencies.

**LiteLLM — **LLM Gateway

references

• JuliusBrussee (2025). caveman. github.com/JuliusBrussee/caveman
• rtk-ai (2025). RTK. github.com/rtk-ai/rtk
• Lucas, T. (2026). Under the hood @ Workpiece. medium.com/workpiece-ai
• LiteLLM docs — caching. docs.litellm.ai/docs/proxy/caching