How a 5-Phase RAG Pipeline Slashed My LLM Token Usage by half

Most RAG systems retrieve 10–15 chunks and send them all to the LLM. In my testing, more than half of those tokens contributed nothing to the answer. Here’s what I built to cut them.

5-Phase RAG Pipeline for Token efficient RAG

Full code on GitHub.

The biggest latency bottleneck I found wasn’t the model or the infrastructure — it was the volume of irrelevant tokens going to the LLM. Context was hitting 8K-12K tokens per query. After profiling, most of it was noise: duplicate content, low-relevance chunks, verbose formatting. The LLM was burning tokens that didn’t help the answer.

The fix: a 5-phase optimization pipeline that reduces context to under 4K tokens — a 50–60% reduction in my testing — while keeping the content the LLM actually needs.

Why Raw Chunks Waste Tokens

The standard RAG approach retrieves 10–15 chunks, concatenates them, and sends them to the LLM. The problem: most of those chunks aren’t equally useful. Some are duplicates. Some are loosely related but not relevant to the specific query. Some contain verbose formatting that inflates token count without adding information.

More chunks doesn’t mean better answers. It means more noise for the model to process, slower responses, and higher costs.

The 5-Phase Pipeline

The 5-Phase Funnel — Context Optimization

I built a *ContextWindowOptimizer* that runs five phases in sequence, each building on the previous one:

# context/context_optimizer.py — 5-phase pipeline
# Phase 1: Intelligent Truncation — remove duplicates, keep critical content
# Phase 2: Semantic Chunking — split on meaning boundaries, not character counts
# Phase 3: Relevance Filtering — score each chunk, drop below 0.7 threshold
# Phase 4: Dynamic Assembly — adapt chunk count to query type
# Phase 5: Hierarchical Loading — load critical content first, lazy-load the rest

Target: under 4,000 tokens per query. Here’s what each phase does.

Phase 1: Intelligent Truncation

Strip the obvious waste before doing any expensive processing. This was the easiest win.

The truncator scans the raw context and applies simple rules: keep system prompts, instructions, and recent conversation turns (last 5). Drop duplicate lines (tracked via normalized content hashing). Drop verbose error messages. Keep structured data like JSON and tables.

# context/context_optimizer.py
# Patterns that always survive truncation:
critical_patterns = [
    "You are", "SYSTEM:", "Instructions:",
    "IMPORTANT:", "WARNING:", "CRITICAL:",
    "### ", "## ", "# ",
]

This phase is fast — string operations, no model calls. It handles the low-hanging fruit: content that’s clearly redundant or clearly critical.

Phase 2: Semantic Chunking

After truncation, the remaining text gets split into chunks that respect meaning boundaries.

The *SemanticChunker* classifies each chunk by domain type — hurricane, forecast, alert, evacuation, temperature, wind. This classification feeds into later phases: when the system needs to prioritize safety content, it knows which chunks are which.

The key difference from naive chunking: splits happen at paragraph and sentence boundaries, not at arbitrary character counts. A chunk about evacuation zones stays intact instead of getting cut in half. Early versions of this system split on character count and the results were noticeably worse.

Phase 3: Relevance Filtering

This is where most of the token savings happen. If I had to pick one phase to implement first, it’s this one.

The *RelevanceFilter* scores every chunk against the query using two paths: semantic scoring (cosine similarity between query and chunk embeddings) or keyword scoring (term overlap fallback when embeddings aren’t available). Chunks below the 0.7 threshold get dropped.

# context/relevance_filter.py
async def score_chunks(
    self, query: str, chunks: list[str]
) -> list[dict[str, Any]]:
    if self.embeddings is not None:
        try:
            return await self._semantic_scoring(query, chunks)
        except Exception as e:
            return self._keyword_scoring(query, chunks)
    else:
        return self._keyword_scoring(query, chunks)

Both paths apply a domain boost. High-signal weather terms (“hurricane,” “evacuation,” “storm surge”) get +0.15. Medium-signal terms (“forecast,” “temperature”) get +0.05. Location terms get another +0.05. This way, domain-critical content survives filtering even if its raw similarity score is borderline.

I set 0.7 as the threshold after testing a few values. At 0.6, too much noise survived. At 0.8, I was dropping chunks the LLM genuinely needed. Your domain may land differently — worth validating before hardcoding it.

Phase 4: Dynamic Assembly

Dynamic Assembly — Query Type vs. Context Budget

Not every query needs the same amount of context. This took me a while to internalize. The *DynamicAssembler* sets chunk limits and token budgets based on query type:

# context/dynamic_assembler.py
CHUNK_LIMITS = {
    "SIMPLE": 2,      # "What's the temperature?"
    "STANDARD": 5,    # Normal weather queries
    "COMPLEX": 8,     # Multi-part analysis queries
    "EMERGENCY": 15,  # Life-safety queries
}

TOKEN_BUDGETS = {
    "SIMPLE": 0.5,    # 50% of target tokens
    "STANDARD": 0.8,  # 80% of target tokens
    "COMPLEX": 1.0,   # 100% of target tokens
    "EMERGENCY": 1.2,  # 120% — safety gets extra room
}

A simple query gets 2 chunks and 50% of the token budget. An emergency query gets 15 chunks, 120% of the budget, and 2x priority weight on safety-related content. The assembler also sorts chunks by a combined score of relevance and semantic type priority — evacuation chunks surface first for emergency queries, regardless of raw similarity.

Phase 5: Hierarchical Loading

Hierarchical Loading Budget Distribution

The final phase decides what loads first.

The *HierarchicalLoader* classifies content into three tiers:

• CRITICAL — system prompts, safety warnings, user query (loads immediately)
• SECONDARY — forecasts, conditions data (loads if budget allows)
• OPTIONAL — background context (lazy-loaded only for complex queries)

Budget distribution changes by query type:

# context/hierarchical_loader.py
BUDGET_DISTRIBUTION = {
    "SIMPLE":    {"CRITICAL": 0.7, "SECONDARY": 0.3, "OPTIONAL": 0.0},
    "STANDARD":  {"CRITICAL": 0.5, "SECONDARY": 0.4, "OPTIONAL": 0.1},
    "COMPLEX":   {"CRITICAL": 0.4, "SECONDARY": 0.4, "OPTIONAL": 0.2},
    "EMERGENCY": {"CRITICAL": 0.6, "SECONDARY": 0.3, "OPTIONAL": 0.1},
}

For a SIMPLE query, 70% of the budget goes to critical content, 30% to secondary, and nothing optional loads at all. For COMPLEX queries, optional content gets 20% of the budget because multi-part questions need broader context.

The Caching Layer

The pipeline reduces tokens. Caching avoids recomputing.

• Application-level caching. The *EmbeddingCache* implements an in-memory LRU cache with an optional Redis backend. Keys are SHA-256 hashed, TTL is configurable (default: 1 hour). Check memory first, fall back to Redis, compute only on miss.
• API-level prompt caching. Anthropic’s *cache_control* feature caches static content at the API level. Per Anthropic’s published pricing, cache writes cost 1.25x the base price, but cache reads cost just 0.1x — so after the first request, every subsequent request with overlapping context is 90% cheaper on those tokens.

# cache/l3_anthropic_cache.py
system_blocks.append({
    "type": "text",
    "text": base_system_prompt,
    "cache_control": {"type": "ephemeral"},  # 5-min TTL
})

Token reduction plus caching compounds: fewer tokens processed, and lower cost per token on cache hits.

Measuring It

I added three Prometheus metrics to track the pipeline in production:

• *context_token_reduction_percent* — histogrammed by query type, bucketed at 10–80%
• c*ontext_optimization_latency_seconds* — how long the pipeline takes
• *context_optimizations_total* — count by query type and success/error status

Without these, you’re optimizing blind. The histogram buckets tell you whether most queries hit the 50–60% reduction target or whether certain query types underperform.

When NOT to Use This

A few situations where raw chunks are fine:

• Fewer than 3 chunks. The pipeline overhead isn’t worth it. Send them directly.
• Highly cohesive documents. If all chunks come from a single source, there’s less redundancy to cut.
• Legal or medical precision. When every word matters and filtering risks losing a critical qualifier. “Usually safe” and “safe” are different sentences in a courtroom.
• Early prototypes. Ship the simple version first. Optimize when your latency data tells you to.

Where to Start

Three changes, in order of impact:

1. Add relevance filtering. Score each chunk against the query, drop anything below 0.7. This alone cuts the most tokens. Add domain-specific boosting so high-value content doesn’t get filtered by accident.

2. Make chunk counts a function of query type. Don’t send 10 chunks for every query. Simple questions need 2. Complex ones need 8. A lookup table costs nothing; mismatched context costs answer quality.

3. Add hierarchical loading. Classify content into critical, secondary, and optional tiers. Load critical first. Skip optional for simple queries. This keeps the LLM focused on what matters.

Full context optimization pipeline is on GitHub.