Your LLM prompts are probably wasting 90% of tokens. Here’s how I fixed mine.

Your LLM prompts are probably wasting 90% of tokens. Here’s how I fixed mine.

I keep running into the same problem with LLM apps.

You build a retrieval pipeline, hook it up to an API, and then quietly ship prompts that are full of stuff the model doesn’t need. Extra chunks. Duplicates. Half-relevant context that just bloats everything.

This work is an extension of my previous work: Context-Fusion

https://medium.com/@rotsl/contextfusion-the-context-brain-your-llm-apps-are-missing-c2fd9e632b99

And you pay for all of it.

CFAdv is basically an attempt to stop doing that.

It builds on context-fusion, but adds something that turns out to matter more than I expected: even if you pick the right context, you can still mess it up by putting it in the wrong place.

Most pipelines are still doing this

chunks = retriever.top_k(query, k=5)
prompt = "\n\n".join(chunks)
response = llm(prompt)

Let’s be honest about the default pattern:

That’s it.

No budget. No filtering beyond retrieval. No thought about ordering.

More context is assumed to be better. It often isn’t.

CFAdv splits the problem in two

Instead of one “context step”, it does two separate things:

1. Decide what gets in

2. Decide where it goes

That separation is the whole point.

Step 1: selecting context under a budget

Instead of top-k, CFAdv treats selection like an optimization problem.

Each chunk gets a score based on things like relevance, trust, freshness, diversity, and token cost. Then the system tries to pick the best combination under a fixed token budget.

At a high level:

def value(chunk):
    utility = (
        0.25 * chunk.relevance +
        0.20 * chunk.trust +
        0.15 * chunk.freshness +
        0.15 * chunk.structure +
        0.15 * chunk.diversity
    )
    risk = (
        0.40 * chunk.hallucination +
        0.35 * chunk.staleness +
        0.25 * chunk.privacy
    )
    return utility - risk

density = value(chunk) / max(chunk.tokens, 1)

The small trick that makes a big difference

There’s a simple filter before any of that:

floor = max_score * 0.15
selected = [c for c in candidates if c.score >= floor]

Anything below 15% of the best chunk just gets dropped.

That sounds minor, but it changes behavior a lot.

If your data is clean, everything stays. If it’s noisy, most of it disappears. So you don’t fill your prompt with mediocre content just because you have space.

Step 2: ordering for attention

This is the part I underestimated.

Even if you pick the right chunks, models don’t treat all positions equally. Stuff at the start tends to get more attention than stuff buried in the middle.

So CFAdv reorders the selected chunks based on similarity to the query.

Basic version:

def cosine(a, b):
    return (a @ b) / (norm(a) * norm(b))

scores = [cosine(embed(query), embed(chunk)) for chunk in chunks]
weights = softmax(scores)

ordered = [chunk for _, chunk in sorted(
    zip(weights, chunks),
    reverse=True
)]

Higher weight goes earlier in the prompt.

No embeddings API required

Instead of calling an external model, it uses a simple hashed bag-of-words vector.

import hashlib
import numpy as np
import re

def embed(text, dim=64):
    vec = np.zeros(dim)
    tokens = re.findall(r"\b\w+\b", text.lower())

    for t in tokens:
        h = int(hashlib.sha256(t.encode()).hexdigest(), 16)
        vec[h % dim] += 1.0
        vec[(h >> 16) % dim] += 0.5

    return vec / (np.linalg.norm(vec) + 1e-8)

It’s not fancy. No positional info, no learned weights. But for short chunks it works surprisingly well.

Two levels of ordering

There’s also a second layer.

Instead of treating everything as one list, CFAdv groups context into blocks like system, history, retrieval, and tools.

Then it does two passes:

First, it sorts chunks inside each block.

Second, it sorts the blocks themselves.

Sketch:

# intra-block
for block in blocks:
    block.chunks.sort(key=lambda c: similarity(query, c), reverse=True)

# cross-block
block_scores = {
    block: similarity(query, mean_embed(block.chunks))
    for block in blocks
}

ordered_blocks = sorted(blocks, key=lambda b: block_scores[b], reverse=True)

So you’re not just shuffling pieces. You’re reshaping the whole prompt.

The full pipeline

CFAdv is an 8-stage pipeline, but it’s easier to think of it like this:

docs = ingest(files)
blocks = normalize(docs)
variants = represent(blocks)

candidates = retrieve(query, variants)
selected = plan(candidates, budget=120)

ordered = attention_fuse(query, selected)
packet = assemble(ordered)

prompt = compile(packet, mode="qa")

Each step is stateless, which makes it easier to test and reason about.

What happens in practice

You can cut most of the prompt without losing the answer, as long as:

• retrieval pulls in some noise

• there is redundancy

• the query only needs a subset of the data

If everything is relevant, the system mostly leaves it alone.

If only one chunk survives selection, ordering doesn’t matter.

Where this actually helps

This kind of pipeline shines when:

• your retrieval step is messy

• you’re concatenating multiple documents

• prompts are long enough for attention effects to matter

If you already have clean, minimal context, you won’t see much change.

The part I keep coming back to

This isn’t really about attention or embeddings.

It’s about treating prompt assembly as something worth optimizing.

Right now most systems act like prompts are just containers. You throw things in and hope the model figures it out.

CFAdv flips that.

It asks a simple question: what is the smallest amount of context that still works?

Then it enforces it.

And once you start thinking that way, it’s hard to go back to dumping chunks into a string and calling it a day.

Try it yourself

If you want to see how this works in practice or plug it into your own workflow:

GitHub repo:

https://github.com/rotsl/CFAdv

Contains the full Python library, CLI, benchmarks, and tests. You can run it locally, inspect the pipeline stages, or integrate it into your own RAG setup.

Live demo:

https://rotsl.github.io/CFAdv/

Lets you compare raw prompts vs CFAdv-compiled prompts side by side. Useful for quickly seeing how much context gets removed and how ordering changes.

If you’re already using retrieval plus concatenation, the repo is the easiest place to start. Swap your prompt assembly step with CFAdv’s planner and fusion stages and see what drops out.