The LLM Compression Hack That Shrinks Prompts 5×

How a tiny “prompt-codec” layer, retrieval pruning, and rolling summaries cut tokens and cost — without hurting quality.

A practical guide to compressing LLM prompts 5× using a prompt-codec, retrieval pruning, semantic caching, and rolling conversation summaries — code included.

If you’ve ever watched a prompt balloon from 200 to 2,000 tokens because “one more guideline” felt important, you know the pain. I did, too — until I stopped writing prompts and started encoding them.

The trick isn’t a magical compression algorithm inside the model. It’s a thin gateway that rewrites what you send to the model: compact codes instead of boilerplate, only the relevant context, and a rolling summary that keeps the chat sharp. Net effect in my stack: ~5× fewer prompt tokens per call with no measurable drop in accuracy.

Let me show you the playbook.

The big idea: prompts as code, not essays

I treat prompts like network packets. Each packet has three parts:

1. Header (Rules): stable instructions (tone, safety, output schema).
2. Payload (Context): passages and facts the model must see.
3. Intent (Ask): the user’s actual request.

My prompt-codec (PCodec) turns the wordy header and repetitive scaffolding into short codes (R1, R2…S3), aggressively prunes the payload, and keeps the intent intact.

The model still receives a full, readable prompt — because the gateway expands the codes right before the API call. The user and app pass tiny strings; the LLM reads the complete text. Meanwhile we also compress the LLM-side tokens by avoiding duplicate context, deduping long rules via a rolling summary, and using retrieval pruning so we only send the top-K sentences, not whole documents.

What changed my bill (and latency)

• PCodec codes replaced a 1,200-token style/safety block with ~120 tokens (rolling summary of rules + short references).
• Retrieval pruning cut context from 1,500→250 tokens by selecting the sentences that match the query (not whole pages).
• Semantic cache answered ~25–35% of calls with zero tokens.
• Plan/execute split turned a single bloated call into two small ones (a 60–120 token planner + a 300–500 token executor).

Across a month: average prompt tokens 2,300 → 460; p95 latency down 40–50% on high-traffic flows.

PCodec in practice (tiny and boring — by design)

1) Define a compact rulebook (server-side)

I write long rules once, then distill them into a 200-token summary with numbered codes. Example:

R1: Stay factual; defer if unsure.
R2: Output JSON that matches schema S1.
R3: When summarizing, extract not paraphrase.
R4: Assume prior definitions in S* unless overridden.
…
S1: {"title": str, "bullets": [str], "citations": [str]}

The gateway stores the full text, plus this short rule index. Calls only include the index; the gateway expands (or re-summarizes) if needed.

2) Keep a rolling rules summary

Instead of pasting the 1,200-token policy every time, I keep a rolling 150–220 token summary (“R-Summary”) that the model has just seen in the thread. When the policy changes, I regenerate the summary once and pin it for N requests.

3) Send only the relevant sentences

I run BM25 or cosine similarity over sentence-level chunks (≈50–120 tokens each) and keep the top 8–12. No more sending a 1,000-token page because one line was relevant.

Code: a minimal prompt-codec gateway (Python)

This is the skeleton I actually use in front of provider SDKs. It enforces budgets, expands codes, prunes context, and consults caches.

# pcodec.py
from __future__ import annotations
import time, hashlib, json
from typing import List, Dict, Tuple

# --- toy embedding + search (swap with your vector DB) ---
def embed(text: str) -> List[float]:
    # placeholder; replace with real embeddings
    return [hash(text) % 997 / 997.0]

def similarity(a: List[float], b: List[float]) -> float:
    return 1.0 - abs(a[0] - b[0])  # fake but monotonic for demo

def top_sentences(query: str, sentences: List[str], k: int = 10) -> List[str]:
    q = embed(query)
    ranked = sorted(sentences, key=lambda s: similarity(q, embed(s)), reverse=True)
    return ranked[:k]

# --- caches ---
_exact: Dict[str, Tuple[float, str]] = {}
_semantic: List[Tuple[List[float], str, float]] = []

def hash_key(payload: dict) -> str:
    return hashlib.sha1(json.dumps(payload, sort_keys=True).encode()).hexdigest()

# --- rulebook (server-side) ---
RULES = {
    "R1": "Be factual; if unsure, say so and request a source.",
    "R2": "Output JSON that matches schema S1 exactly; no commentary.",
    "R3": "Prefer extractive summaries (quotes) to paraphrase.",
    "R4": "Keep responses under 250 words unless asked to expand."
}
SCHEMAS = {
    "S1": '{"title": "string", "bullets": ["string"], "citations": ["string"]}'
}

def rules_summary(codes: List[str]) -> str:
    parts = [f"{c}: {RULES[c]}" for c in codes if c in RULES]
    return "RULES:\n" + "\n".join(parts)

# --- main entry point ---
def pcodec_call(user_query: str, sentences: List[str], rule_codes: List[str], budget_tokens=700, ttl=900):
    # 0) Semantic/exact cache
    payload = {"q": user_query, "r": rule_codes}
    key = hash_key(payload)
    now = time.time()
    if key in _exact and now - _exact[key][0] < ttl:
        return {"text": _exact[key][1], "cache": "exact"}

    # 1) Compose short header + prune context
    header = rules_summary(rule_codes)  # ~150–220 tokens in practice
    chosen = top_sentences(user_query, sentences, k=10)   # 10 short facts
    context = "CONTEXT:\n" + "\n".join(f"- {s}" for s in chosen)

    # 2) Budget guard: if still too big, trim context
    approx_tokens = lambda s: int(len(s) / 4)  # crude char→token heuristic
    while approx_tokens(header + context) > budget_tokens:
        chosen = chosen[: max(6, len(chosen) - 2)]
        context = "CONTEXT:\n" + "\n".join(f"- {s}" for s in chosen)

    prompt = header + "\n\n" + context + "\n\nASK:\n" + user_query

    # 3) Fake LLM call (replace with real provider)
    answer = f"(demo) Using {len(chosen)} sentences and rules {','.join(rule_codes)} -> ..."

    # 4) Caches
    _exact[key] = (now, answer)
    _semantic.append((embed(user_query), key, now))
    return {"text": answer, "cache": "miss"}

Why this matters:

• The rules section is compact, stable, and readable.
• The context is the top facts, not the whole doc.
• A budget ensures you never exceed a token cap.
• Caches make repeats free.

Hook this into your real SDK and replace the embeddings with your store of choice (FAISS, SQLite-IVF, whatever fits).

Plan/execute split (small → smaller)

One giant prompt tries to do everything. I split it:

1. Planner (60–120 tokens): “What fields do we need? Which sources?” → returns a tiny plan.
2. Executor (300–500 tokens): Pulls only those fields/sentences and produces the answer.

That alone cut several flows by 2–3×, and combined with PCodec we consistently hit the 5× reduction.

Guardrails that keep quality intact

• Extractive bias: When pruning context, prefer direct quotes or short factual sentences. Models stay truer to source.
• Schema-first outputs: Ask for a JSON schema (referenced by code S1) so you don’t waste tokens with verbose formatting guidance.
• Hard caps: Refuse to expand if adding two more passages exceeds the budget; ask the user to refine instead of silently truncating.

But… does compression confuse the model?

It can — if you compress the meaning. The codec compresses form, not facts:

• Don’t abbreviate domain terms; abbreviate repeated instructions.
• Don’t pass opaque IDs to the model; fetch the sentences those IDs represent and pass those.
• Keep the ask plain. Your “human intent” field should read like a single, clear sentence.

In blind A/Bs on my content QA flow, humans rated answers from the compressed prompts as equal or better 78% of the time, mostly thanks to cleaner, more relevant context.

Results you can expect (ballpark)

• Docs Q&A: 1,800–2,600 → 350–600 prompt tokens (planner + top-10 sentences).
• Summarization: 2,000+ (full article) → 400–700 (extractive bullets + claim lines).
• Code-assist: 1,200 → 250–400 (rules as codes + snippet deltas instead of full files).

You won’t hit 5× everywhere. But you’ll hit it often enough to notice on your invoice.

Rollout checklist (one afternoon, honestly)

• Write your long policy once; distill to a 200-token rules summary with codes (R1…R10, S1…).
• Add a sentence-level retriever; cap context by token budget, not passage count.
• Implement a prompt budget guard that trims before the API call.
• Split at least one flow into plan → execute.
• Add exact + semantic caches with a tight TTL.
• Track prompt tokens per route; publish a leaderboard of top token hogs.

Closing thought

Good prompts feel like craft. Great prompts feel like protocols — small, structured, and ruthlessly relevant. The moment I started encoding rules, pruning context at sentence granularity, and summarizing what stays the same, my prompts stopped bloating. Costs calmed down. Latency did, too. And users noticed the only thing that matters: answers that arrive faster and still feel right.

Want my production-ready PCodec with a real embedding index and provider adapters? Drop a comment. If enough folks ask, I’ll share the repo and a few battle-tested budgets.