I Hand-Tuned My Retrieval Stack for Six Months. A Self-Editing Agent Makes That Look Like a Bug.

Chroma’s Context-1 is a small model that outperforms much larger ones at retrieval. The interesting part isn’t the parameter count — it’s what it says about where the intelligence in a RAG system actually belongs.

I build contract intelligence systems that run air-gapped. No API calls, no cloud inference, nothing leaves the building. That constraint shapes every architectural decision I make, and it’s why a headline about a 20B model beating frontier models at retrieval reads very differently to me than it probably does to you.

For most teams, a smaller model that performs well is a cost story. For me it’s an existence story. A frontier model behind an API isn’t expensive in my environment it’s unavailable. So the question I care about isn’t “can I save on tokens.” It’s “does the capability exist inside my perimeter or not.”

But before getting to that, I want to talk about what’s actually broken in the pipeline I spent months building.

Abstract

The pipeline I’m proud of is also the problem

The retrieval stack in my system looks like a lot of production RAG stacks in 2026:

• BGE dense embeddings for semantic recall
• BM25 for lexical precision, because contracts are full of clause numbers, party names, and Incoterms that embeddings smear into mush
• Reciprocal rank fusion to merge the two
• A cross-encoder reranked to clean up the top-k
• Purpose-built chunk types for the metadata that generic chunking destroys

Every one of those components works. Each one earned its place by fixing a specific failure I could reproduce. I can defend all of it in a design review.

And every one of those components is a frozen guess.

I picked the fusion weights based on a query distribution I imagined at build time. I set the reranker depth based on what looked good on a validation set. I chose chunk boundaries based on the contracts I happened to have in front of me last quarter. The pipeline executes exactly the same way whether the incoming question is “what is the delivery lead time” or “does the liquidated damages clause survive termination for convenience, and does that conflict with the force majeure carve-out.”

Those two queries need completely different retrieval behavior. The first needs one precise lookup. The second needs three or four, plus the judgment to notice when the answers contradict each other. My pipeline gives them identical treatment, because retrieval strategy is something I decided months ago and baked into a config file.

That’s the actual limitation. Not embedding quality. Not reranker accuracy. The fact that strategy is a design-time decision in a system that faces runtime variety.

What “self-editing” actually changes

[VERIFY: describe Context-1's mechanism accurately from the paper — what specifically gets edited, whether it's query reformulation, retrieval-plan revision, an explicit critique/revise loop, or something else. Don't let me guess at this.]

[VERIFY: the benchmark claims — which evals, which baseline models, what context lengths, what the actual deltas were. Put the specific numbers here. This is the paragraph the comments section will interrogate.]

The general shape of the idea, though, is what matters for anyone running RAG in production: it moves retrieval strategy from configuration into inference. The system decides how to search as part of searching, and revises when the first attempt doesn’t hold up.

If that works, a whole category of my hand-tuning becomes obsolete — not because it was wrong, but because it was a static approximation of a decision that should have been dynamic. My RRF weights are a compressed, lossy prior over “what kind of query is this.” An agent that can look at the query and the intermediate results doesn’t need my prior. It has the actual case in front of it.

What it wouldn’t touch

I want to be precise about the boundary, because “agentic retrieval solves RAG” is the kind of claim that costs people six months.

Retrieval is not my system’s main failure mode. My hardest problems are downstream:

• Absence assertions. The model confidently reporting that a clause doesn’t exist when it does. Better search reduces this, but it doesn’t fix the underlying willingness to make a negative claim from a null result.
• Evidence-value decoupling. The extracted value and the quoted supporting text drifting apart — the citation looks right, the number is wrong, and both pass a surface check.
• Deterministic table extraction. Commercial terms live in tables. That’s a parsing problem with a deterministic answer. I don’t want a language model being creative there, no matter how good it is at search.
• Verification gates. Cross-consistency checks between extracted fields. Purely mechanical, and load-bearing.

A better retriever makes the generator’s job easier. It doesn’t make the generator honest. Those are separate engineering problems with separate solutions, and conflating them is how you end up with a system that retrieves beautifully and still hallucinates a termination date.

The part that’s specific to running local

Here’s where the small-model result stops being an efficiency note and starts being architectural.

Agentic retrieval means multiple inference passes per query. Search, evaluate, revise, search again. In an API-based system that’s a cost multiplier you can model on a spreadsheet. On a single on-prem GPU it’s a latency multiplier against fixed hardware, and it competes directly with the generation pass for the same VRAM.

A 20B retrieval specialist and a larger generator can plausibly coexist on one box. A frontier-scale model doing multi-pass retrieval cannot — not on hardware a mid-size manufacturer will actually approve. So the size result isn’t a footnote to the capability result. For air-gapped deployments, the size result is the capability result. It’s the difference between “we could adopt this” and “we’ll read about it.”

The open questions I’d want answered before putting it near production:

• How many passes does it take on average, and what’s the p95? Agent loops have long tails, and a contract review that usually takes 4 seconds and sometimes takes 40 is a UX problem.
• Does it degrade gracefully on domain-specific corpora it wasn’t trained for? Industrial supply agreements are not Wikipedia.
• Can I inspect the revision trace? In a compliance context, “the system decided to search differently” needs to be auditable, not magic.

The takeaway

The instinct in RAG has been to put the intelligence in the generator and treat retrieval as plumbing — a solved problem you tune once and stop thinking about. Everything about the last two years of tooling reflects that assumption.

Results like this one suggest the plumbing was where the headroom was all along. Which is uncomfortable, because it means a lot of us optimized the wrong half of the system, and got real gains anyway, and mistook that for validation.

I’m going to test it against my own corpus rather than trust the benchmark. But I’ve already stopped defending my fusion weights in design reviews.