How to give your local LLM harness GraphRAG abilities

Your local model doesn’t have web search. It can’t pip install the docs of a library it’s never seen. It forgets everything between sessions. It “knows” what its training corpus knew, and only that.

A curated retrieval layer fixes all three problems. The setup is small (a few hundred lines of code plus an inverted-index file and a directory of markdown), the runtime cost is one BM25 query per user prompt (~50ms on SQLite-backed indexes), and the impact is concrete: I’ve used this pattern to take a stubborn unit-test-counting failure from “never passes” to “passes 50–60% of the time” on a single hand-curated knowledge chunk. The model never learned the pattern. The retrieval layer just hands it the right index card at the right moment.

This post walks through the architecture, the implementation pieces, the gotchas you’ll hit, and the calibration that makes the difference between noise and signal.

— -

## The problem in one example

A common test prompt for a local code agent says something like:

”Add tests/test_core.py with at least 6 cases covering 1, 4, 9, 40, 90, 1994. Make the CLI command print MCMXCIV.”

The harness check is pytest -q reports ≥6 cases passing.

A 26B-class local model often writes three test functions:

def test_to_roman():

assert to_roman(1) == “I”

assert to_roman(4) == “IV”

# …

def test_to_int(): …

def test_roundtrip(): …

Three functions, all green. The check fails because pytest counts FUNCTIONS as cases (each @pytest.mark.parametrize row also counts as a case, but three asserts in one function is one case). The model conflates “asserts” with “cases” because nothing in its training data was crisp on that distinction.

You can’t fix this with a vague system prompt to “use pytest correctly.” You can’t fix it with web search — the model is local. You can fix it by giving the model a 200-word chunk that explains the counting rule with a concrete example, and surfacing that chunk automatically when the user prompt mentions test-count requirements.

That’s what a retrieval-backed knowledge layer does for you.

— -

## The architecture

The trick is in the last step. The model sees the retrieved chunks AS IF it had called a retrieve tool itself. From its perspective, the relevant cookbook content is just present in context — no awareness that the harness put it there. The model treats tool results as authoritative facts it asked for; it treats system messages as constraints it can selectively ignore. Synthetic-retrieve gets reasoned over; system messages get glossed.

— -

## Implementation walk-through

### Step 1: write knowledge chunks as markdown

Each chunk is a markdown file. Keep it short — local models have small working memory and can’t read 5000-token chunks effectively. Aim for 200–800 words. Single-purpose. One concrete code example. The “why” briefly so it generalizes.

Example: python/parametrize_test_counts.md:

**# When a PRD says “at least N test cases”**

PRDs frequently say things like:

> “Add `tests/test_core.py` with ****at least 6 cases**** covering 1, 4, 9, 40, 90, 1994.”

The check counts ****pytest test cases****, not test functions. A file with

three functions like `test_to_roman()`, `test_to_int()`, `test_roundtrip()`

is ****3 cases****, not 6 — even if each function asserts on multiple inputs.

**## The right pattern: `@pytest.mark.parametrize`**

When the PRD lists concrete inputs (1, 4, 9, 40, 90, 1994), parametrize

over exactly those inputs. Each row becomes one pytest case.

```python

import pytest

from romanize.core import to_roman

CASES = [(1, “I”), (4, “IV”), (9, “IX”), (40, “XL”), (90, “XC”), (1994, “MCMXCIV”)]

@pytest.mark.parametrize(“n,expected”, CASES)

def test_to_roman(n, expected):

assert to_roman(n) == expected

pytest -q reports 6 passed.

Skip long preambles. The model isn’t going to read 2000 words.

### Step 2: ingest

Walk the cookbook directory, split each file into chunks at heading boundaries, tokenize (lowercase + drop common stopwords), compute IDF over the corpus, write everything to SQLite.

```bash

# pseudo-code: your own ingest command

$ rag-tool ingest <cookbook_root>

Ingested 37 files, 403 text chunks. Index at <cache_dir>/index.sqlite

Re-run after every cookbook edit. The ingest should be idempotent — hash chunk content, only re-index changed chunks.

### Step 3: the auto-prefetch hook

This is the core of the architecture. After the user types a prompt, BEFORE the model sees it for the first time, your agent loop does:

def auto_prefetch_retrieve(user_msg: str) -> None:

# 1. Extract a query from the user message

query = user_msg[:400] # cap to avoid BM25 token blow-up

if len(query) < 10:

return # skip trivial prompts

# 2. BM25-rank chunks

hits = index.search(query, top_k=8)

# 3. Filter by score threshold

QUALITY_THRESHOLD = 8.0

good_hits = [h for h in hits if h.score >= QUALITY_THRESHOLD]

if not good_hits:

log_zero_hit_gap(query) # see “self-feeding” section below

return

# 4. Format as a tool-call + tool-result PAIR

synthetic_call = make_tool_call(

name=”retrieve”,

arguments={“query”: query[:200]},

)

body = “\n\n — -\n\n”.join(

f” — — {h.source}:{h.line_start}-{h.line_end} (score={h.score:.1f}) — -\n”

f”{h.content}”

for h in good_hits[:5]

)

# 5. Inject as if the model had called retrieve itself

inject_synthetic_tool_result(

synthetic_call,

f”=== TEXT ===\n{body}”,

)

Where you hook this depends on your stack. In an OpenAI-tool-calling-style agent loop, the typical place is right after the user message is appended to messages and before the first LLM call. The synthetic tool-call + tool-result pair lives in the message history exactly like the model had emitted the call itself.

### Step 4: optional follow-up nudge

After the retrieve injection, you can inject a short system note that says “the retrieved chunks are authoritative; read them carefully.” Use this only for high-confidence hits — otherwise it dilutes:

AUTHORITATIVE_SCORE = 100.0

if top_score >= AUTHORITATIVE_SCORE and chunk_has_canonical_answer:

inject_system_note(

“The retrieve tool result above contains a curated chunk “

“whose question matches the user’s. Read that chunk and “

“emit the value verbatim. Do not re-derive.”

)

Without this, on borderline cases the model will sometimes call web_search or re-derive from scratch instead of trusting the retrieve.

— -

## Make it self-feeding

The cookbook is only as good as what’s in it. Hand-curating is fine for the first 30 chunks, but the long tail of “things that should be in the cookbook but aren’t” is huge. The right answer is a gap detector that finds the missing knowledge for you.

### Gap detection

Log every retrieve call:

[RETRIEVE] query: ‘Add a — verbose flag to the CLI’

[RETRIEVE] 8 total hits, 5 above threshold 8.0 → injected

[RETRIEVE] query: ‘How do I configure nginx log format’

[RETRIEVE] 0 above-threshold hits → gap!

When a query returns zero above-threshold hits, enqueue it for review:

def log_zero_hit_gap(query: str) -> None:

enqueue_gap({

“query”: query[:200],

“ts”: now(),

“session”: current_session_id(),

})

Dedup by fingerprint (so recurring “how do I X” doesn’t flood the queue), and aggregate across sessions.

### Auto-ingest from gaps

A background process (cron or daemon) consumes the queue:

1. Read the gap queue.

2. Cluster gap queries by topic.

3. For each cluster, run a code-grep verification — does any existing repo code use the term? If yes, generate a draft chunk from the code + comments using whatever automation you trust.

4. Stage drafts in <cookbook_root>/_staging/<topic>.md with a TODO marker.

5. Wait for a human to promote (move out of _staging/).

The human review is the quality gate. Auto-generated chunks would degrade quickly if you trusted them blindly. In my experience, a ~10–15% promotion rate from staged drafts is the right ballpark — high enough that the gap detector is doing useful work, low enough that quality stays high.

— -

## Gotchas

I want to be honest about what didn’t work first time.

### BM25 buries specific phrases

Classic embarrassment: a prompt says ”Add tests/test_core.py with at least 6 cases covering 1, 4, 9, 40, 90, 1994" and the perfect cookbook chunk on parametrize-counting scores ~47 against a query containing those exact phrases. But against the FULL user prompt (which mentions “Roman numerals”, “argparse”, “subcommands”, “MCMXCIV”, “python -m”), the chunk scores 47 while project-relevant chunks score 120+. BM25 prioritizes prompt-dominant topics; specific instructions like “at least 6 cases” get buried.

The fix isn’t more BM25 tuning. The fix is a separate regex-pattern injection layer that fires AFTER the auto-retrieve, on prompts matching specific patterns you’ve seen fail. Hand-crafted, narrow, deterministic. Five patterns covers a lot of ground (test-count requirements, rename tasks, abstraction extraction, schema migration, preserve-existing-behavior). Each is ~5 lines of regex + a 200-word system note.

If the cookbook is your library, pattern injection is your index card with key concepts pre-flagged.

### Score threshold tuning

The right threshold depends on your corpus:

• A small hand-curated cookbook (≤200 chunks): low threshold (≈4–8) is safe. Every chunk is precise.

• A medium general-purpose corpus (~1000 chunks): higher threshold (~10–20) to filter noise.

• A huge fallback corpus (~100k+ chunks — arXiv abstracts, all of Wikipedia, full Stack Overflow dumps): much higher threshold (~30+) or you’ll inject loosely-relevant junk on every prompt.

A good architecture uses a fallback chain: query the project-specific cookbook first, then a home cookbook, then optionally a giant fallback. Only inject from the first layer that returns above-threshold hits. The per-project cookbook is small and very precise; the home cookbook is medium and general; the giant fallback is huge and noisy. Each layer has its own threshold.

### Chunk length

Local models choke on long context. A 5000-word “everything about pytest” chunk is worse than five 800-word chunks each focused on one pattern. Split on heading boundaries; keep each chunk single-purpose.

I had to rewrite about half my initial chunks for length. The old ones tested at score 100+ but the model would only read the first ~30% before context-budgeting. Splitting them improved both retrieval (more focused chunks → better score signal) and model uptake.

### “The model didn’t use the retrieved chunk”

This happens. Watch for it. Two common causes:

1. The chunk was retrieved but the model preferred prior knowledge. Mitigation: the authoritative-score system note (Step 4 above). For genuinely authoritative chunks (your own curated answers), tell the model in plain text to use them.

2. The chunk was retrieved but the formatting confused the model. Format chunks consistently. I use — — path:line_start-line_end (score=N.N) — - as a header, body below, three-dashes between chunks. Predictable formatting helps the model parse what it’s reading.

If you debug a “model ignored the chunk” case, log the actual retrieved content the model saw, not just the score. Add something like [RETRIEVE] synthesized N chunks, top score X.X, content N chars to your session log specifically for this. You’ll save yourself hours.

### Cookbook quality > quantity

A 500-chunk cookbook of mediocre content performs worse than a 30-chunk cookbook of high-quality chunks. Quality measures:

• Concrete. Real code examples, not abstract principles.

• Single-purpose. One pattern per chunk.

• Narrow scope. “This is what to do when X happens.” Not “everything about Y.”

• The why, briefly. So the model can generalize, but a few sentences is enough.

The cookbook is your engineering asset. Treat it like documentation — review it, version it, ruthlessly delete chunks that don’t earn their context budget.

— -

## What it’s NOT good for

Be honest with yourself about scope:

• Multi-step planning across files. Retrieval gives the model knowledge; it doesn’t give it the working memory to apply that knowledge across 5 files simultaneously. Hard model-capability ceiling — no cookbook gets you past it.

• Algorithmic correctness on novel inputs. A chunk explaining “how to count records correctly” doesn’t help if the model misreads the input. You need verification tools (post-edit pytest, test-count validators), not more knowledge.

• Long-horizon refactors. Multi-week structural changes need humans, not better retrieval.

• Tasks where the cookbook chunk would be longer than the task itself. If you’d write 1500 words to explain “how to do X” but X is a one-paragraph task, the chunk is dead weight.

The retrieval layer’s job is to teach the model patterns it didn’t see during training but a human can teach in 200 words. That’s a narrow band, but it’s a profitable band.

— -

## Calibration: how to know it’s working

Track three metrics:

1. Retrieval hit rate. How many user prompts trigger a synth-retrieve injection? Aim for 30–70% on typical workloads. ❤0% means your threshold is too high or your cookbook coverage is too thin. >80% suggests the threshold is too low and you’re injecting noise.

2. Chunk-usage signal. When the model saw a chunk, did it use the pattern? Measure on a small held-out set: prompts where you KNOW which chunk should fire. Run with and without retrieval. Diff outcomes.

3. Gap rate. Zero-hit queries per session. Aim for <5%. Higher means the cookbook is missing common terrain.

For (2) especially, set up a side-by-side A/B. Same model, same task, retrieval-on vs retrieval-off, same minute. Otherwise you’re chasing variance.

— -

## The honest framing

A retrieval-backed knowledge layer with a curated cookbook gives your local model knowledge of patterns it never saw during training. It doesn’t make the model smarter — it gives it a librarian who hands the right index card at the right moment.

For specific failure modes (the test-count case, structured refactor patterns, “I always forget to back up before migrating”), the lift is real and measurable. For generic capability gaps (multi-file planning, novel algorithmic correctness), the lift is small to none. The cookbook is most useful for specific failure modes you’ve seen and curated for. It’s least useful for generic capability gaps.

The marginal cost of adding a cookbook chunk is low. The marginal cost of debugging the same “model doesn’t know X” failure across many sessions is high. If you’re running a local agent in production, even 5% of users hitting “model doesn’t know X” is enough to motivate a 200-word chunk that closes the gap.

Start with one chunk. Pick a failure you’ve seen three times. Write the chunk. Ingest. Watch the next time it would have failed.

If you see the model use the chunk and succeed — that’s enough. You’ve started a library.

— -

The pattern in this post is generic — BM25-on-SQLite isn’t novel, neither is auto-injection-as-synthetic-tool-result. What’s novel is treating the cookbook as a first-class engineering asset, not a “FAQ docs” afterthought. If you implement this and want to compare notes, I’d be interested to hear which calibration choices you ended up making, and which chunks earned their context budget.