Most RAG failures don’t crash. They silently return bad answers. I built a repair layer for that.

There’s a specific kind of frustration that comes from watching a RAG pipeline return confident nonsense and having no idea where it went wrong.

Was it retrieval? Did BM25 pull in irrelevant chunks? Did the model hallucinate on top of good context? Did the grounding check even run? You stare at the output, shrug, and tweak a prompt somewhere hoping it helps.

I kept running into this. Not because the tools are bad (LangChain, LlamaIndex, RAGAS), but none of them close the loop. They’ll score a bad run. They won’t fix it.

So I built ragbolt.

What the problem actually is

Most RAG failures fall into three buckets:

Retrieval comes back weak. BM25 scores are low, the chunks don’t match the query, and whatever gets generated is working from bad evidence. The system doesn’t know this happened.

Generation goes sideways. The model returns something malformed, empty, or just confidently off-base. No signal, no trace, no retry.

Grounding fails silently. The response uses language that isn’t supported by any retrieved chunk. It sounds fine. It’s not.

These aren’t edge cases. They happen on normal queries with normal corpora. And when they happen, most pipelines just return the output anyway.

What ragbolt does

ragbolt wraps your existing pipeline. It doesn’t replace it.

It detects which failure class occurred, applies one bounded repair, and re-verifies. The loop looks like this:

1. Retrieve with BM25. If the top score is below threshold → expand top_k, retry once.
2. Generate. If the response is empty or errored → fail fast, record it.
3. Verify grounding with EGA (Evidence-Gated Generation). If unsupported ratio is too high → reduce context to the top chunk, retry generation once.
4. Return the outcome: ACCEPTED, REPAIRED_ACCEPTED, ABSTAINED, or FAILED.

Every run emits a trace:

{
  "run_id": "9187274c-cee0-453e-b531-20c0070e6f5e",
  "corpus_id": "smoke_corpus",
  "query": "how does retrieval augmented generation work",
  "outcome": "ACCEPTED",
  "failure_classes": [],
  "repair_attempts": 0,
  "top_score": 1.159,
  "chunks_retrieved": 1,
  "chunk_ids": ["c2"],
  "unsupported_ratio": 0.0,
  "raw_top_score": 1.159,
  "timestamp_utc": "2026-05-09T18:34:38.730084+00:00"
}

That trace is append-only. Every run accumulates in the same file. You can audit exactly what happened, when, and why.

Design decisions I made deliberately

Bounded repair, not infinite retry. The policy is two repairs max, one per failure class, in a fixed order. This isn’t laziness — it’s a deliberate constraint. Unbounded repair loops are an agentic system. ragbolt is not an agent. It’s a wrapper with a hard ceiling.

No framework. ragbolt doesn’t ask you to restructure your pipeline around it. It’s a layer you drop in. If you’re already using LangChain, it drops in as a retriever or QA component. LlamaIndex — same thing.

Deterministic retrieval by default. BM25 is the default retriever because it’s reproducible. You can run the same query twice and get the same result. FAISS hybrid is available for production use, but the stub works for evals.

The verifier is a protocol, not a class. The EGAVerifier is a typing.Protocol with runtime_checkable. You can swap in the lexical stub for testing and a cross-encoder NLI model for production without changing anything else.

Using it

pip install ragbolt

Ingest your documents:

ragbolt ingest docs/ --output corpus.json

Run a query:

ragbolt run corpus.json "what is the refund policy" \
  --provider anthropic \
  --verifier production

Explain what happened:

ragbolt explain rag_trace.json

Run ID  : 9187274c-cee0-453e-b531-20c0070e6f5e
Corpus  : smoke_corpus
Query   : what is the refund policy

Retrieval: 3 chunk(s) retrieved (top BM25 score: 1.1596)
  Chunks : c1, c2, c3

Failures : none
Repairs  : 0 attempt(s)

Outcome  : ✓ Response accepted — fully grounded, no repairs needed.

Run a batch of queries from a file:

ragbolt batch corpus.json queries.txt --provider anthropic

Calibrate your thresholds based on actual trace history:

ragbolt calibrate rag_trace.json --apply

What it integrates with

If you’re already using LangChain:

from ragbolt.adapters import RagboltRetriever

retriever = RagboltRetriever("corpus.json", provider_name="anthropic")
docs = retriever.get_relevant_documents("your query")

LlamaIndex:

from ragbolt.adapters import RagboltQueryEngine

engine = RagboltQueryEngine("corpus.json", provider_name="anthropic")
response = engine.query("your query")
print(response.metadata["outcome"])

It also exposes a REST API if you want to run it as a service:

ragbolt serve corpus.json --port 8000
# POST /query
# GET  /health
# GET  /trace

What it’s not

Worth being specific here because the space is full of tools that overpromise:

• Not a RAG framework. You don’t build pipelines inside ragbolt.
• Not an eval dashboard. It emits traces and reports — you visualize them yourself.
• Not an agent system. The repair loop has a hard ceiling of two attempts.
• Not a grounding verifier. EGA is a component, not the product.

The positioning that felt right: ragbolt is to a RAG pipeline what a circuit breaker is to a distributed system. It doesn’t prevent failure. It detects it, bounds it, and gives you a signal.

Where it’s at

Five versions shipped in the last few weeks:

• 0.1.0 — BM25, corpus schema, bounded repair, CLI
• 0.2.0 — Anthropic/OpenAI providers, FAISS hybrid, NLI verifier
• 0.3.0 — eval pipeline, explain command, streaming, GitHub Actions, docs
• 0.4.0 — ingest, batch, serve (FastAPI), calibrate, OpenTelemetry export
• 0.5.0 — LangChain and LlamaIndex adapters

pip install ragbolt           # core
pip install ragbolt[full]     # + FAISS + NLI verifier
pip install ragbolt[serve]    # + FastAPI REST API
pip install ragbolt[all]      # everything

Repo: https://github.com/bh3r1th/ragbolt

Disclaimer: The posts here represent my personal views, not those of my employer or any specific vendor. Any technical advice or instructions are based on my knowledge and experience.