Building a Validated Knowledge Base From Trusted and Untrusted Sources — Part One

How we moved from “dump documents into a vector store” to an iterative, human-in-the-loop system for turning messy sources into validated, agent-ready knowledge.

The motivation

Does this sound familiar? You have many sources of information — some reliable, some not — and you’re struggling to combine them into a knowledge base you can actually trust. This article is for you.

I suspect most companies hit this problem and nobody talks about it after the RAG demo.

Most “knowledge base” projects follow the same arc: ingest PDFs, Word documents, Confluence content, Markdown, maybe some source code. Chunk, embed, retrieve. Ship a chat UI and call it done.

It works beautifully in the demo. Then reality arrives.

You ask about an API parameter — the documentation says one thing, the implementation does another, and the chatbot confidently picks whichever chunk ranked highest. You validate ten items in a spreadsheet, and the rest of the corpus remains a black box.

The core failure mode is subtle: ingestion is not knowledge construction. Indexing text is not the same as understanding a domain, surfacing contradictions, or building a catalogue a downstream agent can trust.

We ran into this while building a framework for domain-specific knowledge bases — systems meant to power MCP tools and autonomous agents, not just answer ad hoc questions. Our first version did what many RAG tutorials recommend: parse raw sources, ask an LLM for “20 topics to validate,” and store whatever a human approved. After ingesting a full codebase plus documentation, the knowledge base still had zero structured understanding of how the system worked. Validation only persisted those twenty hand-picked items. Everything else stayed frozen in chunks.

That is not a knowledge base. That is a search index with a validation sticker on a few pages.

The concept

What we needed was a loop:

Learn broadly → discover gaps and contradictions → rank what matters → validate with a human → lock trustworthy facts → repeat.

This article describes that concept, how we implemented it, what alternatives we considered, and the trade-offs we accept.

What we were actually trying to solve

Our requirements came from a concrete use case: building versioned, topic-scoped catalogues from heterogeneous sources — source code, config files, internal docs, PDFs — that agents could query with confidence about what is validated versus what is still draft.

Several constraints shaped the design:

Sources differ in reliability. Code and config are quasi-ground-truth; prose docs may be outdated.

Contradictions are the signal. A doc says MOVE_X = 2, code defines MOVE_X = 1 — that is the work, not a side effect to ignore.

Humans cannot review everything. A 4,000-line Python module is not a checklist of 20 bullet points.

Agents need structure, not just chunks. MCP tools want typed catalogue elements, not raw paragraph retrieval.

Knowledge evolves. Topics have versions; validated records must not be silently overwritten.

Sessions are long. Validation spans days; pause/resume and parallelization must be first-class.

The goal was not “chat with your docs.” The goal was a factory for validated knowledge — one Docker instance per domain, many instances in an ecosystem, each producing a catalogue agents can depend on.

How the system works

1. Slice, don’t boil the ocean

We introduce validation sessions to slice the set of information that needs to be validated. Rather than tackling everything at once, you concentrate first on a single interface, validate everything related to it, then move on to the next. Decomposing a system into modules and interfaces is straightforward and makes progress trackable.

Every session is anchored to a topic version. If the domain changes materially, you start v2 rather than overwriting v1. Old validated knowledge stays intact.

2. Define scope

Anchor the session to a topic, version, module, domain, and interface. This scoping means that if the topic changes later, the previously processed information remains usable as-is.

3. Learn before you ask

On session start, the system does not jump straight to questions. It first builds draft knowledge:

• An inventory pass over fully-reliable code and config using language-agnostic LLM extraction
• Deterministic AST parsing for Python symbols
• Vector search for topic-relevant chunks across the corpus
• Draft catalogue elements indexed immediately — searchable, but clearly marked unvalidated

Only after the knowledge base has something to say about the domain does discovery begin.

4. Discover open points automatically

A batched, multi-pass discovery engine compares drafts against reliable excerpts, scans file contexts, and runs gap and contradiction analysis until it hits configurable targets (e.g. 50–80 open points). Examples of what it finds:

• “Documentation states undo is unlimited; code enforces a maximum stack depth.”
• “Interface accepts values 1 and 2 in the spec; implementation only handles 3.”
• “Function calculate_hash is referenced in docs but not found in the codebase.”

When the initial cap is reached, you can gather more in additional batches — with deduplication against everything already in the session.

5. Rank by dependency, not by LLM whim

Not all open points are equal. Some block others. The system builds a dependency graph and ranks the validation queue by:

• Readiness — can this be answered now, or is it blocked by unresolved prerequisites?
• Unlock value — resolving this item unblocks how many others?

Humans start at the top of the queue, not at whatever the model found first.

6. Human-in-the-loop validation

A dedicated UI presents the ranked queue. For each open point, the human can accept an AI-drafted answer, correct it, or add knowledge directly (not tied to a specific open point). Validated entries become locked records. Draft entries remain searchable but flagged.

7. Re-rank and continue

Each resolution triggers re-evaluation: dependencies update, scores change, the queue reshuffles. The human always sees what matters now, not what mattered at session start.

8. Pause, resume, monitor

Long validation cycles need checkpoints. Sessions pause at defined phases and resume later without rewinding state. A live system map shows which components are active — inventory, draft synthesis, open-point discovery, dependency ranking — with progress bars and open-point tallies.

System architecture

System overview

The diagram below shows the end-to-end topology of a single KB framework instance — from raw files on disk to validated knowledge consumed by humans and agents.

What it depicts:

Raw sources are split into two reliability tiers. Code and config in fully_reliable/ are treated as ground truth; prose docs in not_reliable/ may be outdated. This split drives contradiction detection later.

Connectors ingest files and chunk them into searchable units stored in PostgreSQL. Chunking is language-aware for code and header-based for Markdown.

The orchestrator is the central coordinator. It drives the agent pipeline in sequence when a human starts a session: inventory → synthesis → open-point discovery → dependency ranking.

AI agents each have a distinct role. Inventory extracts structure from code; synthesis builds draft catalogue entries; discovery finds gaps and contradictions; dependency analysis orders the validation queue; retrieval powers chat and search.

The HITL UI is the human-facing surface for starting sessions, validating open points, chatting with the knowledge base, and monitoring live component status. Humans trigger orchestration; they do not call agents directly.

Index builder runs when a catalogue entry is locked (validated), upserting embeddings into the vector store. Draft entries can also be indexed earlier for exploratory search.

Staleness monitor watches source file hashes and flags changed documents so knowledge can be refreshed.

Persistence uses PostgreSQL for structured data (catalogue, sessions, open points, workflow checkpoints) and pgvector (or optionally Qdrant) for semantic search.

Downstream consumers include human validators today and MCP-compatible agents tomorrow. Both read from the same locked catalogue and vector index.

Key idea: Ingestion alone does not produce knowledge. The orchestrator turns raw material into draft catalogue entries, then into ranked open points, then into locked facts — with humans in the loop at the validation stage.

Diagram 1: High-level KB framework — ingest heterogeneous sources, build draft knowledge, discover open points, validate via HITL, serve locked catalogue to agents.

In Part Two, we look at the trade-offs honestly — what this architecture costs versus what it gives you — walk through the building blocks, a concrete example where the system caught intentional documentation inconsitencies, and explore what becomes possible once you have a catalogue of facts you can actually trust: chat agents that cite their confidence level, documentation generation, impact analysis, and more.

Continue to Part Two.