I Built a Triplestore Knowledge Graph for Our System — V2

It Changed How I Think About Documentation

Over the last few weeks, I went down a rabbit hole that started as a small experiment and turned into something far more consequential: a triplestore knowledge graph designed to model architectural claims, not just facts.

What began as a simple node-and-edge visualisation quickly evolved into a proper knowledge graph system, where every relationship carries provenance, context, timestamps, evidence and an indication of how it was established.

Knowledge graphs: Concepts and Relationships

The immediate goal was practical: improving the quality of architecture and disaster recovery documentation for a critical system. But the deeper lesson ended up being about how engineering organisations manage knowledge as systems evolve.

The Real Problem With Documentation

Every long-lived engineering system accumulates documentation. Wikis, diagrams, runbooks, architecture pages and decision records are often written at different times, by different people, for different purposes.

Each document reflects a view of the system at the moment it was written. Meanwhile, the system itself continues evolving through feature work, refactors, infrastructure changes, configuration updates, deployments and migrations.

Eventually, the documentation drifts.

Most teams experience this, but in systems where reliability and operational clarity matter — treasury systems, payment systems and critical infrastructure — documentation drift becomes a real operational risk. Incorrect assumptions about dependencies, ownership, recovery procedures or failover chains can surface during incidents, when the cost of confusion is highest.

The problem is not simply that engineers do not write documentation. The deeper problem is that traditional documentation systems present statements as static, while system knowledge is continuously changing.

Not All Evidence Proves the Same Thing

One of the first insights from looking at this problem through a systems lens is that architectural knowledge comes from multiple sources. But those sources do not form a simple hierarchy from “most trustworthy” to “least trustworthy”. They answer different questions.

Source code can show that a dependency has been implemented. Build artefacts can show that the relevant code was packaged. Deployment configuration can show that it was deployed into a particular environment, while runtime traces can show that the relationship was actually observed.

Tests can show that a relationship behaved as expected under defined conditions. Architecture documentation can explain why it exists. Engineers can describe operational realities that may not yet be encoded anywhere else.

None of these sources is complete on its own.

A dependency declared in code may be unused, disabled, test-only or hidden behind a feature flag. A runtime trace may prove that an interaction occurred, but not that every valid failover path has been exercised. A wiki page may contain outdated implementation details while still preserving the only accurate explanation of an architectural decision.

The useful distinction is therefore not simply between trusted and untrusted sources. It is between different kinds of evidence: declared, implemented, built, deployed, observed, tested, approved and inferred.

Recognising these differences naturally leads to a richer evidence model. Architectural claims should not merely be assigned a universal confidence score. They should record what kind of evidence supports them, where that evidence came from, when it applied and in which environment it was valid.

Representing Architecture as a Knowledge Graph

To explore this idea, I built a pipeline that extracts architectural relationships from several sources and records them in a triplestore.

Wiki documentation is processed through an LLM to identify entities, claims and architectural relationships. Source code is parsed directly to identify structural relationships such as declared dependencies, service boundaries and client integrations. Deployment and configuration information can describe what is present in a particular environment, while runtime telemetry can contribute relationships that have actually been observed.

Engineer domain knowledge can also be recorded explicitly as structured claims.

At the simplest level, relationships still take the familiar form:

subject → predicate → object

But each statement is accompanied by metadata describing its provenance, evidence type, environment, source references, timestamps, extraction confidence, review status, validity period and explanatory notes.

The important shift is that the graph does not just store statements. It stores statements with evidence and context.

Two identical architectural claims can coexist in the graph while representing different things. One may have been extracted from a design document, another may have been observed in production, and a third may apply only to the disaster recovery environment.

Rather than collapsing them into a single fact, the graph preserves their origin, scope and status.

Modelling Time and Reverification

Another critical concept is that architectural knowledge is temporal. Infrastructure changes, services migrate, configurations are replaced and dependencies evolve.

Documentation systems usually handle this poorly. A statement written two years ago may appear just as authoritative as one verified yesterday. But age alone does not make a claim false.

A stable architectural decision may remain valid for years, while a feature-flagged dependency may change several times in a day. The more useful question is not simply how old a statement is, but when it was valid, when it was observed and when it was last verified.

In the knowledge graph model, relationships can therefore carry several temporal dimensions: when the relationship was valid in the system, when the graph learned about it, when it was last observed, when it was last verified and when it was superseded.

Different kinds of claims can also have different reverification policies. A runtime dependency may need to be checked after each deployment. A disaster recovery procedure may need to be revalidated after an infrastructure change or recovery exercise. An architecture decision may remain valid until explicitly superseded.

This is more useful than allowing confidence to decay mechanically over time. Old claims do not automatically become false. They become increasingly expensive to trust without renewed evidence.

Tracking Changes to Evidence

The graph also tracks exact source references. Where possible, a claim is linked to a specific file, symbol, commit, document section, deployment manifest, trace, test result or architecture decision.

When an underlying source changes, the system can identify which claims may require reassessment. That does not mean every source change invalidates every derived relationship.

A code file may change without affecting a dependency. A document may be reformatted without changing its meaning. A deployment may alter runtime behaviour without modifying the source file from which the original claim was extracted.

The correct conclusion is therefore not that the claim has become false. It is that the evidence changed and the claim may need to be checked again.

This creates a living representation of architectural knowledge rather than a static archive.

Handling Disagreement Explicitly

Architectural knowledge is rarely perfectly consistent. Different documents may describe the same system differently. Code may differ from deployment configuration, while runtime behaviour may differ from both. Operational knowledge may reveal exceptions that were never included in the formal design.

Instead of hiding these differences, the graph records them explicitly. But not every disagreement is a contradiction.

Two claims may apply to different environments or have been valid at different times. One may describe the intended architecture while another describes observed behaviour. One may refine, supersede or provide an exception to another.

The graph therefore needs to distinguish between contradiction, supersession, environmental differences, temporal differences, refinement, exceptions, disputed claims and claims that have not yet been observed.

This creates an audit trail showing how knowledge about the system evolved. For complex systems, preserving this history is valuable because it captures the reasoning, evidence and context behind architectural changes rather than simply replacing older statements.

Why This Matters for AI-Assisted Documentation

This model also changes how LLMs can be used safely for documentation generation. When language models are fed raw documentation, they have limited ability to distinguish reliable information from outdated, incomplete or contradictory assertions. The result is often plausible-sounding but unreliable output.

A knowledge graph gives the model structure and context. Instead of asking an LLM to invent architecture documentation, the model can render claims already recorded in the graph and expose the evidence supporting them.

The system should not turn claims into facts simply because they cross an arbitrary confidence threshold. It should preserve their epistemic status. A relationship may be observed, deployed, declared, tested, inferred, disputed, stale, unverified or superseded.

The generated documentation can communicate those distinctions directly.

For example, stating that the Payment API was observed calling the Treasury Database in production after a particular deployment is different from stating that a disaster recovery design document says the Payment API can fail over to a secondary database, but that the path has not been verified by a recent recovery exercise.

Both statements are useful. They simply represent different kinds of knowledge.

For areas such as disaster recovery documentation, where understanding dependency chains is critical, the graph becomes an authoritative index of architectural claims and their evidence. It does not replace source code, deployment systems, telemetry, tests or human judgement. It connects them.

The LLM then translates that structured evidence into readable documentation, queries, explanations and review queues.

A Shift in How We Think About Documentation

The larger takeaway from this experiment is that the documentation challenge in software engineering is fundamentally a knowledge representation and reconciliation problem.

Traditional documentation tools organise pages. A knowledge graph organises claims, relationships, evidence and context.

The graph can tolerate incomplete knowledge without assuming that missing information is false. At the same time, operational systems cannot remain completely open-ended.

For critical areas, we still need explicit validation rules. Every production service may need an owner. Every critical dependency may require a recovery classification. Every exposed API may need a documented authentication mechanism, and every disaster recovery component may require a tested recovery procedure.

The useful model therefore combines two approaches. It supports open-world knowledge, where missing information remains unknown, while also supporting closed-world validation, where required information must be present for a particular operational purpose.

By representing architectural knowledge as contextual claims with provenance, evidence, time and ownership, the system can evolve alongside the software it describes. In practice, this means documentation becomes a continuously maintained knowledge system rather than a static set of pages.

The Graph Is Not the System

One important lesson is that the graph should not be described as the single source of truth. It is derived from other systems.

Code, configuration, deployments, telemetry, tests, architecture decisions and engineer knowledge all remain important sources of evidence. The graph is better understood as the system of record for architectural claims: a place where those claims can be compared, queried, reviewed and reconciled.

That distinction matters.

For disaster recovery, the strongest evidence is not a relationship stored in a graph. It is a successful, recent and auditable recovery exercise. The graph should record that evidence, not replace it.

Governance Matters

Once the graph becomes operationally useful, several additional concerns become unavoidable.

Relationships need owners. Claims need reviewers. Critical assertions need defined verification methods and review dates.

The vocabulary itself also needs governance so that relationships such as depends_on, calls, uses and requires do not gradually become inconsistent duplicates.

The graph may also contain sensitive architectural information. Internal endpoints, failover locations, privileged dependencies, database topology and known gaps should not automatically be visible to every user or every model.

Access control, source classification, audit trails, redaction and retrieval permissions must therefore be part of the architecture. A knowledge graph can improve AI grounding, but it does not remove the need for security, evaluation or human accountability.

Measuring Whether It Works

A confidence model is only useful if it is evaluated. The system therefore needs more than a visual graph and an impressive demonstration.

It should measure entity and relationship extraction accuracy, unsupported claim rates, entity-resolution errors, evidence coverage, stale-claim detection, contradiction classification and the time required to verify disputed claims.

It should also measure the accuracy of generated architecture documentation and the proportion of critical dependencies supported by recent observations or tests.

Without evaluation, confidence is simply a number generated by the system. With evaluation, it can become a calibrated signal that engineers understand and trust.

Where This Goes Next

The current prototype includes a lightweight visualiser that allows engineers to explore the graph, inspect relationships, view their supporting evidence and filter by environment, evidence type and review status.

The next step is integrating the graph into the engineering workflow so that code changes, deployments, configuration updates, runtime observations, documentation changes and architecture reviews continuously update the knowledge model.

Longer term, this opens the door to generating operational documentation — including disaster recovery runbooks — directly from the graph. It also enables architectural impact analysis, dependency verification, ownership checks, stale-knowledge review queues, incident investigation, architecture drift detection and evidence-backed AI retrieval.

The deeper lesson is simple: systems evolve, and our documentation systems should evolve with them.

Treating architectural knowledge as a structured and continuously reconciled system of claims, evidence and observations turns documentation from a maintenance burden into a living model of the system.

The graph does not replace reality. It makes our understanding of reality inspectable.

And in environments where reliability matters, that distinction is more than academic.