Your AI Doesn’t Know If It’s Lying — Ours Does.

How We Built a Self-Healing, Observable AI Brain — Entirely on Private Infrastructure.

Part 2 of our Sovereign AI Infrastructure series *Read Part 1: “One query. Three models. Zero cloud. How a 200ms classifier saves 77% of our AI compute cost.”*

For the business head: We now know in real time whether our AI is answering correctly, making things up, or ignoring the documents it was given — automatically, without a human reviewing every response.

For the architect: We added LLM-as-a-Judge evaluation, full distributed tracing, RAG faithfulness scoring, and Langfuse observability — all self-hosted, zero cloud dependency.

For the AI professional: Every inference request now produces a Langfuse trace with classifier span, RAG retrieval span, and four automated quality scores — running as background tasks with zero latency impact.

The Problem Nobody Admits

Most enterprise AI deployments have a dirty secret.

Nobody actually knows if the AI is performing well.

Teams track uptime. They track cost. Some track latency. But the question that actually matters — “Is the AI giving correct, grounded, trustworthy answers?” — goes largely unmeasured.

There are three failure modes that quietly destroy user trust:

Failure 1 — HALLUCINATION
The model confidently states something false.
"Our refund policy allows returns within 60 days."
Reality: your policy says 14 days.
Nobody caught it because nobody was measuring it.

Failure 2 — RAG IGNORANCE
You invested in a private knowledge base.
The model retrieved the right documents.
Then ignored them entirely and answered from training data.
Cost: you paid for RAG and got nothing from it.

Failure 3 — IRRELEVANCE
The model answered a different question than the one asked.
The user rephrases and tries again.
Three attempts later they give up and call a human.
Cost: you built AI to reduce support load and increased it.

If you cannot measure these — you cannot fix them.

In Part 1 we built the routing intelligence. In this post we close the loop — measurement, observability, and automated quality evaluation.

What We Added in Phase 3

Two new layers on top of the existing gateway:

┌─────────────────────────────────────────────────────────────┐
│                    PHASE 3 ADDITIONS                        │
├──────────────────────────┬──────────────────────────────────┤
│   RAG PIPELINE           │   OBSERVABILITY LAYER            │
│                          │                                  │
│   Private documents      │   Langfuse (self-hosted)         │
│        ↓                 │        ↓                         │
│   bge-large embedding    │   Full distributed tracing       │
│   (remote GPU VM)        │   per request                    │
│        ↓                 │        ↓                         │
│   Qdrant vector DB       │   LLM-as-a-Judge evaluation      │
│   (local, sovereign)     │   4 automated quality scores     │
│        ↓                 │        ↓                         │
│   Context injection      │   Real-time dashboards           │
│   into prompt            │   cost · latency · quality       │
└──────────────────────────┴──────────────────────────────────┘

Layer 1 — RAG: Making Your AI Know What Your Organisation Knows

Large language models know a lot. They do not know your HR policy, your Q3 sales report, your engineering runbook, or your product roadmap.

RAG (Retrieval Augmented Generation) solves this by giving the model access to your private documents at query time — without fine-tuning, without retraining, without sending your data anywhere.

How it works

Your Documents (PDF, DOCX, TXT, MD)
           ↓
  Document Ingestion Pipeline
  ├── Split into 512-char overlapping chunks
  │   (overlap prevents context loss at boundaries)
  ├── Generate embeddings via bge-large
  │   (335M param model — runs on your GPU VM)
  └── Store vectors in Qdrant
      (local vector DB — your disk, your control)

User Query: "What is our remote work policy?"
           ↓
  Query embedded via same bge-large model
  (CRITICAL: same model as ingestion — vectors
   must live in the same semantic space)
           ↓
  Cosine similarity search in Qdrant
  Top 4 most relevant chunks retrieved
           ↓
  Context injected into system prompt
           ↓
  Model answers FROM YOUR DOCUMENTS
  with source citation

The sovereignty guarantee

What goes where:

Your documents → YOUR Qdrant instance (local disk)
Embedding model → YOUR GPU VM (bge-large via Ollama)
Vector search → YOUR Qdrant (local, millisecond latency)
Model call → YOUR LLM (local or private GPU VM)

External services touched: ZERO
Documents leaving your network: ZERO
API calls to OpenAI/Anthropic/Google: ZERO

This is not marketing language. Every component in this stack is open source, self-hosted, and fully air-gap capable. Your patient records, legal contracts, and financial models never leave your walls.

The Routing + RAG Decision Tree

The classifier we built in Part 1 now makes two decisions simultaneously in a single 200ms call:

User query arrives
       ↓
llama3.2 classifier reads the query
       ↓
Returns structured JSON:
{
  "complexity": 2,
  "needs_rag": true,
  "reason": "Asks about internal company policy",
  "confidence": 0.94
}
       ↓
       ├── needs_rag = true?
       │         ↓
       │   Search Qdrant → retrieve top 4 chunks
       │   Inject context into prompt
       │
       └── needs_rag = false?
                 ↓
           Skip Qdrant entirely
                 ↓
Route by complexity:
  1 → llama3.2        (local, fast, cheap)
  2 → deepseek-r1     (GPU VM, reasoning)
  3 → gpt-oss:20b     (GPU VM, powerful)

The result is four distinct request patterns, each optimal for its use case:

Simple + General | No RAG | llama3.2 | "What is Python?"
Simple + Private | RAG ✅ | llama3.2 | "What is our Wi-Fi password?"
Complex + General | No RAG | gpt-oss:20b | "Design a DR architecture"
Complex + Private | RAG ✅ | gpt-oss:20b | "Analyse our Q3 report and suggest strategy"

Layer 2 — Langfuse: Full Observability on Private Infrastructure

Langfuse is an open-source LLM observability platform. We self-host it on Docker, connected to our existing PostgreSQL instance. No cloud account. No data leaving the network.

What gets logged per request

TRACE: inference_request
│
├── INPUT:  "What is our maternity leave policy?"
│
├── SPAN: llm_classifier              (200ms)
│     ├── model:      llama3.2
│     ├── complexity: 1/3
│     ├── needs_rag:  true
│     ├── confidence: 94%
│     └── reason:     "Asks about internal HR policy"
│
├── SPAN: rag_retrieval               (380ms)
│     ├── found:      true
│     ├── sources:    [hr_policy.txt]
│     ├── chunks:     2
│     └── scores:     [0.82, 0.74]
│
├── OUTPUT: "According to our HR policy, maternity
│           leave is 26 weeks full pay..."
│
└── METADATA:
      model_used:        llama3.2
      tier:              1/3
      latency_ms:        1840
      prompt_tokens:     312
      completion_tokens: 89
      total_tokens:      401
      rag_used:          true
      rag_sources:       [hr_policy.txt]

What the dashboard shows in real time

┌─────────────────────────────────────────────────────────┐
│  LANGFUSE DASHBOARD — AI Inference Gateway              │
├──────────────┬──────────────┬──────────────┬────────────┤
│  Total cost  │  Requests    │  Avg latency │  Scores    │
│  $0.024      │  847 today   │  1.8s        │  0.87 avg  │
├──────────────┴──────────────┴──────────────┴────────────┤
│  MODEL LATENCY (p50 percentile)                         │
│  ollama/llama3.2        ████░░░░░░  0.8s               │
│  ollama/deepseek-r1     ████████░░  4.4s               │
│  ollama/gpt-oss:20b     ██████████  26.9s              │
├─────────────────────────────────────────────────────────┤
│  SPAN LATENCIES                                         │
│  lm_classifier    p50: 1.6s    p99: 4.4s               │
│  rag_retrieval    p50: 1.7s    p99: 1.7s               │
└─────────────────────────────────────────────────────────┘

This answers the questions every CTO and business head should be asking:

• Which model is the bottleneck?
• Is the classifier adding too much overhead?
• Is RAG retrieval fast enough for production SLAs?
• What is our total AI infrastructure cost per day?

Layer 3 — LLM-as-a-Judge: Automated Quality Measurement

This is the part most teams skip. And it is the most important.

We use a small, fast LLM (llama3.2) as a quality judge. After every response is sent to the user, the judge runs in the background and scores the response on four dimensions.

The four scores

SCORE 1 — RELEVANCE
"Did the response actually answer the question?"

Score 1.0 → Response directly and completely answers
Score 0.7 → Mostly answers, misses some aspects
Score 0.4 → Related but does not really answer
Score 0.0 → Off-topic or refuses entirely

SCORE 2 — HALLUCINATION RISK
"Did the model state anything false or invented?"

Score 1.0 → No hallucinations, all claims credible
Score 0.7 → Minor uncertainty, mostly accurate
Score 0.4 → Some suspicious claims detected
Score 0.0 → Clear false information stated as fact

SCORE 3 — COMPLETENESS
"Did the response cover all aspects of the question?"

Score 1.0 → Thorough, covers everything
Score 0.7 → Covers main points, misses detail
Score 0.4 → Superficial or partial
Score 0.0 → Barely addresses the question

SCORE 4 — FAITHFULNESS (RAG queries only)
"Did the model answer FROM the retrieved documents?"

Score 1.0 → Every claim grounded in context
Score 0.7 → Mostly from context, minor additions
Score 0.4 → Mix of context and hallucinated content
Score 0.0 → Ignored the documents entirely

The architecture of automated evaluation

User sends query
       ↓
Smart Router → Model responds → Response sent to user
                                        ↓
                               asyncio.create_task()
                               (background — user sees no delay)
                                        ↓
                               Judge model (llama3.2) reads:
                               ├── Original question
                               ├── Model response
                               └── RAG context (if used)
                                        ↓
                               Returns 4 JSON scores
                               with reasons per score
                                        ↓
                               Scores posted to Langfuse
                               via langfuse.score() API
                                        ↓
                               Dashboard updates in ~10 seconds
                               Trends visible over time

Why background evaluation matters

Evaluation adds 3–6 seconds of compute time (three separate judge calls). The user must not wait for this. By running it as an asyncio.create_task() in Python, the evaluation fires after the response is already in the user's browser — completely invisible.

What This Tells You About Your System

The scores accumulate over time in Langfuse and reveal patterns that would otherwise be invisible:

Scenario 1 — Hallucination score drops below 0.7 over 48 hours

Signal: Model started making things up more frequently
Cause:  A new document was ingested that confused the model
        OR the wrong tier is being selected for complex queries
Action: Check classifier accuracy in Langfuse traces
        Consider raising the complexity threshold for Tier 2→3

Scenario 2 — Faithfulness score consistently below 0.6

Signal: Model is ignoring retrieved RAG context
Cause:  Chunk size too large — context is too noisy
        OR retrieved chunks are not specific enough
Action: Reduce chunk size from 512 to 256 characters
        Raise MIN_SCORE threshold from 0.50 to 0.65

Scenario 3 — Relevance drops for one specific model tier

Signal: Tier 2 (deepseek-r1) giving irrelevant answers
Cause:  Classifier routing wrong queries to this tier
        OR the model needs a better system prompt
Action: Review classifier span outputs in Langfuse traces
        Adjust complexity boundary between Tier 1 and Tier 2

Scenario 4 — All scores healthy, latency spikes at Tier 3

Signal: Good quality but gpt-oss:20b is slow at peak hours
Cause:  GPU VM under load from concurrent requests
Action: Add a second Tier 3 instance
        LiteLLM load balances automatically

The Complete Production Stack

Everything self-hosted. Everything open source. Everything sovereign.

┌─────────────────────────────────────────────────────────────┐
│                  COMPLETE ARCHITECTURE                      │
├─────────────────────────────────────────────────────────────┤
│  USER LAYER                                                 │
│  Open WebUI (port 3000) — chat interface                    │
│  RAG Manager  (port 8501) — document upload UI              │
├─────────────────────────────────────────────────────────────┤
│  INTELLIGENCE LAYER                                         │
│  Smart Router (port 5000)                                   │
│  ├── LLM Classifier    → complexity + RAG decision          │
│  ├── RAG Retriever     → Qdrant search + context injection  │
│  ├── Guardrail Handler → friendly blocked messages          │
│  └── Evaluation Engine → 4 quality scores (background)     │
├─────────────────────────────────────────────────────────────┤
│  GATEWAY LAYER                                              │
│  LiteLLM (port 4000)                                        │
│  ├── Model routing     → 3 tiers, fallback, retry           │
│  ├── Cost tracking     → per model, per user, per team      │
│  ├── Virtual keys      → spend limits, rate limits          │
│  └── Guardrails        → content policy enforcement         │
├─────────────────────────────────────────────────────────────┤
│  DATA LAYER                                                 │
│  Qdrant        → private vector DB (your disk)              │
│  PostgreSQL    → logs, keys, spend data, Langfuse traces    │
├─────────────────────────────────────────────────────────────┤
│  OBSERVABILITY LAYER                                        │
│  Langfuse (port 3001) — self-hosted                         │
│  ├── Full request traces with nested spans                  │
│  ├── Model latency by percentile (p50/p90/p95/p99)          │
│  ├── Token usage and cost per model per day                 │
│  └── Quality score trends over time                         │
├─────────────────────────────────────────────────────────────┤
│  MODEL LAYER                                                │
│  Local Mac                                                  │
│  └── llama3.2 (Tier 1 — simple queries)                    │
│                                                             │
│  Remote GPU VM (private)                                    │
│  ├── deepseek-r1  (Tier 2 — reasoning)                     │
│  ├── gpt-oss:20b  (Tier 3 — complex)                       │
│  └── bge-large    (embeddings for RAG)                      │
└─────────────────────────────────────────────────────────────┘

External cloud services: ZERO
Data leaving your network: ZERO
Monthly OpenAI/Anthropic bill: ZERO

The Business Case, Made Plain

For the Business Head

You now have answers to questions your board will ask:

“Is our AI trustworthy?” Langfuse shows hallucination scores trending above 0.85 across all queries this week. Yes.

“Are we getting value from the document knowledge base?” RAG faithfulness score: 0.88. The model is using retrieved context 88% of the time. Yes.

“What is our AI costing us?” $0.024 per day across 847 requests. Under $1 per day for the entire organisation’s AI usage.

“Could a data breach expose our AI queries to vendors?” No vendor has access. No query leaves our infrastructure. Zero.

For the Technical Architect

The system is built on five open-source components that your team fully controls:

ComponentPurposeReplacesLiteLLMAPI gatewayAzure OpenAI / OpenAI APIQdrantVector searchPinecone / Weaviate CloudLangfuseObservabilityHelicone / Datadog LLMvLLMModel servingOpenAI APIFastAPI routerRouting logicCustom middleware

Each component is independently upgradable, replaceable, and auditable. No vendor lock-in at any layer.

For the HR and Compliance Team

Your employee data, policy documents, HR records, and internal procedures stay in your database, on your servers, in your jurisdiction.

The AI reads your HR handbook to answer employee questions. It does not send that handbook to any external service to do so. GDPR Article 32 compliance is structural, not contractual.

What Scores Look Like After a Week of Production Traffic

WEEKLY QUALITY SUMMARY
─────────────────────────────────────────────────────
Metric              Value    Trend      Status
─────────────────────────────────────────────────────
Relevance           0.91     ↑ +0.03    ✅ Healthy
Hallucination       0.88     → stable   ✅ Healthy
Completeness        0.79     ↑ +0.05    ✅ Improving
Faithfulness (RAG)  0.84     ↑ +0.08    ✅ Improving
─────────────────────────────────────────────────────
Total requests      5,847    ↑
RAG queries         2,103    36% of total
Cost (7 days)       $0.17    ↓ -22% (routing optimised)
─────────────────────────────────────────────────────
Models by volume:
  llama3.2        3,241  (55%) — simple queries
  deepseek-r1     1,874  (32%) — reasoning queries
  gpt-oss:20b       732  (13%) — complex queries
─────────────────────────────────────────────────────

55% of queries handled by the cheapest local model. 13% escalated to the powerful model. Routing is working exactly as designed.

The Key Insight

Most organisations approach AI observability as an afterthought — something to add once things start going wrong.

We built it in from day one because you cannot govern what you cannot measure.

When a response is wrong, you know within 10 seconds. You know which model produced it, what context was retrieved, what score the judge gave it, and why. You have a trace ID, a timestamp, and a span breakdown.

That is not just observability. That is accountability.

And accountability — over AI systems that touch your employees, your customers, and your data — is not optional. It is the price of deploying AI responsibly.

What Comes Next

Phase 4 — Safe Model Management (coming in Part 3)
  ├── Canary routing: 10% traffic to new model
  ├── Quality comparison: old model vs new via Langfuse scores
  ├── Automated promotion if quality improves
  └── One-command rollback if quality drops

Phase 5 — RAG Accuracy Improvement
  ├── Hybrid search: dense + sparse vectors
  ├── Cross-encoder re-ranking
  └── Chunk strategy per document type

Phase 6 — Fine-tuning Pipeline
  ├── Export high-scoring traces as training data
  ├── Fine-tune small model on your domain
  └── Replace classifier with domain-specific model

The Stack, One More Time

Open source: LiteLLM · Qdrant · Langfuse · FastAPI · Ollama · vLLM
Models:      llama3.2 · DeepSeek-R1 · bge-large
Hosting:     Your servers. Your GPU. Your database. Your rules.
Cloud cost:  $0/month
Vendor lock: None
Data risk:   None

Built with FastAPI · LiteLLM · Qdrant · Langfuse · vLLM · Open WebUI All components self-hosted. Zero cloud AI API dependencies. Zero tokens sent to external providers.