The Verification Layer Every AI Agent Needs (and How I Built One Twice)

Introduction: The Day I Stopped Trusting My Own Agent

A few months ago I was reviewing an agent’s output for a research summarization task. The answer was clean, confident, and cited a source. It read like something a careful analyst would write. The problem was that the cited source did not say what the agent claimed it said. Not a small misreading either. The agent had inverted the conclusion.

What unsettled me was not that it happened once. It was how easy it would have been to miss it. The output had every surface signal of trustworthiness: structure, tone, a citation. Nothing about how it looked told me it was wrong. I only caught it because I happened to know the source material well enough to notice the gap.

That is the moment I stopped asking “does the agent respond well” and started asking a different question: how do I actually verify what an AI system tells me, not just whether it responds fluently. Those are two completely different engineering problems, and most of the systems I had built up to that point only solved the first one.

This article is about the second problem. I built a verification layer to sit between an agent’s raw output and whatever consumes that output, whether that is a user, a downstream system, or another agent. I built it twice, once in Python and once in .NET, because I wanted to know if the problem looked different depending on which ecosystem you approach it from. It mostly does not. But the differences that do show up are worth talking about.

What You’ll Find Here

This is a standalone deep dive, not part of my Agentic Architectures series, though it follows the same format because consistency helps if you have read my other work. Here is what this article covers:

• A four layer verification architecture: hallucination detection, RAG grounding checks, agent decision verification, and source trust scoring
• Working Python implementations for each layer, using open tools like Ragas and DeepEval where they fit
• Parallel .NET implementations using Semantic Kernel, so you can see the same logic expressed in both ecosystems
• Local, self-hosted examples using Ollama alongside cloud examples, so you are not locked into a paid API to try any of this
• An honest account of what broke in production, including false positive rates and the latency cost of adding four verification layers to a pipeline that used to have zero

The Anatomy of a Verification System

Before writing any code, I had to decide what “verification” actually meant, because the word gets used loosely. I settled on four distinct concerns, each answering a different question:

1. Did the model say something that is not true, regardless of source? (hallucination detection)
2. Does the retrieved context actually support the claim being made? (RAG grounding)
3. Should this decision go through automatically, or does it need a human? (agent decision verification)
4. How much should I trust the source this information came from in the first place? (source trust scoring)

These are not the same problem wearing different hats. A claim can be perfectly grounded in a retrieved document and still be wrong, because the document itself is wrong. A claim can be true and still risky enough that it needs a human to sign off. Treating these as one blob called “verification” is how you end up with a system that catches obvious hallucinations but sails right past a subtly bad source.

Here is the flow I ended up with:

                 +----------------------+
   User Query -> |   Agent / LLM Call   |
                 +----------------------+
                            |
                            v
                 +--------------------------+
                 |  Layer 1: Hallucination  |
                 |  Detection               |
                 +--------------------------+
                            |
                            v
                 +---------------------------+
                 |  Layer 2: RAG Grounding   |
                 |  Verification             |
                 +---------------------------+
                            |
                            v
                 +--------------------------+
                 |  Layer 3: Agent Decision |
                 |  Verification Layer      |
                 +--------------------------+
                    |               |
              auto-approve     escalate to human
                    |               |
                    v               v
                 +--------------------------+
                 |  Layer 4: Source Trust   |
                 |  Scoring                 |
                 +--------------------------+
                            |
                            v
                 +--------------------------+
                 |   Verified Output        |
                 +--------------------------+

Each layer can reject, flag, or pass the output forward. None of them are gates in the sense of a single pass or fail. They attach a score and a reason, and the layer after them decides what to do with that information. That design choice mattered more than I expected once I got to production, and I will come back to it.

Layer 1: LLM Output Verification (Hallucination Detection)

The textbook definition of hallucination, a model generating factually incorrect content, is not wrong, but it is not specific enough to build against. In practice, the hallucinations that hurt me were not wild fabrications. They were small inversions, confident overstatements, and details that sounded plausible but were not present anywhere in the source material.

Python Implementation

I used Ragas for this, because its faithfulness metric is built exactly for this comparison: given a claim and a set of source contexts, how much of the claim is actually supported.

from ragas.metrics import faithfulness
from ragas import evaluate
from datasets import Dataset
def check_hallucination(question: str, answer: str, contexts: list[str]) -> dict:
    data = Dataset.from_dict({
        "question": [question],
        "answer": [answer],
        "contexts": [contexts],
    })
    result = evaluate(data, metrics=[faithfulness])
    score = result["faithfulness"][0]
    return {
        "score": score,
        "passed": score >= 0.75,
        "reason": "low faithfulness score" if score < 0.75 else "supported"
    }
# Example usage
result = check_hallucination(
    question="What was the reported revenue growth last quarter?",
    answer="Revenue grew by 12% compared to the previous quarter.",
    contexts=["Quarterly revenue increased by 4%, driven primarily by the enterprise segment."]
)
print(result)
# {'score': 0.2, 'passed': False, 'reason': 'low faithfulness score'}

If you prefer DeepEval, the same check looks like this:

from deepeval.metrics import HallucinationMetric
from deepeval.test_case import LLMTestCase
def check_hallucination_deepeval(question: str, answer: str, contexts: list[str]) -> dict:
    metric = HallucinationMetric(threshold=0.5)
    test_case = LLMTestCase(
        input=question,
        actual_output=answer,
        context=contexts
    )
    metric.measure(test_case)
    return {
        "score": metric.score,
        "passed": metric.score <= 0.5,
        "reason": metric.reason
    }

Local Alternative with Ollama

Both Ragas and DeepEval can run their judge model locally through Ollama instead of calling out to OpenAI or Bedrock. Here is the same faithfulness check running against a local llama3.1:8b model, which is what I use when I am iterating on the verification logic itself and do not want to pay per call:

from langchain_community.llms import Ollama
from ragas.llms import LangchainLLMWrapper
local_llm = LangchainLLMWrapper(Ollama(model="llama3.1:8b"))
result = evaluate(
    data,
    metrics=[faithfulness],
    llm=local_llm
)

.NET Implementation with Semantic Kernel

Semantic Kernel does not ship a faithfulness metric out of the box, so I built the check as a verification prompt pattern, essentially using a second model call to grade the first one against its sources. This mirrors what Ragas is doing under the hood.

using Microsoft.SemanticKernel;
public class HallucinationChecker
{
    private readonly Kernel _kernel;
    public HallucinationChecker(Kernel kernel)
    {
        _kernel = kernel;
    }
    public async Task<VerificationResult> CheckAsync(
        string question, string answer, List<string> contexts)
    {
        var contextBlock = string.Join("\n---\n", contexts);
        var prompt = $"""
            You are a strict fact checker. Given a question, an answer, and 
            source context, score from 0.0 to 1.0 how well the answer is 
            supported by the context alone. Respond with only the number.
            Question: {question}
            Answer: {answer}
            Context: {contextBlock}
            """;
        var result = await _kernel.InvokePromptAsync(prompt);
        var score = double.Parse(result.ToString().Trim());
        return new VerificationResult
        {
            Score = score,
            Passed = score >= 0.75,
            Reason = score < 0.75 ? "low faithfulness score" : "supported"
        };
    }
}
public record VerificationResult
{
    public double Score { get; init; }
    public bool Passed { get; init; }
    public string Reason { get; init; } = "";
}

Wiring this to Ollama instead of Bedrock in Semantic Kernel is a matter of swapping the connector, since Semantic Kernel treats models as interchangeable services:

var builder = Kernel.CreateBuilder();
builder.AddOllamaChatCompletion(
    modelId: "llama3.1:8b",
    endpoint: new Uri("http://localhost:11434")
);
var kernel = builder.Build();

That is the entire migration from cloud to local for this layer. Same prompt, same checker class, different connector line.

Layer 2: RAG Grounding Verification

Hallucination detection asks “is this true.” Grounding verification asks a narrower and, honestly, easier question: “is this actually in the documents I retrieved.” A claim can fail grounding even if it happens to be true, because the system had no business asserting it based on what it pulled.

I check this with embedding similarity between the claim and the retrieved chunks, rather than another LLM call, because it is faster and cheaper to run on every single claim in an answer.

Grounded vs Ungrounded Claim Examples
--------------------------------------------------------
Claim                          | Retrieved Chunk Support | Verdict
--------------------------------------------------------
"Revenue grew 4% QoQ"          | Direct match             | Grounded
"Enterprise segment drove it"  | Direct match             | Grounded
"Growth outpaced competitors"  | No mention in context    | Ungrounded
"Q3 outlook is positive"       | Paraphrased, present     | Grounded

Python Implementation

import ollama
import numpy as np
def get_embedding(text: str) -> np.ndarray:
    response = ollama.embeddings(model="nomic-embed-text", prompt=text)
    return np.array(response["embedding"])
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def check_grounding(claim: str, chunks: list[str], threshold: float = 0.6) -> dict:
    claim_embedding = get_embedding(claim)
    scores = [cosine_similarity(claim_embedding, get_embedding(c)) for c in chunks]
    max_score = max(scores)
    return {
        "score": max_score,
        "passed": max_score >= threshold,
        "best_matching_chunk": chunks[scores.index(max_score)]
    }
result = check_grounding(
    claim="Growth outpaced all competitors in the segment",
    chunks=["Quarterly revenue increased by 4%, driven by the enterprise segment."]
)
print(result)

.NET Implementation

using Microsoft.SemanticKernel.Embeddings;
public class GroundingChecker
{
    private readonly ITextEmbeddingGenerationService _embeddingService;
    public GroundingChecker(ITextEmbeddingGenerationService embeddingService)
    {
        _embeddingService = embeddingService;
    }
    public async Task<VerificationResult> CheckAsync(
        string claim, List<string> chunks, double threshold = 0.6)
    {
        var claimEmbedding = await _embeddingService.GenerateEmbeddingAsync(claim);
        double bestScore = 0;
        string bestChunk = "";
        foreach (var chunk in chunks)
        {
            var chunkEmbedding = await _embeddingService.GenerateEmbeddingAsync(chunk);
            var score = CosineSimilarity(claimEmbedding.Span, chunkEmbedding.Span);
            if (score > bestScore)
            {
                bestScore = score;
                bestChunk = chunk;
            }
        }
        return new VerificationResult
        {
            Score = bestScore,
            Passed = bestScore >= threshold,
            Reason = bestScore < threshold ? $"best match: {bestChunk}" : "grounded"
        };
    }
    private static double CosineSimilarity(ReadOnlySpan<float> a, ReadOnlySpan<float> b)
    {
        double dot = 0, magA = 0, magB = 0;
        for (int i = 0; i < a.Length; i++)
        {
            dot += a[i] * b[i];
            magA += a[i] * a[i];
            magB += b[i] * b[i];
        }
        return dot / (Math.Sqrt(magA) * Math.Sqrt(magB));
    }
}

Registering the local Ollama embedding service in Semantic Kernel:

builder.AddOllamaTextEmbeddingGeneration(
    modelId: "nomic-embed-text",
    endpoint: new Uri("http://localhost:11434")
);

Same model, nomic-embed-text, running through Ollama, called from both ecosystems. That consistency was intentional. I did not want to compare a cloud-only Python setup against a local-only .NET setup, because that would not tell me anything about the languages themselves.

Layer 3: Agent Decision Verification Layer

This is the layer that decides whether a decision the agent is about to act on needs a human, or whether it is safe to let it proceed automatically. This is separate from whether the content is true. A perfectly grounded, perfectly faithful claim can still be too consequential to auto-approve.

I use a confidence threshold combined with an impact classification. Low impact and high confidence gets auto-approved. Anything else escalates.

from enum import Enum
class Impact(Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
def decision_gate(confidence: float, impact: Impact) -> dict:
    if impact == Impact.HIGH:
        return {"action": "escalate", "reason": "high impact always requires review"}
    if impact == Impact.MEDIUM and confidence < 0.9:
        return {"action": "escalate", "reason": "medium impact below confidence bar"}
    if impact == Impact.LOW and confidence < 0.7:
        return {"action": "escalate", "reason": "even low impact needs a confidence floor"}
    return {"action": "auto_approve", "reason": "within thresholds"}
print(decision_gate(confidence=0.95, impact=Impact.MEDIUM))
# {'action': 'auto_approve', 'reason': 'within thresholds'}
print(decision_gate(confidence=0.6, impact=Impact.LOW))
# {'action': 'escalate', 'reason': 'even low impact needs a confidence floor'}

The .NET version of the same gate, kept deliberately identical in logic so the comparison is fair:

public enum Impact { Low, Medium, High }
public record DecisionResult(string Action, string Reason);
public static class DecisionGate
{
    public static DecisionResult Evaluate(double confidence, Impact impact)
    {
        if (impact == Impact.High)
            return new DecisionResult("escalate", "high impact always requires review");
        if (impact == Impact.Medium && confidence < 0.9)
            return new DecisionResult("escalate", "medium impact below confidence bar");
        if (impact == Impact.Low && confidence < 0.7)
            return new DecisionResult("escalate", "even low impact needs a confidence floor");
        return new DecisionResult("auto_approve", "within thresholds");
    }
}

This is the one layer where I noticed the languages diverge a little. The C# enum plus record pattern made the decision states feel exhaustive and self-documenting in a way that was harder to enforce in Python without adding a validation layer on top of the plain enum. I will talk more about that in the comparison section.

Layer 4: Source Trust Scoring

Grounding tells you the claim matches the source. It says nothing about whether the source itself deserves to be trusted. I score sources on three simple factors: recency, source type, and cross-reference count, meaning how many independent sources say the same thing.

Source Trust Scoring Model
--------------------------------------------------------
Factor                | Weight | Example Score
--------------------------------------------------------
Recency                | 0.3    | 0.9 (updated last week)
Source type             | 0.4    | 0.7 (internal doc, not peer reviewed)
Cross-reference count   | 0.3    | 0.5 (only one source agrees)
--------------------------------------------------------
Weighted Trust Score     |        | 0.71

Python Implementation

def source_trust_score(recency: float, source_type: float, cross_reference: float) -> float:
    weights = {"recency": 0.3, "source_type": 0.4, "cross_reference": 0.3}
    score = (
        recency * weights["recency"]
        + source_type * weights["source_type"]
        + cross_reference * weights["cross_reference"]
    )
    return round(score, 2)
print(source_trust_score(recency=0.9, source_type=0.7, cross_reference=0.5))
# 0.71

.NET Implementation

public static class SourceTrustScorer
{
    public static double Score(double recency, double sourceType, double crossReference)
    {
        const double recencyWeight = 0.3;
        const double sourceTypeWeight = 0.4;
        const double crossReferenceWeight = 0.3;
        var score = recency * recencyWeight
            + sourceType * sourceTypeWeight
            + crossReference * crossReferenceWeight;
        return Math.Round(score, 2);
    }
}

Nothing exotic here on either side. I kept this layer simple on purpose. Every time I tried to make the trust model smarter, with weighted decay curves or learned weights, I ended up with a model that was harder to explain to a reviewer than the thing it was scoring. A trust score you cannot explain in one sentence is not a trust score, it is a black box wearing a lab coat.

.NET vs Python: What I Actually Learned Choosing Between Them

I expected this section to be about performance or tooling maturity. It ended up being about something else entirely: how each language shapes the way you think about verification state.

Python let me move fast. I had Layer 1 working with Ragas in under twenty minutes because the ecosystem already has purpose built libraries for exactly this problem. Semantic Kernel does not have that yet, so I had to build the hallucination check myself using a raw verification prompt. That took longer, but it also meant I understood every line of what was happening, instead of trusting a library’s internal scoring logic.

Where .NET pulled ahead was in Layer 3. The moment I introduced an enum and a record for decision states, the compiler stopped me from accidentally handling an impact level inconsistently. In the Python version, nothing stops you from adding a fourth Impact value and forgetting to handle it in decision_gate, at least not until a test catches it, if a test exists. That is a small thing until you are three months into production and someone adds a new impact tier without reading the whole function.

Semantic Kernel also felt like it was translating LangGraph concepts rather than natively expressing them. LangGraph’s graph-based state machine maps cleanly onto how I think about agent flows. Semantic Kernel’s plugin and planner model does the same job, but I found myself mentally converting from one mental model to the other more than I expected. That is not a criticism of Semantic Kernel, it is a genuinely different design philosophy, but it means the .NET version took me noticeably longer to architect even though the individual verification checks were often just as short as their Python counterparts.

My honest takeaway: if your team is already Python heavy and doing anything RAG adjacent, the ecosystem tooling gives you a real head start on Layers 1 and 2 specifically. If your team is .NET heavy for other reasons, enterprise integration, existing services, type safety requirements, you lose some of that head start on the AI specific tooling but you gain guardrails on the decision logic that Python makes you build by hand.

Production Reality Check

Here is what actually happened once this ran against real traffic instead of my test cases.

False positive rate on hallucination checks was higher than I expected. The faithfulness metric flagged roughly 18% of answers as unsupported in the first two weeks, and when I manually reviewed a sample, about a third of those flags were wrong. The claims were true, just phrased in a way that did not lexically overlap with the source enough for the check to feel confident. I had to add a second pass using paraphrase-tolerant embedding comparison before trusting the flag.

Latency cost was real and I underestimated it. Adding four verification layers to a pipeline that previously had none added an average of 2.1 seconds per response, mostly from the two extra LLM calls in hallucination detection and the embedding calls in grounding verification. For a chat interface, that is noticeable. I ended up running Layers 1 and 2 in parallel instead of sequentially, which brought it down to about 1.3 seconds, but it is still not free.

The escalation threshold in Layer 3 needed constant tuning. My first version escalated far too much, to the point where the human reviewers started rubber stamping escalations without reading them carefully, which defeats the entire purpose of having a human in the loop. I had to raise the auto-approve threshold for medium impact decisions twice before the escalation volume matched what a human could actually review thoughtfully.

Source trust scoring was the layer nobody argued with. Ironically the simplest layer, the one I almost cut for being “too basic,” turned out to be the one stakeholders trusted most, precisely because they could look at the three factors and understand the number. The fancier layers got more scrutiny and more pushback specifically because they were harder to explain.

If I rebuilt this today, I would keep all four layers, but I would build the escalation tuning dashboard first instead of last. I treated it as an afterthought and it turned out to be the thing that determined whether the whole system was actually usable day to day.

Closing

Verification is not a feature you bolt onto an agent once it is already in production and something has gone wrong. It is infrastructure, the same way logging and monitoring are infrastructure. You do not add observability after the outage. You do not add verification after the bad answer already shipped to a customer.

Building this twice, once in each ecosystem, did not change my mind about which layers matter. It changed my mind about how much the tooling around you shapes what you build first and what you build carefully. Pick the ecosystem your team already trusts, and expect to build at least one of these layers by hand regardless of which one you pick.

Other Articles

• Article 1: The Agentic AI Maturity Model
• Article 2: Advanced Coordination and Reasoning Patterns
• Article 3: AgentOps
• Article 4: Agentic Protocols — MCP and A2A
• Article 5: Harness Engineering and the Agent Runtime Layer
• Article 6: Multi-Agent Orchestration Patterns
• Article 7: Agent Memory Architectures
• Article 8: Evaluation and Continuous Improvement Pipelines
• Article 9: Agentic Architectures — Article 9: Agent Security and Red-Teaming
• Workflow Design Is a Thinking Discipline

Tags

ai-verification, hallucination-detection, dotnet, python, rag