Stop Shipping Untested Prompts: Content-Addressable Eval Attestation for Agentic Systems

While shipping some refactored AI workflows in rp1 recently, we added git worktree support to an agent. Manual testing looked good. The PR was clean. CI was green.

Three days later, we noticed the agent was silently failing to create isolated branches. Sometimes it worked. Sometimes it didn’t. No errors, just quiet misbehavior. Users were confused why their changes weren’t appearing on the expected branches.

The root cause was boring: we’d tweaked the agent’s instructions, tested it by hand a few times, and shipped. The prompt change was never run through automated evaluations. We got lucky during manual testing; production traffic found the edge cases we missed.

The fix: make prompt changes provably test-gated, by content hash.

We call this approach content-addressable attestation. We built it as part of Ready Player One — rp1, an agentic development platform. This post explains why we built it, how it works, and how you can build your own.

Why Existing Controls Fail

You might think code review catches prompt bugs. It doesn’t, for the same reason code review doesn’t catch logic bugs: humans aren’t compilers.

You might think “just run evals on prompts” is enough. It’s not, because prompts have dependencies. Your custom slash command often delegates to an agent. That agent invokes a skill. Someone changes the skill; your command’s tests never run because the command file didn’t change. Dependency drift is invisible.

The fundamental problem: in LLM systems, behavior is a function of text. That text deserves the same supply-chain guarantees we demand for binaries and dependencies: reproducibility, provenance, tamper-evidence, auditability.

The Requirement

Here’s the invariant we enforce:

A prompt cannot be merged unless the exact prompt content, plus its entire dependency chain, has a recorded passing evaluation.

Not “version 2.0.0 passed tests.” Not “someone said they tested it.” The requirement is: this SHA-256 hash, representing this exact text, passed this evaluation suite, at this commit.

Everything else is implementation detail.

How It Works

Four components make this work:

1. Hash computation strips metadata and hashes only the prompt content. Version strings, descriptions, and other frontmatter are ignored; only the text that affects LLM behavior matters.

2. Dependency extraction parses prompts for delegation patterns and builds a graph. A command that delegates to two agents and a skill produces a four-file dependency chain. All files are hashed together.

3. Evaluation execution runs Promptfoo suites with LLM-based assertions. Only 100% pass rates update the attestation; partial passes don’t count.

4. CI verification recomputes hashes on every PR. If the current hash doesn’t match the recorded attestation, the merge is blocked.

The source of truth is a JSON manifest committed to git:

{
  "commands": {
    "build-feature": {
      "prompt_hash": "sha256:6940348d...",
      "deps_hash": "sha256:c1def3cf...",
      "last_eval": { "passed": true, "timestamp": "2026-01-22T07:35:37Z" }
    }
  }
}

An attestation is a tamper-evident record that says: this content passed these tests at this commit. Because it’s stored in git alongside the prompts, you get a complete audit trail with no external service dependency.

Workflow: Before and After

Before attestation:

1. Edit prompt
2. Eyeball in code review
3. Merge
4. Find out it’s broken when users complain

After attestation:

1. Edit prompt
2. Run bun run eval my-command
3. On pass, run bun run attest my-command
4. Commit prompt + attestation together
5. Push; CI verifies hashes automatically
6. Merge allowed only if attestation is current

If you skip step 3, CI blocks the merge. No exceptions.

Handling Nondeterminism

LLM outputs are probabilistic. A test that passes once might fail on the next run. Readers will wonder: how do you trust these evaluations?

Our approach: run each test multiple times. If all runs pass, the attestation is valid. This catches flaky prompts that work “most of the time” but fail on edge cases. The worktree incident would have been caught; multiple runs would have exposed the inconsistent behavior that manual testing missed.

You can tune the tradeoff:

• More runs = higher confidence, slower CI
• Fewer runs = faster CI, risk of false confidence

We bias toward catching regressions over speed. A flaky prompt that passes CI is worse than a slow CI that catches problems.

Threat Model

What this prevents:

• Untested prompt changes reaching production
• Dependency drift (skill changes breaking parent commands)
• “I forgot to run tests” merges
• Version string lies (“v3.0.0” with no actual testing)

What this does not prevent:

• Eval suite blind spots (tests that don’t cover real failure modes)
• Model drift (provider updates change behavior under the same prompt)
• Malicious actors with commit access (they can fake attestations)
• Nondeterminism escaping multiple runs (rare but possible)

What you can add for defense in depth:

• Pin model versions in eval configs
• Run periodic revalidation (nightly CI that re-runs all attestations)
• Store eval artifacts for post-incident analysis
• Require multiple reviewers for attestation.json changes

Building Your Own

Here’s a minimal implementation in five steps.

Step 1: Set Up Promptfoo

Install Promptfoo and create an evaluation suite:

# evals/my-command/evals.yaml
providers:
  - id: openai:gpt-5.2

tests:
  - description: "Handles basic request"
    vars:
      input: "Create a new user"
    assert:
      - type: llm-rubric
        value: "Response asks for required fields (name, email)"

  - description: "Handles edge case"
    vars:
      input: "Create user with existing email"
    assert:
      - type: llm-rubric
        value: "Response detects conflict and suggests resolution"

Step 2: Hash Computation

import { createHash } from "node:crypto";

      166
      167  function computeHash(content: str

function computeHash(content: string): string {
  const body = stripFrontmatter(content);
  return `sha256:${createHash("sha256").update(body).digest("hex")}`;
}

function stripFrontmatter(content: string): string {
  const match = content.match(/^---\n[\s\S]*?\n---\n?/);
  return match ? content.slice(match[0].length) : content;
}

Step 3: Dependency Extraction

function buildDependencyGraph(entryPath: string): string[] {
  const visited = new Set<string>();
  const queue = [entryPath];

   while (queue.length > 0) {
    const current = queue.shift()!;
    if (visited.has(current)) continue;
    visited.add(current);
    const content = readFileSync(current, "utf-8");
    // Match your delegation syntax: "Agent: foo", "Skill: bar", etc.
    const deps = content.matchAll(/(?:Agent|Skill):\s*(\S+)/gi);
    for (const [, name] of deps) {
      queue.push(resolveFilePath(name));
    }
  }
  return Array.from(visited);
}

function computeDepsHash(files: string[]): string {
  const hashes = files.map(f => computeHash(readFileSync(f, "utf-8")));
  const sorted = [...hashes].sort(); // Deterministic ordering
  return `sha256:${createHash("sha256").update(sorted.join("|")).digest("hex")}`;
}

Step 4: Attestation Commands

async function attest(commandPath: string): Promise<void> {
  // Run eval (with multiple runs for nondeterminism)
  const passed = await runPromptfooEval(commandPath, { runs: 3 });
  if (!passed) throw new Error("Eval failed");
  const graph = buildDependencyGraph(commandPath);
  const manifest = JSON.parse(readFileSync("attestation.json", "utf-8"));
   manifest.commands[commandPath] = {
    prompt_hash: computeHash(readFileSync(commandPath, "utf-8")),
    deps_hash: computeDepsHash(graph),
    last_eval: { passed: true, timestamp: new Date().toISOString() }
  };
   writeFileSync("attestation.json", JSON.stringify(manifest, null, 2));
}

function verify(): boolean {
  const manifest = JSON.parse(readFileSync("attestation.json", "utf-8"));
  for (const [path, attestation] of Object.entries(manifest.commands)) {
    const graph = buildDependencyGraph(path);
    const currentDepsHash = computeDepsHash(graph);
    if (currentDepsHash !== attestation.deps_hash) {
      console.error(`Stale: ${path}`);
      return false;
    }
  }
  return true;
}

Step 5: CI Integration

# .github/workflows/ci.yml
attestation:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: oven-sh/setup-bun@v2
    - run: bun install --frozen-lockfile
    - run: bun run verify-attestations  # exits 1 if stale

Make this a required check. No green CI, no merge.

Full Schema

For reference, here’s the complete attestation manifest structure:

{
  "schema_version": "1.0.0",
  "commands": {
    "build-feature": {
      "prompt_hash": "sha256:6940348da866fb8a4e2c...",
      "deps_hash": "sha256:c1def3cf850e94445fa8...",
      "version": "3.0.0",
      "last_eval": {
        "passed": true,
        "timestamp": "2026-01-22T07:35:37.344Z",
        "git_commit": "ae02a41",
        "result_file": "evals/output/build-feature-2026-01-22.json"
      }
    }
  },
  "files": {
    "commands/build-feature.md": "sha256:6940348da866fb8a...",
    "agents/feature-builder.md": "sha256:ee45f612c99d0d48...",
    "skills/git-workflow.md": "sha256:c0f5beca9cba362e..."
  }
}

Conclusion

Prompts aren’t comments. They’re executable behavior.

Treat them as production artifacts with provenance, not as prose that gets eyeballed in review. If you can’t prove what you tested, you’re just hoping in YAML.

The implementation is straightforward: hash the content, track dependencies, run evals, gate CI. About 500 lines of TypeScript, using open-source tools (Promptfoo, Node.js crypto, your CI system of choice).

The worktree bug that started this post? It would have been caught. Multiple eval runs would have exposed the flakiness. The attestation would have failed. The merge would have been blocked.

That’s the point. Not perfection; just proof.

Build this system. Stop shipping untested prompts.

Learn More

This attestation system is one piece of rp1, an open-source agentic development platform that handles the full workflow: requirements gathering, design, implementation, testing, and code review. If you’re building with AI agents like Claude Code and still writing your workflow prompts by hand, give rp1 a try.