I Replaced My Best Claude Skill With An Agent That Refuses To Ship Until It Grades Itself

Anthropic shipped Outcomes on May 6, 2026. It’s the most important agent primitive of the year, and almost nobody is writing about the architecture.

For the last eight months I’ve been running a local Claude Code skill called ORACLE PRIME. It scans about thirty sources every morning, scores topics on six dimensions, and produces a publish-ready brief for the articles I’m going to write that week.

It works. It also lies to me.

Not deliberately. The brief is well-formatted, the topics look reasonable, the headlines look smart. But sometimes a topic clears the score floor because the scoring rubric was too generous in the parts of the brief I never re-read. Sometimes a headline drifts into LLM-cliché territory because the voice-check is a paragraph at the bottom of the skill file that nothing actively enforces. Once, the brief recommended a topic that was already on my “do-not-write-again” list. Nothing failed loudly. The brief just wasn’t as good as I assumed it was, and I had no system to catch it.

I rebuilt the whole thing on Claude Managed Agents over a weekend. The new version refuses to ship a brief until a separate AI grader, running in its own context window, confirms that every criterion in a twelve-item rubric is satisfied. If the grader is unhappy, the agent gets the specific gaps as feedback and iterates. Up to five times. If it still can’t satisfy the rubric, the session idles with whatever the final draft is and a webhook tells me to look at it manually.

What follows is the working architecture, the six API calls that make it run, the cost math down to per-scan rupees, and the design pattern I think will outlast every current agent framework.

The pattern in one paragraph

The pattern is this: you stop asking your agent “are you done?” and instead hand it a rubric and a separate grader that answers that question on its behalf. The agent writes. The grader reads. The grader either says “satisfied” or hands back a per-criterion list of what’s missing. The writer revises against the gaps. The loop terminates when the grader is happy, or when you hit the iteration ceiling. The grader never sees the writer’s reasoning. It re-reads the artifact each time as if it’s the first time.

This is not Anthropic-specific. It’s not Managed-Agents-specific. It’s a design pattern. You can build it on LangGraph, you can build it on CrewAI, you can build it on a hand-rolled loop. Managed Agents just gives you the cleanest primitive for it. The pattern survives whichever framework wins.

The pattern that matters isn’t the agent. It’s the second agent reading the first agent’s homework.

I’ll come back to the design-pattern argument near the end. First, the architecture.

Why Outcomes is the actual story, not multi-agent orchestration

Anthropic announced three things on May 6 at Code with Claude SF: Dreaming, Outcomes, and Multiagent. The press cycle that week was almost entirely about multi-agent — fifty parallel scanners feel cinematic, parallelism is easy to demo, “agents that spin up agents” headlines well.

Multi-agent is useful. I use it. The coordinator in my new ORACLE PRIME spawns three parallel source scanners and routes work to two more specialists.

But multi-agent without Outcomes is just orchestration. You still have no enforced definition of done. You still ship whatever the coordinator decides looks good, and you still hope the user catches the quality regressions. The thing that actually changes the economics of building agents is the grader. The grader is what lets you charge money for the output of an agent system without quietly losing reputation every time the model has a bad day.

The grader is the moat.

The six API calls

The whole system is six API calls. I’m going to list them, then walk through the configs that matter.

1. Upload the rubric to the Files API once, get a file_id back, never touch it again unless the rubric changes
2. Create the environment — a Python 3.12 container with the right packages and an allowlist of outbound hosts
3. Create the four specialist agents — source scanner, CVS scorer, psychology extractor, red-team critic — each with its own system prompt, model choice, and tool scope
4. Create the coordinator agent with multiagent.agents wired to the four specialist IDs above
5. Open a session against the coordinator inside the environment, with a title and metadata
6. Send a user.define_outcome event carrying the rubric file_id and a task description, with max_iterations: 5

That’s it. The agent starts working the instant the event is sent. Everything from there happens server-side until a webhook fires on session.status_idled.

The two design choices that matter inside those six calls are the rubric and the coordinator’s system prompt. Everything else is plumbing.

The coordinator system prompt

The coordinator’s job is narrow: hydrate memory, delegate the phases that have specialists, do the synthesis phases itself, and submit the final brief for grading. Here’s the working system prompt, cut down to the parts that earn their place:

name: oracle-coordinator
model:
  id: claude-opus-4-7

system: |
  You are ORACLE PRIME COORDINATOR. You measure success by the
  Outcomes grader, not by what you think looks good. When the grader
  returns `needs_revision`, you treat the feedback as orders.

  EXECUTION PROTOCOL — run in strict sequence:

  Phase 1 — Memory Hydration (DO THIS YOURSELF)
    Read /mnt/oracle/memory/anti-list.json
    Read /mnt/oracle/memory/voice-fingerprint.json
    Read /mnt/oracle/memory/calibration-config.yaml

  Phase 2-3 — Source Sweep (DELEGATE)
    Spawn 3 parallel oracle-source-scanner threads:
      Thread A: 0-24h horizon
      Thread B: 2-7d horizon
      Thread C: 8-30d horizon
    Wait. Aggregate.

  Phase 4 — CVS Scoring (DELEGATE to oracle-cvs-scorer)
    Reject any candidate below CVS 7.0. Do not negotiate this.

  Phase 5 — Psychology Extraction (DELEGATE)

  Phase 6 — Fusion + Tier Classification (DO THIS YOURSELF)

  Phase 7 — Red-Team Critique (DELEGATE)
    If critic remains unsatisfied after 1 revision, ship anyway.
    The Outcomes grader is the final arbiter.

  DO NOT
  * Do not summarize past sessions in your output.
  * Do not include any reasoning trace in the final brief.
  * Do not deliver if any topic is in the anti-list. Verify first.

multiagent:
  type: coordinator
  agents:
    * {type: agent, id: "${SOURCE_SCANNER_AGENT_ID}"}
    * {type: agent, id: "${CVS_SCORER_AGENT_ID}"}
    * {type: agent, id: "${PSYCHOLOGY_EXTRACTOR_AGENT_ID}"}
    * {type: agent, id: "${REDTEAM_CRITIC_AGENT_ID}"}

The “do not” section is where most of the engineering happens. Coordinator prompts are usually written as a list of instructions. They should be written as a list of refusals. The instructions are what the agent does. The refusals are what makes the output usable.

One specific choice worth flagging: Opus 4.7 for the coordinator, Sonnet 4.6 for the I/O-bound scanners and the psychology extractor, Opus 4.7 again for the scorer and critic. This is a roughly sixty-percent cost reduction against running everything on Opus, with no measurable quality loss on the agents that are mostly fetching and pattern-matching. The agents that need depth — synthesis, scoring, adversarial critique — get the bigger model. The agents that need throughput get the cheaper one.

The rubric is the contract

The rubric is a markdown document the grader reads in a fresh context window after each iteration. The grader doesn’t see the coordinator’s thinking. It only sees the final artifact and the rubric. Here’s the rubric criteria as I ship them now:

# ORACLE PRIME — Publish-Ready Brief Rubric

## C1. Topic Count & Velocity Tiering
* Exactly 7-10 topics
* Each topic classified STRIKE / AMBUSH / CITADEL / MOAT
* No "uncategorized" or "mixed" allowed

## C2. CVS Scoring Floor
* Every topic CVS >= 7.0
* All six dimensional scores surfaced

## C3. Anti-List Compliance
* Checked against /mnt/oracle/memory/anti-list.json
* Confirmation block at top of brief:
  "Anti-list check: PASSED. N entries checked. Zero matches."

## C4. Headline Forensics
* Exactly 3 distinct headline variants per topic
* Each headline <= 14 words
* No two variants differ by fewer than 5 words
* No flagged clickbait tokens

## C5. Voice Fingerprint Preservation
* No LLM-cliché phrases:
    "delve into", "navigate the landscape",
    "in the realm of", "unlock the potential",
    "transform your workflow"
* Grader runs 5-pass spot-check; reports drift verbatim

## C9. Red-Team Critique Log
* Brief includes "## Red-Team Audit" footer
* Top 3 critiques + coordinator response

## C10. Output Schema Conformance
* File at /mnt/session/outputs/brief.md
* Frontmatter present
* Headers match schema exactly

## C11. Length Ceiling
* <= 8,000 words total
* <= 600 words per topic

I’ve cut four of the twelve criteria here for readability — the full version covers arbitrage evidence on every strike topic, Boost-pattern alignment on the top three, cover-image prompts, and pipeline-lock metadata for the downstream writer. The full file is twelve criteria. Every one is binary. The grader passes the whole rubric or it returns the specific failures.

A rubric is not a wish list. It’s a contract. The discipline is to write criteria that can be verified by reading the artifact. “The brief should be insightful” is not a rubric line. “Every topic with tier STRIKE or AMBUSH includes a cited cross-platform source with URL and timestamp” is. The grader can check the second. It cannot check the first.

The session-trigger script

The script that actually fires the session is the smallest piece of code in the system. I run it from a cron job on my VPS at 03:00 IST every morning:

import os, anthropic
from datetime import datetime, timezone

client = anthropic.Anthropic()
now = datetime.now(timezone.utc)

# 1. Open the session
session = client.beta.sessions.create(
    agent=os.environ["COORDINATOR_AGENT_ID"],
    environment_id=os.environ["ORACLE_ENV_ID"],
    title=f"ORACLE PRIME daily scan — {now.strftime('%Y-%m-%d')}",
    metadata={
        "product": "oracle-prime-hosted",
        "operator": "anup",
        "scan_date": now.strftime("%Y-%m-%d"),
    },
)

# 2. Define the outcome — the agent starts working on receipt
client.beta.sessions.events.send(
    session_id=session.id,
    events=[{
        "type": "user.define_outcome",
        "description": (
            f"Produce a publish-ready intelligence brief for "
            f"{now.strftime('%A, %B %d, %Y')}. Run the full 7-phase "
            f"pipeline, delegate to specialists per your system prompt, "
            f"and write the final brief to /mnt/session/outputs/brief.md. "
            f"The brief must satisfy every criterion in the attached rubric."
        ),
        "rubric": {
            "type": "file",
            "file_id": os.environ["RUBRIC_FILE_ID"],
        },
        "max_iterations": 5,
    }],
)
print(f"Session kicked off: {session.id}")

Twenty-three lines, including imports. The session fires, the cron job exits, the VPS goes back to sleep. The actual work runs server-side for about seventeen minutes. Then a webhook arrives.

The webhook handler

When the session idles — either because the grader returned satisfied or because we hit max_iterations_reached — Anthropic fires session.status_idled to my registered endpoint. The handler verifies the signature, fetches the session, downloads the brief, and posts it to my Telegram. The cost gets logged to Supabase for analytics:

from fastapi import FastAPI, Request, HTTPException
import anthropic, httpx, os
from pathlib import Path

app = FastAPI()
client = anthropic.Anthropic()  # reads ANTHROPIC_WEBHOOK_SIGNING_KEY

@app.post("/webhook")
async def webhook(request: Request):
    raw = await request.body()
    try:
        event = client.beta.webhooks.unwrap(
            raw.decode("utf-8"),
            headers=dict(request.headers),
        )
    except Exception:
        raise HTTPException(400, "invalid signature")

    if event.data.type != "session.status_idled":
        return {"received": True}

    session = client.beta.sessions.retrieve(event.data.id)
    if session.metadata.get("product") != "oracle-prime-hosted":
        return {"received": True}

    outcomes = session.outcome_evaluations or []
    if not outcomes or outcomes[-1].result != "satisfied":
        return {"received": True}

    # Download the brief
    files = client.beta.files.list(scope_id=event.data.id)
    brief = next((f for f in files.data if f.filename == "brief.md"), None)
    if not brief:
        return {"received": True}

    path = Path(f"/tmp/brief-{event.data.id}.md")
    client.beta.files.download(brief.id).write_to_file(str(path))

    # Send to Telegram
    async with httpx.AsyncClient() as ac:
        await ac.post(
            f"https://api.telegram.org/bot{os.environ['TG_TOKEN']}/sendDocument",
            data={"chat_id": os.environ["TG_CHAT_ID"]},
            files={"document": ("brief.md", path.open("rb"))},
        )

    return {"received": True}

unwrap() is doing two things in one call: HMAC verification with my signing secret, and a hard rejection of any payload older than five minutes. Both of those are mandatory for a production webhook. Forgetting the staleness check is how replay attacks happen.

The Telegram delivery is one HTTP POST. That’s the whole product interface for me as an operator. The brief lands on my phone. I read it on the commute. I pick the article to write.

The cost math

Anthropic prices Managed Agents on two dimensions: standard token rates for whichever model the session uses, plus eight cents per session-hour of active runtime. Idle time is free. The session-hour rate replaces the Code Execution container-hour pricing, so you’re not double-charged for the container.

Here’s the per-scan cost of one ORACLE PRIME run, computed from the calculator I ship alongside the configs:

Phase                         Model               In       Out       $
  -------------------------------------------------------------------------
  Coordinator (1, 6, 7)         claude-opus-4-7   30,000    20,000  $0.650
  Source Scanner ×3 (2-3)       claude-sonnet-4-6 80,000    15,000  $0.715
  CVS Scorer (4)                claude-opus-4-7   50,000    10,000  $0.500
  Psychology Extractor (5)      claude-sonnet-4-6 60,000    12,000  $0.410
  Red-Team Critic (7)           claude-opus-4-7   25,000     8,000  $0.325
  Outcomes Grader (separate)    claude-opus-4-7   30,000     5,000  $0.275
  -------------------------------------------------------------------------
  TOKEN SUBTOTAL                                                    $2.875
  Session runtime (1.0 hr × $0.08/hr)                               $0.080
  TOTAL PER SCAN (no cache)                                         $2.955
  TOTAL PER SCAN (70% cache hit on stable context)                  $2.360

Two-thirty-six per scan in steady state. About a hundred and ninety-seven rupees. For a daily run that gives me an hour-long, self-graded, multi-agent, parallel-scanned intelligence brief on the desk every morning. Thirty scans a month is fifty-nine dollars, give or take.

Compare that to what I was paying in opportunity cost when I ran the same skill locally. The local version blocked my laptop for fifteen minutes every morning. It required me to be at the laptop. It produced output I had to manually re-read for anti-list violations, voice drift, and CVS sanity. The new version costs me less than a coffee a week and runs while I’m asleep.

The runtime cost is barely visible in the table. Eight cents an hour is a rounding error against the token spend. This is by design. Anthropic priced runtime to be almost free so that the real economic decision is which model goes where, not whether to use Managed Agents at all. Pick the right model for each agent and the architecture pays for itself.

Two-thirty-six per scan, runs while I sleep, refuses to ship if it doesn’t meet the rubric. The unit economics flipped completely.

What happens when the grader returns needs_revision

This is the part of the architecture that earns the design pattern’s reputation. When the grader is unhappy, it doesn’t just refuse — it returns a structured list of which criteria failed and why, in the grader’s own words. The coordinator gets that as input on the next iteration.

A real revision log from a session that ran last week, paraphrased:

• Iteration 0: grader returns needs_revision. "C4: Topic 5 has only 2 headline variants; rubric requires 3. C8: Topic 2's hook opens with 'In today's fast-paced AI landscape', which appears in the drift indicators list."
• Coordinator regenerates the headline variant for Topic 5, rewrites Topic 2’s hook to open differently.
• Iteration 1: grader returns needs_revision. "C4 satisfied. C8 satisfied. New issue C2: Topic 7 CVS score reported as 6.9; rubric requires floor 7.0."
• Coordinator drops Topic 7 entirely, replaces it with the next-ranked candidate from the scored pool.
• Iteration 2: grader returns satisfied. Session idles. Webhook fires. Brief lands in Telegram.

The session burned three iterations instead of one. That cost me about another seventy cents in additional grader runs and coordinator revisions. In exchange, the brief that actually shipped had zero clichéd hooks, zero topics below the score floor, and zero rubric violations. Worth it every time.

The max_iterations: 5 cap is a safety net, not a target. If a session hits the cap, something is wrong with either the rubric or the writer. Either the writer can't satisfy the rubric (writer side problem — pick a stronger model, expand the system prompt) or the rubric is unverifiable from the artifact alone (rubric side problem — rewrite the criteria to be more concrete).

The grader is also the unlock that makes selling agent output viable. When a paying customer gets a brief in their inbox, the worst-case scenario isn’t a low-quality brief — the worst-case scenario is no brief, because the grader refused to satisfy. That’s a much better failure mode than the alternative. Silent quality degradation is reputational damage. Loud refusal is a system working correctly.

The design pattern survives every model upgrade

Every agent framework today is being reinvented every six weeks. LangGraph, CrewAI, AutoGen, Letta, the Claude Agent SDK, the dozen new orchestrators that landed in the last quarter — they’re all moving targets. Locking your business logic to any of them is a losing bet.

The self-grading pattern isn’t a feature of Managed Agents. Managed Agents has the cleanest implementation of it right now, and Anthropic markets it as Outcomes, but the pattern is architecture, not product. You can build the same loop on top of LangGraph by adding an eval-node that re-reads the artifact and routes the graph back to the writer node if the rubric fails. You can build it on a hand-rolled loop in fifty lines of Python — two model calls, a rubric prompt, an if rubric_satisfied: break block.

What Anthropic added by shipping Outcomes is the primitive: a built-in, context-isolated grader with structured feedback events, an iteration counter, and a clean termination contract. That makes the pattern cheaper to ship. It doesn’t make the pattern Anthropic’s.

The reason this matters: when a new agent framework wins next quarter — and one will — your agents move. Your rubric stays. The rubric is the part of your system that encodes what “good” means for your domain, in language a grader can verify. That’s the durable asset.

If you take one thing from this piece, take this. Don’t fall in love with whatever harness you’re using. Fall in love with the rubric. The rubric is what you’ll port forward.

Resources

• Claude Managed Agents — Overview
• Define Outcomes — Anthropic Docs
• Webhooks — Anthropic Docs
• Multiagent Sessions — Anthropic Docs
• Outcomes Cookbook — verify with grader
• Pricing reference
• wowhow.cloud — the AI store

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!