How to Evaluate AI Agent Skills Without Relying on Vibes

A practical, platform-agnostic guide to skill evaluation, from first test case to production pipeline

Terminate your worries by using evals to get measured confidence.

This article is deeply indebted to OpenAI’s excellent guide, “Testing Agent Skills Systematically with Evals”. What follows is my attempt to expand on their methodology, make it accessible to newcomers, and show how these principles apply regardless of which AI platform you’re building on. If you haven’t read the original, I’d encourage you to. It’s one of the clearest pieces of technical writing on this topic.

Okay, let’s go.

Here’s another one of my countless confessions. For months, I evaluated my AI agent skills by running them a few times and asking myself: “Does this feel right?”

Vibes. That’s what I was relying on. And I suspect I’m not alone.

One version felt faster. Another seemed more reliable. Then something would break in production that worked fine in my tests. The skill wouldn’t trigger. It would skip a required step. It would leave files in the wrong place. I’d fix it, run a few more manual tests, and convince myself the problem was solved.

It wasn’t.

The uncomfortable truth is this: we’ve built increasingly powerful AI agents, but our ability to measure them hasn’t kept pace. And the tools we do have might be misleading us. As Andrej Karpathy put it:

“LLM evaluations are notoriously difficult. The eval is often harder than the task itself.”

This guide exists because “it feels better” isn’t good enough. We need proof.

What are agent skills and why does evaluation matter?

At its core, a skill is an organised collection of prompts and instructions for a language model. Think of it as a reusable behaviour you want your agent to perform reliably: setting up a project, parsing data, writing tests, or handling customer queries.

[embed]Writing OpenCode Agent Skills: A Practical Guide with Examples Writing OpenCode Agent Skills: A Practical Guide with Examples Everything you need on your toolbelt to create reusable…jpcaparas.medium.com

Here’s something interesting. Despite the fragmented AI landscape, the industry has quietly converged on a common format for defining these skills.

Whether you’re using OpenAI, Anthropic, Google, or open-source frameworks, you’re probably working with some variant of JSON Schema:

OpenAI (Functions/Tools): JSON Schema with name, description, and parameters

Anthropic Claude: JSON with name, description, and input_schema

Google Gemini: Protocol buffer-style function declarations

LangChain: Python decorators or Tool objects that compile to the same structure

MCP (Model Context Protocol): An open standard for tools, resources, and prompts

This convergence matters. It means the evaluation principles in this guide work across platforms. You’re not locked into one vendor’s approach.

But why evaluate at all? Consider the gap between AI agents and humans on real-world tasks:

Sources: SWE-bench, GAIA, WebArena, OSWorld Leaderboards (2024–2025)

On coding tasks (SWE-bench Verified), top agents reach about 74% of resolved issues compared to roughly 90% for humans. That’s impressive. But on interactive desktop tasks (OSWorld), agents manage just 22% while humans hit 72%. The gap isn’t uniform, and evaluation helps you understand where your specific skills sit on this spectrum.

Define success before you write the skill

This is the foundation everything else builds on. Before writing a single line of skill logic, write down what “success” means in terms you can actually measure.

A useful framework splits success criteria into four categories:

Outcome goals: Did the task complete? Does the app run? Is the output correct?

Process goals: Did the agent invoke the skill when it should? Did it follow the steps you intended? Did it use the right tools?

Style goals: Does the output follow your conventions? Is the code formatted correctly? Are the files in the right places?

Efficiency goals: Did it get there without thrashing? Did it avoid unnecessary commands or excessive token use?

Here’s an example definition of done for a skill that sets up a demo React app:

Definition of done:
- npm run dev starts successfully
- package.json exists in project root
- src/components/Header.tsx exists
- src/components/Card.tsx exists
- No more than 15 command executions
- Total token usage under 50,000

Notice how each criterion is binary. You can check it programmatically. There’s no ambiguity about whether the skill succeeded.

Keep this list small and focused on must-pass checks. The goal isn’t to encode every preference up front. It’s to capture the behaviours you care about most.

Start small with a targeted prompt set

“50–100 well-chosen examples often outperform thousands of poorly chosen ones.” — Anthropic Engineering

You don’t need a massive benchmark to get value from evaluation. For a single skill, 20–50 prompts is enough to surface regressions and confirm improvements early.

Start with a simple CSV structure:

id,should_trigger,prompt
test-01,true,"Create a demo app using the setup-demo skill"
test-02,true,"Set up a minimal React demo with Tailwind"
test-03,true,"I need a quick UI prototype for testing"
test-04,false,"Add Tailwind styling to my existing React app"
test-05,false,"Fix the CSS in my current project"

Each row tests something different:

Explicit invocation (test-01): The prompt names the skill directly. This confirms basic functionality.

Implicit invocation (test-02): The prompt describes what the skill does without naming it. This tests whether your skill’s description is strong enough for automatic selection.

Contextual invocation (test-03): A realistic, slightly noisy prompt that should still trigger the skill.

Negative controls (test-04, test-05): Prompts that should NOT trigger the skill. These catch false positives where your skill activates too eagerly.

The negative controls are critical. I’ve watched skills hijack prompts they were never meant to handle because the description was too broad. Without negative test cases, you won’t catch this until production.

As you discover failures (prompts that don’t trigger when they should, or outputs that drift from expectations), add them as new rows. Over time, this small CSV becomes a living record of what your skill must continue to get right.

Deterministic graders: your first line of defence

Deterministic checks give you fast, explainable signals. They’re cheap to run, produce consistent results, and make failures easy to debug.

Here’s what they can catch:

Command execution: Did the agent run the commands you expected?

File creation: Do the required files exist?

Sequence verification: Were steps executed in the right order?

Output format: Does the result match expected patterns?

The key is capturing a trace of what actually happened. Most agent frameworks can output structured logs in JSON or JSONL format. Here’s a simplified example of parsing such a trace:

import json
from pathlib import Path
​
def load_trace(trace_path: str) -> list[dict]:
    """Load JSONL trace file into list of events."""
    events = []
    with open(trace_path) as f:
        for line in f:
            if line.strip():
                events.append(json.loads(line))
    return events
​
def check_command_executed(events: list[dict], command: str) -> bool:
    """Check if a specific command was executed."""
    for event in events:
        if event.get("type") == "command_execution":
            if command in event.get("command", ""):
                return True
    return False
​
def check_file_exists(project_dir: str, filename: str) -> bool:
    """Check if a file was created."""
    return Path(project_dir, filename).exists()
​
# Example usage
trace = load_trace("./evals/artifacts/test-01.jsonl")
results = {
    "ran_npm_install": check_command_executed(trace, "npm install"),
    "has_package_json": check_file_exists("./demo-app", "package.json"),
    "has_header_component": check_file_exists("./demo-app", "src/components/Header.tsx"),
}
print(results)

This pattern works across platforms. OpenAI’s Codex outputs JSONL traces. LangSmith captures similar structured data. Even if you’re rolling your own agent, you can log events in this format.

The beauty of deterministic graders is debuggability. When a check fails, you open the trace file and see exactly what happened. Every command appears in sequence. There’s no mystery about why something broke.

LLM-as-judge for qualitative evaluation

Deterministic checks answer “did it do the basics?” but they can’t answer “did it do it well?”

Many requirements are qualitative: Is the code well-structured? Does it follow conventions? Is the output readable? These are hard to capture with file existence checks or command counts.

The solution is LLM-as-judge evaluation, where you use a language model to grade outputs against a rubric. Here’s a simplified example:

def evaluate_with_rubric(code_path: str, rubric: dict) -> dict:
    """
    Use an LLM to evaluate code against a rubric.
    Returns structured scores.
    """
    code_content = Path(code_path).read_text()

    prompt = f"""
    Evaluate this code against the following criteria.
    Return a JSON object with scores (0-100) for each criterion.

    Criteria:
    {json.dumps(rubric, indent=2)}

    Code to evaluate:
    {code_content}
    """

    # Call your preferred LLM API here
    # response = llm.generate(prompt, response_format="json")
    # return response

A simple rubric might look like:

{
  "criteria": {
    "structure": "Code is organised into logical components",
    "conventions": "Follows TypeScript/React best practices",
    "styling": "Uses Tailwind classes consistently",
    "readability": "Code is clear and well-commented"
  },
  "pass_threshold": 70
}

But LLM-as-judge has known pitfalls:

Position bias: LLMs tend to prefer options that appear first or last. Mitigation: randomise order, test both positions.

Verbosity bias: Longer responses often get higher scores. Mitigation: use explicit length-agnostic criteria.

Self-preference: Models prefer outputs from their own family. Mitigation: use a different model family as the judge.

Inconsistency: The same input can get different scores across runs. Mitigation: lower temperature, run multiple times, aggregate results.

Research shows GPT-4 class models achieve 70–85% agreement with human evaluators on well-defined tasks. That’s useful, but it’s not a replacement for human judgment on high-stakes decisions.

The evaluation economics

Here’s where many teams get stuck. Evaluation isn’t free. Costs spiral quickly if you’re not careful.

The smart approach is tiered evaluation:

Tier 1 (Deterministic): Run on every commit. Costs essentially nothing. Catches obvious regressions fast.

Tier 2 (LLM-as-Judge): Run on PRs or nightly builds. At $0.01–0.20 per evaluation, you can afford hundreds per day but not thousands.

Tier 3 (Human Review): Reserve for calibration, edge cases, and high-stakes decisions. At $0.50–5.00 per evaluation, this is precious and should be used sparingly.

Look, the maths matters. If you’re running 1,000 evaluations per day:

• Deterministic only: ~$0
• LLM-as-judge at $0.10 each: $100/day ($3,000/month)
• Human at $2.00 each: $2,000/day ($60,000/month)

Build evaluation budgets from day one. Decide how much you’re willing to spend on confidence, and design your pipeline accordingly.

“The best eval is one that actually gets run.” — Anthropic Engineering

An expensive evaluation that nobody runs is worthless. A cheap evaluation that runs on every commit is invaluable.

Extending your evals as skills mature

Once you have the core loop in place, you can extend evaluations in directions that matter for your specific skills:

Command count and thrashing detection: Count command executions in your traces. If the number spikes, the agent might be looping or retrying unnecessarily. Set thresholds and alert when exceeded.

Token budget monitoring: Track input and output tokens per run. This catches prompt bloat and helps compare efficiency across skill versions.

Build and runtime checks: Run the actual build after the skill completes. For a React app, that’s npm run build. For a Python project, pytest. These act as strong end-to-end signals.

Repository cleanliness: Ensure runs generate no unwanted files. Check that git status --porcelain is empty (or matches an explicit allow list).

CI/CD integration: Hook evaluations into your deployment pipeline. Block merges if core evaluations fail. Run extended evaluations nightly.

That’s pretty easy to understand if you’ve been used to creating pipelines.

Several tools can help:

LangSmith: Deep tracing and experiment tracking for LangChain users

Braintrust: Real-time scoring with CI/CD integration

Promptfoo: Open source, YAML-based, great for red teaming

Arize Phoenix: OpenTelemetry-based observability with drift detection

DeepEval: 14+ built-in metrics for RAG and agent evaluation

The pattern is consistent: begin with fast checks that explain behaviour, then add slower checks only when they reduce real risk.

Common mistakes and how to avoid them

I’ve made all of these mistakes. Hopefully you can skip a few.

Testing only happy paths. It’s tempting to test the scenarios you expect to work. But agents fail on edge cases, not typical scenarios. Deliberately construct adversarial test cases. What happens with malformed input? Missing dependencies? Conflicting instructions?

Single-run evaluations. Agents are non-deterministic. Running once captures one sample from a distribution. Use pass@k metrics (probability of at least one success in k attempts). Run 5–10 times minimum for any evaluation that matters.

Overfitting to benchmarks. Goodhart’s Law applies: when a measure becomes a target, it ceases to be a good measure. If you optimise purely for benchmark scores, you’ll eventually game the benchmark without improving real-world performance.

Ignoring evaluation costs. $0.10 per evaluation seems cheap until you’re running 100,000 per month. Budget for evaluation from the start.

Static evaluation sets. Production reveals failure modes you never anticipated. Continuously add cases from production failures. Your evaluation set should grow over time.

Evaluating in isolation. Skills operate within systems. A skill that works perfectly in isolation might fail when integrated with other components. Include integration tests in your evaluation suite.

Red = High severity (testing happy paths, single-run evals), Yellow = Medium severity (overfitting, ignoring costs, static sets)

Pulling it all together

Let’s recap the approach:

1. Define success before you write the skill. Make criteria measurable and binary.
2. Start with a small, targeted prompt set. 20–50 cases, including negative controls.
3. Build deterministic graders first. Cheap, fast, debuggable.
4. Add LLM-as-judge for qualitative checks. Use rubrics and watch for known biases.
5. Structure evaluation economics. Tier your evaluations from cheap to expensive.
6. Extend as skills mature. Add command counts, token budgets, build checks.
7. Avoid common mistakes. Test edge cases, run multiple times, budget costs.
8. Feed production failures back into evals. Your test set should be a living document.

The shift from “it feels better” to “I have proof” is uncomfortable at first. It requires discipline. It requires infrastructure. And honestly? It requires admitting that your intuition isn’t always right.

But once that loop exists, every tweak becomes easier to confirm. Every regression becomes clear. You stop arguing about whether a change improved things and start showing the data.

I want to close by acknowledging OpenAI again. Their guide on Testing Agent Skills Systematically with Evals provided the foundation for everything here. If you’re serious about building reliable AI agents, it’s essential reading.

The principles they outline aren’t proprietary to their platform. They’re universal. Whether you’re building with Claude, Gemini, LangChain, or something entirely custom, the same fundamental question applies: how do you know your skill is actually getting better?

Now you have a framework to answer it.

References

• Testing Agent Skills Systematically with Evals — OpenAI’s original guide that inspired this article
• Demystifying Evals for AI Agents — Anthropic’s 8-step evaluation roadmap
• LangSmith Evaluation Documentation — Guide to evaluation with LangSmith
• Braintrust Evaluation Guide — Production-focused evaluation patterns
• Promptfoo Documentation — Open source evaluation and red teaming
• MT-Bench Paper — Research on LLM-as-judge methodology
• SWE-bench Leaderboard — Coding task benchmark data
• GAIA Leaderboard — General AI assistant benchmark data