Stop Trusting AI Test Cases on Sight. Here’s How to Score Them Instead

Every QA team I talk to right now is experimenting with LLMs for test case generation. The workflow is usually the same: write a prompt, get test cases, review them, ship them. Job done.

But here’s the question nobody seems to be asking: how do you actually know if the output is good?

“It looks right” isn’t a quality standard. Neither is “the senior QA reviewed it.” We wouldn’t accept that for code. We shouldn’t accept it for AI-generated test artefacts either.

So I built something to answer it properly.

What I built

llm-eval-lab is a Python framework that takes the same acceptance criteria, sends them to Claude, GPT-4o, and GitHub Copilot, and then scores the generated test cases objectively using two evaluation frameworks — DeepEval and Promptfoo.

I tested two AC formats because real QA teams work with both: Gherkin scenarios (Given/When/Then) and CSV requirements mapping documents. The sample domain is an e-commerce shopping cart — stable enough to be consistent, complex enough to produce meaningful variation in outputs.

The goal wasn’t to crown a winner. It was to build a repeatable method for measuring quality — and to see what the numbers actually said when I ran it.

The evaluation approach

DeepEval — three metrics per output

DeepEval scores each generated output on a 0–1 scale across three dimensions.

Answer Relevancy — are the test cases relevant to this specific acceptance criteria, or generic boilerplate that could apply to any feature? Low scores here mean the LLM drifted, ignored the specifics, or pattern-matched to something familiar instead of reading what was actually in front of it.

Faithfulness — did the LLM stay faithful to what the AC said, or did it hallucinate requirements? A low faithfulness score means the model added conditions, constraints, or scenarios that weren’t in the source material. In testing, that’s not creative — it’s wrong.

Test Case Quality (Custom G-Eval) — this is the primary metric. An LLM judge scores completeness (happy path, negatives, boundaries), specificity (uses actual data from the AC rather than placeholders), executability (clear steps a tester can follow), and structure (consistent format throughout). Pass threshold is 0.5 across all three metrics.

One practical note: DeepEval uses GPT-4o as the evaluation judge. That means running it requires an OpenAI API key with credits — it’s not free. If you want to generate test cases via the Claude or OpenAI APIs directly, you’ll need Anthropic and OpenAI credits respectively. In my run, I collected Copilot outputs manually and dropped them into a folder for evaluation — which is a perfectly valid approach and how most teams would use Copilot anyway.

Promptfoo — assertion-based regression testing

Promptfoo works differently. Rather than scoring outputs continuously, it sends prompts to the LLMs itself via API and checks responses against explicit pass/fail assertions — things like “does the output reference the specific dollar value from the AC” or “does it include negative test cases, not just the happy path.”

It’s less about nuanced quality scoring and more about deterministic gates. The natural home for Promptfoo is CI/CD — run it on every prompt change and fail the build if assertions drop.

In my run, Promptfoo called GPT-4o successfully. Claude errored out because my Anthropic API credits were low at the time. That’s worth being honest about — Promptfoo is powerful but it does need live API access for both models to give you a full comparison. The GPT-4o results alone were still useful: it passed most CSV requirement assertions but failed several Gherkin scenarios, suggesting it handles structured formats more reliably than narrative ones.

Worth noting too: DeepEval and Promptfoo solve different problems even when they overlap on models. DeepEval scores outputs you already have — generated via API where you have credits, or saved manually where you don’t. Promptfoo generates and evaluates in the same step, but only for models it can reach live. That’s why Copilot shows up in my DeepEval results but not in Promptfoo — it has no API for Promptfoo to call.

The results

Here’s what DeepEval returned on a Gherkin AC run:

Claude scored perfectly across all three metrics. OpenAI’s relevancy score was unavailable due to a scoring edge case in that run, but its faithfulness and quality scores were strong and it passed overall. Both are solid results.

Then there’s Copilot.

The finding that actually matters

Copilot scored 1.000 on faithfulness and 0.964 on quality — nearly identical to Claude. The test cases were well-structured, covered the right scenario types, and didn’t hallucinate anything.

But it scored 0.184 on relevancy. And that single metric failed the entire output.

What happened? Copilot generated test cases that were technically correct and well-formed — but not grounded in the specific acceptance criteria I gave it. It produced test cases that could belong to any shopping cart feature. The Gherkin scenario I provided had specific conditions, specific data values, specific edge cases. Copilot’s output didn’t reference them. It looked like testing. It wasn’t testing this.

This is a failure mode that’s easy to miss in manual review. A test case with clear steps, a proper expected result, and sensible structure reads as good work. You’d probably approve it in a review. The evaluator caught what human review would likely let through.

That’s precisely why measurement matters.

What I’d do differently

Run DeepEval on more AC samples. A single Gherkin run is a data point, not a conclusion. The framework is built to scale — the next step is running it across a broader set of ACs, including negative and ambiguous ones, to see if the patterns hold.

Top up Anthropic credits before running Promptfoo. The incomplete Claude comparison in Promptfoo was purely a credits issue. With both models running, the assertion-level comparison would have been more useful.

Add a format comparison. I built the framework to handle both Gherkin and CSV, but I haven’t yet published a side-by-side comparison of how the same LLM performs across both formats. That’s the next experiment — and I suspect the results will be different enough to be worth writing about separately.

The simplification I should be upfront about

There’s a gap in how I scored this, and it’s worth naming rather than glossing over.

DeepEval’s relevancy and faithfulness metrics work by comparing the actual output against an expected_output — what the framework calls a golden answer. In a properly rigorous setup, that golden answer is a real, hand-written ideal test case for each acceptance criteria, created by an experienced QA engineer or pulled from a curated dataset of past test cases. DeepEval calls this a Golden object — input, expected output, sometimes additional context.

I didn’t do that. Writing a genuine golden test case for all 14 ACs in this run would have meant manually authoring perfect output for every scenario before I could measure anything — which defeats some of the point of moving fast. Instead, I used a single generic rubric as the expected output across every AC:

expected_output = (
    "A set of specific, executable test cases covering the happy path, "
    "negative scenarios, and boundary conditions described in the acceptance criteria. "
    "Each test case should have a clear ID, title, preconditions, steps, and expected result."
)

This works fine for what it’s measuring — it gave DeepEval enough to assess relevancy and structure against a consistent standard. But it’s a simplification, not the rigorous version. A rubric tells the judge what shape a good answer takes. A golden dataset tells it what a good answer actually is for this specific AC.

The difference matters more than it sounds. Rubric-based scoring is fast and consistent, but it can’t catch certain failure modes — like an LLM producing well-structured, on-topic-sounding test cases that still miss a specific business rule a real QA engineer would have caught. A golden answer, written by someone who actually knows the domain, would catch that. A generic rubric won’t.

For teams adopting this kind of evaluation seriously, the honest advice is: start with rubric-based scoring to get moving, but treat building a small golden dataset — even just for your highest-risk ACs — as the next maturity step, not an optional extra.

Why this matters for QA teams right now

The teams building AI-assisted testing pipelines need someone who can close the loop — not just generate outputs, but validate them. That’s not a new skill for QA engineers. It’s the skill we’ve always had, applied to a new layer of the stack.

Building llm-eval-lab wasn’t about finding the best LLM. It was about building the muscle for this kind of work — and making the methodology visible enough that other QA engineers can use it, adapt it, and improve it.

The repo is at github.com/vivekbwaj/llm-eval-lab. It’s documented, runnable, and open.

If you’re a QA engineer starting to think about how to evaluate AI outputs — not just generate them — this is a reasonable place to start.