I Ran the Same 50 Questions Through 3 AI Models. Here’s What Actually Happened.

A simple look at what “benchmarking” means, and why the winner on paper isn’t always the model you should use.

First, what even is a benchmark?

Before getting into results, it’s worth explaining what a “benchmark” actually is, because the word gets thrown around a lot.

A benchmark is just a fixed test. You take a set of questions with known correct answers, you ask each AI model the exact same questions in the exact same way, and you count how many each one gets right. That’s it. No trick, no magic. The point is to remove all the guessing and give a fair, apples to apples comparison.

For this test, I used something called GPQA Diamond. It’s a set of very hard, graduate level science questions covering biology, chemistry, and physics. These aren’t trivia questions. They’re the kind of questions where you’d expect a PhD student to struggle, which makes them a good stress test for how well an AI model actually reasons, instead of just recalling facts.

Here’s how I set it up:

I picked 50 questions from the same fixed set for all three models. The order of the multiple choice answers was shuffled the same way for every model, so no model could get lucky by guessing “the answer is usually C.”

I gave all three models the exact same instructions: answer with just a letter, don’t explain yourself. Then I recorded three things for every single answer: whether it was right, how long it took, and how many words (tokens) the model used to get there.

The three models I tested were Claude Fable 5, GPT-5.5, and Gemini 2.5 Flash. And running the whole thing, all 150 questions across all three models, cost me about $8 in API fees. That’s the real, all in cost of getting a comparison like this.

(so, I am 8$ in debt now)

The scoreboard

Here’s how each model did overall:

GPT-5.5 got 42 out of 50 questions right, which is 84%. Claude Fable 5 got 40 out of 50 right, which is 80%. Gemini 2.5 Flash got 27 out of 50 right, which is 54%.

If you only look at this number, the story seems simple: GPT-5.5 wins. But that’s where most benchmark writeups stop, and it’s also where they miss the interesting part.

Benchmark results on a common 50-question GPQA Diamond subset comparing accuracy, latency, and inference efficiency across Claude Fable 5, GPT-5.5, and Gemini 2.5 Flash.

The detail that changes everything

Here’s the thing that number doesn’t tell you. Claude Fable 5 scored a flat zero on every single biology question, and it’s worth being precise about why.

On this benchmark, Fable 5 did not produce an answer for any of the 12 biology questions. Every single biology request came back with finish_reason="content_filter" and no generated content at all (raw_text = null). Since there was nothing to grade, the evaluation pipeline scored all 12 as incorrect.

GPT-5.5 and Gemini 2.5 Flash, asked the exact same 12 questions, both produced actual responses.

That’s a very different kind of failure than just being wrong.

A model that gets blocked before it can answer isn’t confused about the science, it never got the chance to reason about it at all. So Fable’s biology score here mostly reflects serving-time refusal behavior, not its actual reasoning performance on those questions.

If you take biology out of the picture and only look at chemistry and physics, Fable 5’s accuracy jumps to 89%, which is actually a bit higher than GPT-5.5’s 85% on those same two subjects.

Here’s the breakdown by subject, in plain terms:

Biology (12 questions): Fable 5 was blocked on all 12, none returned an answer to score. GPT-5.5 got 80% right. Gemini got a perfect 100%.

Chemistry (25 questions): Fable 5 got 88% right. GPT-5.5 got 80% right. Gemini struggled here, only getting 40% right.

Physics (20 questions): Fable 5 got 90% right. GPT-5.5 also got 90% right. Gemini got 60% right.

So the overall score hides two very different stories.

Fable 5 is strong everywhere except one subject it wouldn’t touch at all. Gemini is the opposite of Fable in biology (perfect) but has a real weak spot in chemistry, where it tends to write long, winding explanations that talk themselves out of the correct answer.

Does thinking longer make a model more likely to be right?

You’d assume a model that “thinks” longer, meaning it writes more words or takes more time before answering, would be more likely to get things right. I checked this directly, and the answer is mostly no.

For Fable 5 and GPT-5.5, there was only a very weak connection between writing more and being correct, basically not enough to matter.

For Gemini, the connection actually went the other way. Its longest, most detailed answers were more likely to be wrong, not right. It had enough wrong answers in the sample (14 of them) that this pattern is real, not just noise.

What this means in plain terms:

don’t assume a longer, more detailed answer from an AI model means it’s more confident or more likely correct.

On this test, that assumption would have actively misled you, especially with Gemini.

Speed and cost tell two different stories

This is where things get genuinely useful if you’re trying to decide which model to actually use.

Claude Fable 5 was by far the fastest at getting to a correct answer, taking about 14 seconds on average per correct answer. That’s roughly 3.7 times faster than GPT-5.5, which took about 52 seconds per correct answer.

But Fable 5 wasn’t the cheapest. Gemini 2.5 Flash has a much lower price per word, so even though it was the slowest model and got the fewest questions right, it ended up being the cheapest option per correct answer. If you don’t mind waiting over a minute for an answer and cost is your main concern, Gemini comes out ahead.

GPT-5.5 didn’t win on speed or on cost. Its entire advantage is that it simply gets more answers right. That’s a completely valid thing to prioritize, it’s just a different priority than speed or cost.

So really there isn’t one “most efficient” model. There’s a fastest one, a cheapest one, and a most accurate one, and they’re three different models.

How each model gets things wrong

It’s also worth looking at the type of mistakes each model made, because they’re pretty different from each other.

Claude Fable 5 mostly avoids being wrong by refusing to engage with biology at all. Outside of that, its mistakes look like normal reasoning slips. On one chemistry question about a reaction between two compounds, it picked the wrong version of the product because it applied the wrong general rule for that type of reaction. A simple, understandable error.

GPT-5.5 tends to fail fast rather than fail after a long chain of reasoning. On one question about detecting hydrogen clouds around distant quasars, it gave a quick, confident, but wrong answer without much visible reasoning behind it. It’s a fast, wrong guess rather than a reasoning error.

Gemini 2.5 Flash tends to overthink itself into the wrong answer. On one chemistry question, it wrote almost 2,000 words working through the problem, correctly figured out part of the puzzle, considered four possible answers, and then talked itself out of the right one and picked a plausible sounding wrong one instead.

So which model should you actually use?

Honestly, that’s the wrong question to ask, and this whole test is a good example of why.

If you need the most correct answers and don’t mind waiting or paying more, GPT-5.5 is the strongest choice.

If you need answers fast and can work around it not answering biology questions, Fable 5 is the strongest choice.

If you need the cheapest possible option, don’t mind a longer wait, and your questions aren’t heavy on chemistry, Gemini 2.5 Flash is the strongest choice.

None of them is simply “the best model.” They’re three good answers to three different questions. The real lesson from running a test like this isn’t the number at the top of the leaderboard. It’s in the details underneath it: which topics get refused, which mistakes are one off slips versus a repeated pattern, and what “efficient” actually means for what you’re building.

Note on method: all three models were tested on the same fixed set of 50 questions, with the same shuffled answer choices and the same short answer instructions. Total cost to run this comparison across all three models: about $8 (my pocket is empty now T-T)

So the overall score hides two very different stories.

If you’re interested in how production AI systems are evaluated beyond benchmark accuracy, this LLM evaluation platforms benchmark by Noveum explores why metrics like faithfulness, tool use, and response quality matter alongside raw accuracy.