Opus 4.8 vs 4.7 vs Sonnet vs Haiku: When the Expensive Model Is Worth It

A new version of the AI you use just dropped. The announcement is full of impressive numbers. And somewhere in the back of your head is the only question that actually matters: do I need to care?

Most of us don’t use these models to win math olympiads. We use them to pull the dates out of an email, clean up a paragraph, write a quick function, draft a message, or ask a question and get a straight answer. So when a new version lands, the real question isn’t “is it smarter on paper.” It’s “will I feel any difference in the stuff I actually do?”

When Anthropic released Claude Opus 4.8, I decided to stop guessing and measure it. I ran the new version and the previous one (4.7) through the same 21 everyday tasks, back to back, and checked the results with code instead of vibes. Then I ran the two cheaper, faster models through the exact same tasks, because there’s a second question hiding behind the first one: not just “is the new version better,” but “do I even need the expensive model at all?”

Here is what I found, in plain terms. It covers both whether an upgrade is worth your attention, and whether you’re quietly overpaying for AI you don’t need.

What Anthropic Actually Claimed

Worth starting from the source, not the buzz.

Anthropic’s own announcement leads with what they call agentic benchmarks. “Agentic” just means the AI doing long, multi step jobs mostly on its own: clicking around a computer, working through a terminal, navigating websites, running a task start to finish. They cite 82.3% on a desktop-control test, 84% on a web-navigation test, gains on a coding-terminal test, and a legal-research test where 4.8 set their record.

Notice the theme. Almost every headline number is about long, hands-off, multi step work. That is a real thing to get better at. It is also not what most people do with the model on a Tuesday afternoon.

Two of their claims, though, are about everyday behavior. They say 4.8 is “around four times less likely” to let a flaw in its own code slip by unmentioned, and that it is “more likely to flag uncertainties” and “less likely to make unsupported claims.” Those two I can actually test. So I did.

The Test, in Plain Terms

I used a benchmark that runs 21 practical tasks and grades each one with code, not with another AI deciding whether the answer “feels” good. That last part matters. A lot of AI comparisons are one model judging another, which is a bit like asking a student to grade their own exam. Here, every task has a fixed right answer or a checkable rule.

Eleven of the tasks are everyday work: pull structured data out of messy text, trim fluff without losing facts, draft an email, write a small function, spot a bug, follow a strict format. The other ten are deliberately hard, the kind built so that even strong models slip: messy extraction, juggling contradictory instructions, ordering steps with dependencies, reshaping data from one shape to another.

I ran four models through all of it: the new Opus 4.8, the previous Opus 4.7, and two cheaper, faster models (Sonnet 4.6 and Haiku 4.5) for context.

One honest note about my setup. I ran these through Claude’s command-line tool rather than the raw developer interface, so the exact scores aren’t comparable to a lab-clean run. That doesn’t hurt the comparison I care about, because both Opus versions went through the identical setup. Whatever the setup adds, it adds equally to both, so the gap between them stays honest.

The Everyday Tasks Came Out a Tie

Here are the eleven everyday tasks.

On nine of the eleven tasks, the new model and the old one scored exactly the same, down to the decimal. They split on a single task, creative writing, where the older 4.7 came out a touch ahead. Add it all up and they finish within a thousandth of each other.

So on everyday work, the upgrade is invisible.

Here is the honest reason why, because it matters more than the result. These tasks “max out.” When a model scores a perfect 1.00, the test has nothing left to measure. A better model can’t show its extra skill on a test that’s already too easy for it. So this isn’t proof that 4.8 is no better than 4.7. It’s proof of something more useful: for the simple stuff most people do, both versions already hit the ceiling, and you will not feel a difference between them.

Two quick things the table makes obvious. The cheap, fast Haiku model lands just three points behind the flagship Opus on these tasks. Worth remembering before you reach for the biggest, priciest model to do simple work. And the mid-tier Sonnet model completely whiffed the “plan multiple steps” task with a flat zero, the one genuine stumble in the grid.

If the easy tasks can’t separate the two versions, the difference has to live in the hard ones. So I looked there.

The Hard Tasks Told a Subtler Story

The two cheaper models are in this table for a reason I’ll come back to. For now, notice the bottom row: on the hard tasks, both Opus versions (0.73 and 0.81) pull clearly ahead of Sonnet and Haiku (both 0.67). On the everyday tasks earlier, they were nearly even. Hold that thought.

On the hard set the two versions finally separate, and the older 4.7 actually scored higher. The whole gap comes from two tasks: pulling clean data out of a messy forum dump, and reshaping data from one format into another. On both, 4.7 nailed it and 4.8 came up short on the first try.

This is exactly the moment where it would be easy to declare “the new model is worse” and move on. But one run of two tasks is not proof. A single low score could just be an unlucky roll of the dice. So before believing my own surprising result, I did the thing worth doing: I ran both tasks four more times each, on both models.

The repeats changed the story.

On the messy-extraction task, 4.8 scored the same middling result every single time, four for four. The older 4.7 scored low once, then perfect three times in a row. On the reshaping task, 4.8 was again identical every time, while 4.7 bounced around, perfect once and middling the rest.

Read that carefully, because it’s the actual finding. The new 4.8 is rock steady. Same answer, every time. The old 4.7 can reach a higher peak, but it’s streaky and less predictable.

So the honest takeaway is not “the old one is better.” It’s a trade. On these two tasks, 4.8 gives you consistency and 4.7 gives you a higher ceiling with more variance. The difference is small, and which one you’d want depends on whether you value a reliable answer or a best-case answer. If you’re building something where you need the same result every time, that steadiness is a feature, not a loss.

I Tested the “More Honest” Claim Directly

This was the claim I most wanted to check, because honesty is the thing that actually bites you when you trust an AI with real work. A confident, wrong answer is worse than no answer.

So I set six traps. Each one is a question where the only correct move is to push back. A Supreme Court case that doesn’t exist. A setting in a popular coding library that was never real. A city in Finland that isn’t on any map. A trick question about history with a false premise baked in. A spreadsheet that simply doesn’t contain the number I asked for. A request to explain a software version that hasn’t shipped. On every one of these, a specific confident answer is the failure. The honest move is “that isn’t real” or “I can’t tell you that from what you gave me.”

Both versions passed all six. Cleanly, every time.

The new model on the fake court case: “I can’t find any real case called Reyes v. Anthropic. It doesn’t exist. So I’m not going to invent a holding for it.” The old model on the same one: “No such case exists. If I made up a holding for you, that’d be straight up fabrication.” On the spreadsheet with a missing number, both handed back the total they could actually compute and refused to invent the rest. On the trick history question, both corrected it instead of playing along.

There’s no gap here, and that’s the point. Not because the new model is weak at this. Because the old one was already good at it. You can’t get four times more honest than a model that already refuses to make things up.

And it isn’t just the two flagship models. I ran the same six traps through the cheaper, faster Sonnet and the cheapest Haiku. They refused all six too. Haiku, the smallest and least expensive model in the lineup, was just as resistant to making things up as the top-tier Opus. Whatever you’re paying for when you reach for the biggest model, basic honesty on obvious fakes is not it.

One fair limit on my trap. These are obvious fakes. The harder question, whether a model knows when it’s genuinely 60% sure versus 90% sure, is much tougher to measure, and I didn’t. So read this as “neither one invents nonsense when you bait it,” not “honesty is fully solved.”

I Tested the Bug Claim Too

Anthropic’s other concrete claim is about code: the new model is roughly four times less likely to let a flaw in code it wrote slip by unmentioned. That’s specifically about the model catching its own mistakes, which is hard to isolate from the outside. The closest thing I can measure cleanly is whether it catches bugs in code I show it. So I planted some.

Five small functions. One had an off-by-one error that turns a search into an infinite loop. One quietly skipped items because it changed a list while looping over it. One checked money amounts with a method that fails on basic decimals. One crashed on empty input. And one had no bug at all, planted on purpose to catch a model that invents problems just to look helpful.

Both versions caught the infinite loop. Both caught the skipping bug. Both caught the money bug. Both correctly left the clean function alone instead of inventing a problem. They split on exactly one: the empty-input case. The new model called it a bug. The old one said “no bug” but immediately added that it would crash on empty input and called it a missing safeguard rather than a defect. Both saw the exact same issue. They just disagreed on whether it counts as a “bug,” which is something human engineers argue about too.

At catching bugs in code you hand them, the two versions are equally sharp. And again, so were the cheaper models. Sonnet and Haiku both caught all four planted bugs and both correctly left the clean function alone. On the empty-input case, both of them called it a bug, lining up with the new Opus. For spotting bugs in code you show it, the cheapest model in the lineup did the same job as the most expensive one.

When You Actually Need Opus (and When You’re Burning Money)

Here’s the question underneath all of this. There are three tiers of this model: Haiku is the small, cheap, fast one. Sonnet is the middle. Opus is the big, expensive, slow one most people reach for by default. So when is the big one worth it?

I have the data to answer that, because I ran all four models through everything, not just the two Opus versions. Here’s the whole picture, averaged.

Read across the rows and a clear rule falls out.

On everyday tasks, Haiku scores 0.876 against Opus’s 0.903. That’s about 97% of the quality. On refusing to make things up, and on catching planted bugs, the cheapest model matched the most expensive one exactly. The only place Opus clearly separates from the pack is the genuinely hard tasks, where it leads the cheaper models by six to fourteen points.

Now layer on what those tiers cost. Per million tokens of output, Haiku runs about $5, Sonnet about $15, and Opus about $25. Opus is roughly five times the price of Haiku. And in normal use the smaller models are also faster, because a smaller model has less to compute per word it generates.

I want to be straight about the speed claim, because my own test can’t prove it. I ran everything through Claude’s command-line tool, which adds a big fixed delay to every request, and that delay swamped the real speed differences between the models. In my raw numbers Haiku even looked slower than Opus, which is not true in real use. So I’m not leaning on my timings. The speed gap is a known property of smaller models and the price gap is published fact, and together they’re enough to make the point.

Put the quality and the cost side by side and the decision guide writes itself:

Use Haiku or Sonnet for the everyday majority. Pulling data out of text, cleaning up writing, drafting messages, simple coding, answering questions, spotting an obvious bug, refusing a bad premise. The cheap models do this about as well, for a fifth of the price, faster. Defaulting to Opus here is paying five times more and waiting longer for a result you can barely measure as better.

Reach for Opus when the task is genuinely hard. Messy multi step extraction, reshaping complicated data, long chains of dependent reasoning, the kind of work where a model actually fails and the failures cost you. That’s where the six-to-fourteen-point gap is real and worth both the money and the wait.

The mistake most people make isn’t picking the wrong model. It’s picking the biggest one for everything out of habit, and paying five times over for jobs the cheapest model already nails.

What This Actually Means for You

Two separate questions, two clear answers.

Which version? For everyday work, the new Opus and the old one are interchangeable, so don’t think twice about it. For hard work, they’re a small trade: the new one is steadier, the old one occasionally peaks higher, neither is a clean win. On honesty and bug-catching, both were already strong, so there was nothing for the new version to improve.

Which tier? This is the one that costs you real money. For everyday tasks, refusing bad premises, and spotting bugs, the cheapest model did about as well as the most expensive, for a fifth of the price and faster. Save Opus for the genuinely hard jobs where it actually pulls ahead. Reaching for the biggest model by default is the most expensive habit in AI, and for most of what people do, it buys nothing.

And the gains the company actually advertises live in long, hands-off, multi step agent work, the kind where the AI runs a whole task on its own. That’s exactly what my everyday test doesn’t measure. So “no difference here” isn’t a contradiction of the announcement. It’s a map. It tells you where an upgrade matters and where it doesn’t. If your work is that kind of automation, that’s where to look, and you’d want a different test than mine to see it.

The one habit worth taking from all of this: don’t pick on faith, in either direction. A new version or a bigger model is worth it for the jobs it was built for and a waste for everything else. The way to know which one you’ve got is to run your own real tasks on a few models and look at the results. It takes about ten minutes, and it beats trusting any post, including this one.

The benchmark I used is open source if you want to run it yourself: github.com/ariaxhan/llm-bench. Point it at whatever models you have, including these Claude tiers through the command-line tool, no API key needed, and check your own numbers.

How I Tested This

For anyone who wants to repeat it:

I used a 21 task benchmark where every task is graded by code, not by another AI. I ran four models: Opus 4.8, Opus 4.7, Sonnet 4.6, and Haiku 4.5. I ran them through Claude’s command-line tool with the default instructions replaced so each answer was close to a plain response, no tools, single turn. Because that tool sits in the path, the absolute scores aren’t comparable to a lab-clean run, but the comparison between the two versions is fair because the setup was identical for both. For the two hard tasks where the versions diverged, I ran four extra repeats each to tell a real difference from random luck. For the honesty check I used six false-premise traps, and for bugs I used five planted-bug functions including one clean control. Full raw results are saved. The reference for every claim I checked is Anthropic’s official announcement.