The Time ChatGPT Undercharged Me $5.50 — and What It Taught Me About How AI Thinks

On this past Thursday, my Chinese grocery sourcer messaged me with a confession. She’d been using ChatGPT as a quick calculator while tallying an order — several bulk items, quantities in kilograms, unit prices in yuan. She asked it to sum three numbers. It gave her a total. She sent the invoice.

WeChat conversation (with consent from the sender) with ChatGPT incorrect result part translated into English.

A few hours later, she realized something felt off. She checked the math herself. ChatGPT had shorted the total by $5.50. We laughed about it — she joked that at least the error was in our favor — and I told her, half-seriously, “Just use a calculator app. Or even Google search. Don’t make the AI do arithmetic in its head.”

But the moment stuck with me. Not because an LLM made a mistake — that’s old news. But because of what kind of mistake it was. It wasn’t a hallucinated fact or a logic gap. It was three multi-digit numbers, pure arithmetic, the kind of thing a pocket calculator from 1985 handles flawlessly. And yet, the same architecture that can now generate novel mathematical proofs tripped on a task my 8-year-old cousin could do.

It also got me thinking about the tool-calling landscape. When my sourcer typed numbers into ChatGPT, she was relying on the model’s native reasoning — its internal arithmetic circuits. But most leading models now support tool calling: they can recognize a math query and route it to an actual calculator function. The catch is, this only works if the model correctly identifies the intent. Here’s a rough snapshot of how the major players handle this nowadays:

The common thread: every one of these models will occasionally default to doing arithmetic “in their head” rather than reaching for a calculator. When they do, they expose the exact fragility this paper dissects. My sourcer didn’t know she needed to ask for tool use. She just typed numbers and hit enter.

I’ve been following the probing literature for a while, so I had a vague intuition: the model probably knew the right answer somewhere inside, but it didn’t make it to the output. Still, I wanted a real explanation. That’s when I found a paper called The Shape of Addition, with accompanying code on GitHub at RL-MIND/Shape-of-Addition. It gave me the most satisfying answer I’ve seen yet.

Here’s what I learned.

The Model Already Knows — It Just Doesn’t Say It

The paper’s core experiment is deceptively simple. They take Qwen3–4B and feed it 10,000 problems where it must add three 10-digit integers. At every step of generating the output digit by digit, they extract the hidden state vectors. Then they train small probing classifiers to ask those vectors different questions.

The experimental setup. Hidden states are extracted at every digit generation step, and probes are trained to decode multiple arithmetic signals from a single vector.

What they find is striking. A single hidden state vector simultaneously encodes:

• The ground truth digit for that position.
• The digit the model will actually output.
• Whether that output is correct or wrong.
• The raw sum of the input digits in that column.
• The discrete carry arriving from the right.
• A continuous “carry potential” (which I’ll come back to).

This is what the authors call probe versatility. And critically, even when the model outputs a wrong digit, a probe can often decode the correct digit from that same vector. The information didn’t disappear. It simply lost an internal election.

That was my first real answer for my sourcer’s ChatGPT incident the past Thursday. The model likely had $5.50 more sitting somewhere in its hidden state. It just didn’t surface it.

The Shape of Addition: Not a Lookup Table, but a Terrain

If the right answer is still encoded alongside the wrong one, what does the model’s internal decision space actually look like? The authors use UMAP to project the high-dimensional hidden states into something visualizable, and what emerges is far from random.

The macro structure. Hidden states cluster into ten distinct “digit basins” around the number anchors 0–9. If a state lands near the “6” basin, the model outputs 6.

The space organizes into what I’d describe as a topographic map:

• Digit basins. Points cluster into ten distinct gravitational wells, one for each output digit 0–9. If your hidden state lands near the “6” basin, you get a 6.
• Carry fibers. Within a single basin — say, all states that yield output “1” — the model further separates by provenance. A “1” that came from no carry sits in a different subregion than a “1” that came from a carry of 2. The model preserves history inside the number.

Zooming into a single digit basin. The model internally separates outputs by their carry origin — a “1” from no carry is geometrically distinct from a “1” from a carry of 2.

• Iso-Raw-Sum Trajectories (IRSTs). This is the paper’s central geometric concept. If the raw sum of a column is fixed — say, it’s 1 — then depending on the incoming carry, the correct output could be 1, 2, or 3. In discrete arithmetic, those are three separate answers. In the model’s internal geometry, they are three points along a single continuous arc that sweeps through all three digit basins.

Four examples of Iso-Raw-Sum Trajectories. Each colored line represents a fixed raw sum; points along the line shift output digits as the carry changes. The model walks a continuous curve and only discretizes at the very end.

What this means is the model doesn’t think in integers. It thinks in continuous curves and then discretizes at the very end. The arithmetic logic is a smooth path; the final spoken digit is just where that path gets snapped to a grid.

The Real Reason It Gets Addition Wrong: Slipping at the Boundary

This brings us to the question my sourcer actually cares about: why does the wrong number come out?

The paper’s answer is the Noisy Quantization Model, and it hinges on that continuous “carry potential” I mentioned earlier.

Carry potential is a real number that accumulates pressure from all columns to the right. It’s not an integer carry decision — it’s more like 0.87 or 1.12. To produce the final output, the model must quantize this continuous value into a discrete carry (0, 1, or 2).

Now imagine the quantization threshold is 1.0. If the carry potential is 1.47, a little noise in the network doesn’t matter — it still rounds cleanly to 1. But if it’s 0.99 or 1.01? A tiny wobble flips the rounding, and suddenly you’ve added or dropped a carry. The output digit shifts by one. The model walked the correct continuous trajectory but slipped right at the basin boundary.

The authors call this geometric slippage, and they back it up with a clear experiment: they plot error rate against carry potential and show that errors spike precisely where the value sits near an integer boundary. The model isn’t confused. It’s just fragile at the rounding step.

That is exactly what happened on Thursday with the $5.50. Somewhere in the chain of digit generation, a carry potential hovered near 0.5 or 1.5 or 2.5, noise nudged it the wrong way, and the output slipped into the neighboring digit basin.

They Even Built a Fix — Without Retraining

Here’s where the paper goes from diagnostic to constructive. If the hidden state already contains the raw sum and the carry potential as separate, decodable signals, can you use them to catch and correct errors on the fly?

Their proposed method is dual-stream consistency checking. At inference time, for each position, they decode both the local raw sum and the global carry potential from the final hidden state. Then they check: does the model’s output digit align arithmetically with these two internal signals? If yes, keep it. If no, recompute the digit from the decoded raw sum and the quantized carry potential.

The result outperforms the raw output and several baselines. The code for this — probes, consistency checks, visualizations — is openly available at the RL-MIND/Shape-of-Addition repo. What I appreciate is that the method isn’t really about boosting a benchmark. It’s an existence proof: the correct arithmetic is recoverable from states that produced wrong outputs. The information was there.

What This Reframes

Before reading this paper, I thought of LLM arithmetic errors the way most people probably do: as a sign the model just doesn’t know how to add properly. Maybe it needs more training data, or bigger parameters, or that “chain-of-thought” thing people talk about.

But the probing literature has been quietly telling a different story for a few years now. Researchers have shown that lightweight classifiers — tiny detective algorithms trained on hidden states — can decode the correct arithmetic answer from inside a model even when the model itself outputs the wrong one. The signal is there. It just doesn’t win the internal election that decides what token comes out. So the hunch wasn’t just a hunch. It was a pattern accumulating across papers: models encode more than they verbalize. What was missing was a clear explanation of why — a mechanism.

This paper provides that mechanism. The errors aren’t ignorance. They’re a discretization failure. The model builds this beautiful internal landscape — continuous curves that trace the exact right arithmetic path through different number regions. It knows where it’s going. Then, right at the last step, it has to collapse all that richness into a single digit token. When the internal signal lands too close to a rounding boundary — think 0.99 or 1.01 instead of a clean 1.5 — the tiniest wobble can push it off by one.

The probing literature’s pattern now has a geometric explanation. The model does know. The paper just shows exactly where the handoff breaks: not in the thinking, but in the final act of turning continuous thought into a discrete spoken number.

So when my sourcer asks why an AI that can win IMO gold medals and work out novel postdoctoral-level math proofs still can’t add three prices without dropping $5.50, the answer isn’t “it’s bad at math.” The math it did internally was correct. It just tripped at the finish line. The same system that holds a flawless continuous trajectory through abstract reasoning still has to round that trajectory into a single integer token at the very end — and sometimes, at the boundary, it slips.

And honestly, that feels strangely human.

Paper: The Shape of Addition (arXiv:2606.03645) Code: github.com/RL-MIND/Shape-of-Addition