Your Loss Function Quietly Decides Which Mistakes to Make. Squared Error Loves the Average. Absolute Error Loves the Middle. Here Is the Proof, By Hand.

Five numbers, and you must summarize them with a single guess. Which guess is best? The startling answer: it depends entirely on how you decide to punish being wrong.

Punish wrongness one way and the best guess is the average. Punish it another way and the best guess is the middle value, which can be wildly different.

Same data, two “best” answers, chosen not by the data but by your private rule for what counts as a bad miss. We will prove both, with small numbers, by hand.

This is one of the most important and least-discussed facts in all of machine learning, because the rule for punishing wrongness is the loss function, and the loss function is a choice — often a default nobody examined. That quiet choice decides what your model aims at. Let us make it loud.

Five numbers and a single guess

Here are five numbers. They could be five house prices, five wait times, five anything.

2,   4,   6,   8,   100

Notice the 100. Four of the numbers huddle between 2 and 8, and one is way out at 100 — an outlier, a mansion among cottages, a freak delay among normal ones.

Your job: pick one single number to represent these five. A summary. A prediction. Call your guess g.

The five numbers will each differ from your guess by some amount, and you want those differences to be small. But “small” needs a definition, and here is where the fork in the road appears. There are two natural ways to measure how badly your guess misses, and they lead to two different best guesses.

Rule one: punish the square of each miss

The first rule says: for each number, take how far your guess is from it, square that distance, and add up the squares.

Your guess is best when this total of squared misses is smallest. This is squared-error loss, the most common loss in all of machine learning.

Why square? Squaring does something specific: it punishes big misses far more than small ones.

A miss of 2 costs 4. A miss of 10 costs 100. A miss of 50 costs 2500. Under squaring, a single large miss is catastrophic — it dominates the total. Squared error hates big misses with a passion.

Let us find the guess that minimizes the total squared miss, and I will show you it is the average.

First, what is the average of our five numbers?

Add them: 2 plus 4 is 6, plus 6 is 12, plus 8 is 20, plus 100 is 120. Divide by 5: 120 divided by 5 is 24. The average is 24.

Let us check that 24 beats its neighbors on total squared miss.

We do not need to test every number — a couple of points around 24 will show it sits at the bottom of the valley.

Try guess equals 24:

• miss from 2 is 22, squared is 484
• miss from 4 is 20, squared is 400
• miss from 6 is 18, squared is 324
• miss from 8 is 16, squared is 256
• miss from 100 is 76, squared is 5776
• total: 484 plus 400 plus 324 plus 256 plus 5776 equals 7240

Now try guess equals 25, one step up:

• miss from 2 is 23, squared is 529
• miss from 4 is 21, squared is 441
• miss from 6 is 19, squared is 361
• miss from 8 is 17, squared is 289
• miss from 100 is 75, squared is 5625
• total: 529 plus 441 plus 361 plus 289 plus 5625 equals 7245

And guess equals 23, one step down:

• miss from 2 is 21, squared is 441
• miss from 4 is 19, squared is 361
• miss from 6 is 17, squared is 289
• miss from 8 is 15, squared is 225
• miss from 100 is 77, squared is 5929
• total: 441 plus 361 plus 289 plus 225 plus 5929 equals 7245

Look at that. At 24, the total squared miss is 7240. Step up to 25, it rises to 7245. Step down to 23, it rises to 7245.

The guess of 24 — the average — sits at the very bottom. Move either way and you do worse.

The average is the champion of squared-error loss. (This is a general truth, provable for any set of numbers, not just ours: the average always minimizes total squared miss.)

But notice where 24 falls relative to the data. Four of our five numbers are 8 or below. The “best” guess of 24 is larger than four of the five numbers it is summarizing.

It got yanked way up toward 100 by that single outlier — because squaring made the outlier’s pull enormous. Squared error let one freak value drag the answer far from where most of the data actually lives. Remember that.

Rule two: punish the plain size of each miss

The second rule says: for each number, take how far your guess is from it, but do not square it — just take the plain distance.

Add up those plain distances. Your guess is best when this total is smallest. This is absolute-error loss.

Without squaring, a big miss is bad but not catastrophic.

A miss of 10 costs 10, not 100.

A miss of 50 costs 50, not 2500.

Outliers still cost something, but they no longer dominate. Absolute error treats a big miss as just a big miss, in proportion.

Let us find the guess that minimizes the total plain distance, and I will show you it is the middle value — the median.

The median is the number in the middle when you line them up in order.

Our numbers in order are 2, 4, 6, 8, 100. The middle one is 6. So the median is 6 — utterly different from the average of 24.

Let us check that 6 minimizes the total plain distance.

Try guess equals 6:

• distance from 2 is 4
• distance from 4 is 2
• distance from 6 is 0
• distance from 8 is 2
• distance from 100 is 94
• total: 4 plus 2 plus 0 plus 2 plus 94 equals 102

Now try guess equals 7, one step up:

• distance from 2 is 5
• distance from 4 is 3
• distance from 6 is 1
• distance from 8 is 1
• distance from 100 is 93
• total: 5 plus 3 plus 1 plus 1 plus 93 equals 103

And guess equals 5, one step down:

• distance from 2 is 3
• distance from 4 is 1
• distance from 6 is 1
• distance from 8 is 3
• distance from 100 is 95
• total: 3 plus 1 plus 1 plus 3 plus 95 equals 103

At 6, the total plain distance is 102. Step up to 7, it rises to 103. Step down to 5, it rises to 103.

The guess of 6 — the median — sits at the bottom. The median is the champion of absolute-error loss.

And here is the beautiful intuition for why.

Under plain distance, moving your guess up by one step changes your total by a simple tally: every number above your guess gets one step closer (saving one each), and every number below gets one step farther (costing one each).

So moving up is worth it only while more numbers sit above you than below. You keep moving until exactly half are above and half below — the middle.

At the median, the numbers above and below are balanced, so no move helps. The outlier of 100 is just one number above; it pulls your guess upward by exactly one vote, the same as any other number above.

Its hugeness does not matter — only its side does. That is why the median barely flinched at the 100, while the average lunged toward it.

The two answers, side by side

Same five numbers. Two loss functions. Two completely different “best” guesses.

loss function    | best guess | what it is | where it sits
-----------------|------------|------------|----------------------------
squared error    |     24     |  the mean  | dragged up by the outlier
absolute error   |      6     | the median | sits among the real data

Twenty-four versus six. That is not a small disagreement — it is a four-fold difference, and it came entirely from the choice of how to punish a miss. The data never told us the answer.

The data is the same. Our private rule about whether big misses are catastrophic (square them) or merely proportional (do not) is what picked 24 over 6, or 6 over 24.

Why this is one of the most important choices you never made

In machine learning, the loss function is the target. Training does one thing: it adjusts the model to make the loss small. So whatever the loss rewards is what the model becomes. And the two losses reward opposite instincts.

A model trained on squared error aims for the mean, and it will twist itself to avoid large misses, because large misses are squared into catastrophe.

This makes it sensitive to outliers — a few extreme data points can drag the whole model toward them, exactly as the 100 dragged our guess to 24. If your data has freak values, or noisy labels, or rare extremes, squared error chases them.

A model trained on absolute error aims for the median, and it shrugs off outliers, because a big miss is just a big miss, not a catastrophe.

This makes it robust — a few extreme points barely move it, exactly as the 100 barely moved our guess from 6. If you want a model that ignores freaks and tracks the bulk of the data, absolute error does that.

Neither is right in the abstract. They answer different questions. Squared error answers “what value balances the total squared deviation,” which is the right question when large errors really are disproportionately bad and the data is clean.

Absolute error answers “what value has half the data on each side,” which is the right question when you want a typical case and you do not want a few extremes hijacking the answer.

The danger is using squared error by default — because it is the default — on data full of outliers, and then being puzzled that your model’s predictions sit above almost all the real values, dragged skyward by a handful of freaks you never decided to chase.

The loss function is not a technical detail you set and forget. It is a values statement. It declares what you consider a bad mistake. And that declaration, not the data, decides what your model reaches for.

What stays with me

What unsettles and delights me about this is that “best” turned out to have no meaning until we specified how we punish being wrong. We tend to imagine that data has a best summary sitting inside it, waiting to be found. It does not.

The five numbers 2, 4, 6, 8, 100 do not contain a single best representative. They contain a 24 if you square your misses and a 6 if you do not, and the choice between those lives in you, not in them.

The answer was downstream of a decision we almost made without noticing.

That generalizes far past five numbers. Every model that has ever been trained reached for whatever its loss function rewarded, and that loss was a choice — frequently an unexamined default, copied from a tutorial, never questioned.

Which means a great many models are quietly aiming at the mean, chasing outliers, sitting above most of their data, not because anyone decided that was right, but because squared error was the box that happened to be checked.

So here is the question to leave you with. The next time you train a model, or read a “best estimate,” or trust an average — someone, somewhere, chose how to punish being wrong, and that choice picked the answer.

So before you believe it: do you know which mistakes that loss function decided were unforgivable and which it decided to shrug off — and is that the choice you would have made, if anyone had told you that you were making one?

Source: the classic result that the mean minimizes squared error and the median minimizes absolute error; a foundation of robust statistics (Huber, 1964) and of loss-function choice in machine learning.