A Good Loss Curve Doesn’t Mean a Good Model

I’ve shipped a model with a great training loss curve that turned out to be a liar.

It happened on a doorbell camera detection project. The model trained well, loss dropped exactly the way you want, validation metrics during training looked solid. Then it hit real footage from a slightly different camera angle, different lighting, a different install height than what it had trained on, and performance fell off a cliff. The model hadn’t learned the objects. It had learned the training distribution.

That’s domain shift, and it’s invisible from inside a training run. Your loss curve only ever tells you how well the model fits the data it already saw. It says nothing about the data it hasn’t.

The gap nobody budgets time for

Here’s the workflow most people actually follow: train the model, watch the loss go down, maybe eyeball a few predictions, and if it looks reasonable, ship it or move to the next experiment.

That “eyeball a few predictions” step is where things quietly go wrong. I’ve written my own evaluation scripts with IoU-based matching just to get real per-class precision and recall on a held-out set — because without that, “the model looks pretty good” is a feeling, not a number. And feelings don’t catch the class that’s underperforming, or the object size range the model consistently misses, until it’s already in production.

The problem isn’t that people don’t care about validation. It’s that setting up a proper evaluation pass which is separate from training, on data the model has genuinely never seen, with real metrics broken out by class takes real effort. So it gets skipped, or done halfway, and the model ships on vibes.

What should happen instead

Pick the model you just trained. Upload a validation set it has never touched. Run inference. Get the actual answer.

That’s the workflow we built into Alpeye:

• Select any trained model from your past runs, not just your most recent one, any of them, so you can compare candidates properly instead of trusting whichever one finished last.
• Upload a validation dataset that’s genuinely separate from training data, the way it should be.
• Run inference directly, no separate script, no re-wiring your evaluation code for a different checkpoint format.
• Get real metrics back — precision, recall, per-class breakdowns. The numbers that tell you whether the model generalizes, not just whether it memorized.

Why this matters more than another training run

When something underperforms, the instinct is always to go back and train more, or train differently. But if you don’t have a clean, repeatable way to test a model against fresh data, you’re not actually debugging. You’re guessing whether the last change helped.

A dedicated inference step against a real validation set turns “I think this version is better” into “here’s the number that proves it.” That’s the difference between iterating on a model and just running the training loop again and hoping.

The best training run in the world means nothing if nobody checks what happens when the model meets data it’s never seen. This closes that gap.

Check it out here: alpeye.com