Why Synthetic Data Helps Train Better Classifiers

When real data is noisy, shifted, or otherwise not AI-ready, synthetic data used as auxiliary tends to train a better classifier than the real data alone.

The observation I keep running into

Most real-world data isn’t AI-ready in the way training assumes. There’s noise, distribution shift, hard samples that don’t fit a clean boundary. Training a classifier on data like that, alone, tends to give a model that’s fragile in exactly the places that matter.

What I’ve been seeing in several works is that adding synthetic data — generated by a model trained on the same real data — alongside the real data improves the classifier. Not as a replacement. As an auxiliary. Papers on ImageNet (Azizi et al., 2023), on distribution-shift benchmarks (He et al., ICLR 2023), on fine-grained classification (Trabucco et al., ICLR 2024) have all shown variants of this.

The interesting part is that the effect isn’t uniform. It shows up cleanly in some settings and barely at all in others. Trying to make sense of that has been clarifying.

Why training on raw data alone tends to be fragile

A classifier’s job is to predict the right label. The training objective doesn’t care how it predicts — only that the prediction is right. So the model latches onto whichever cue is easiest to use.

That’s where shortcut learning comes in. A classifier trained on a dataset where every cow happens to be standing on grass tends to learn “grass means cow.” It’s not learning the shape of the cow. It’s the spurious correlation. Show the model a cow on a beach and it misses.

On a clean validation set drawn from the same distribution as training, the cracks don’t show. The grass cue holds. The cracks show up when the test distribution drifts — noisy images, unusual backgrounds, hard samples that don’t fit the training prior.

What a generative model sees differently

Classifiers and generators are trained on the same data but with different objectives. A classifier minimizes prediction error on labels. A generator learns the data distribution itself — it has to produce plausible samples, not just discriminate between them.

That difference in objective ends up shaping the representation each model learns. A generator can’t get away with shortcuts the same way a classifier can. If it leaned only on “grass” to represent cows, the samples it produced would all be grass-with-cow blobs, and the distribution wouldn’t match. It needs broader structure to produce convincing data.

My hunch is that this is why synthetic data from a generator can shift a classifier’s learning bias. When the auxiliary data carries more of the underlying structure of the manifold, the classifier picks up something closer to robust features rather than spurious ones. The cleanest demonstration of this idea I know is the Stylized-ImageNet experiment (Geirhos et al., ICLR 2019) — replacing texture with random style nudges the classifier from texture-based shortcuts toward shape-based features, with large gains on shifted test sets.

It’s also probably why, from the results I’ve seen, the effect tends to be much stronger with diffusion-based generators than with GAN-based ones. Diffusion models seem to model the distribution more faithfully, so the synthetic data they produce carries more of that broader structure.

Where the auxiliary boost actually shows up

On a clean in-distribution test, the boost barely shows. A few percentage points at most, sometimes nothing. The classifier doesn’t really need help when the test set looks just like training.

The boost gets visible when the test data drifts. Noisy versions of the original images, unusual viewpoints, fine-grained categories where each class is sparse, low-data settings where the classifier has little to work with. In these settings, adding synthetic data alongside the real data tends to give a clear improvement.

The way I read it: the synthetic data is giving the classifier a different angle on the same problem. It pulls the model away from shortcut features and toward the more stable signals that survive distribution shift. A form of augmentation, with a regularization effect that’s hard to get from the raw data alone.

The honest caveat

Two things worth saying out loud.

First — synthetic data isn’t new information. The generator was trained on the same real data, so it can’t add information that wasn’t already there. Recent work on model collapse (Shumailov et al., Nature 2024) — training models recursively on generated data and watching the distribution degrade — is a reminder of this. The synthetic data’s value isn’t in adding signal. It’s in changing what the classifier pays attention to.

Second — and this is the catch behind the result I described above. Most of the strong distribution-shift improvements in recent papers come from generators that were pretrained on external data (Stable Diffusion, Imagen) — they’re bringing in outside knowledge through the back door. A small generator trained only on the target dataset probably won’t give the same effect. The choice of generator matters a lot.

Closing

The way I think about synthetic data now: not a substitute for real data, but a way to lend the classifier a different angle on what it’s looking at. The value shows up most clearly when the real data is noisy or shifted — which, in practice, is most of the time.

It’s a smaller claim than “synthetic data fixes everything.” But it’s the one I trust.

Further reading

• Azizi et al., “Synthetic Data from Diffusion Models Improves ImageNet Classification” (2023)
• He et al., “Is Synthetic Data from Generative Models Ready for Image Recognition?” (ICLR 2023)
• Trabucco et al., “Effective Data Augmentation With Diffusion Models (ICLR 2024)
• Tian et al., “StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners” (NeurIPS 2023)
• Geirhos et al., “ImageNet-trained CNNs are biased towards texture” (ICLR 2019)
• Geirhos et al., “Shortcut Learning in Deep Neural Networks” (Nature MI 2020)
• Shumailov et al., “The Curse of Recursion: Training on Generated Data Makes Models Forget” (Nature 2024)
• Shumailov et al., “The Curse of Recursion: Training on Generated Data Makes Models Forget” (Nature 2024)