When Real Data Is Rare, Dangerous, or Illegal to Use — Synthetic Data Is Not a Compromise. It’s the Solution.

How the AI industry learned to train on data it couldn’t collect — and why synthetic data is now central to some of the most ambitious deployments on earth.

Training an autonomous vehicle AI on real-world collision data is, you’ll note, a problem that solves itself in a deeply unsatisfying way.

To train a system to avoid collisions, you need examples of collisions. To get examples of real collisions, you need to observe real collisions. Real collisions harm people. Deliberately causing them is illegal, unethical, and impractical at the scale needed to train a robust model. And waiting for them to happen naturally produces too few examples of the specific edge cases — the child running suddenly into the road, the truck jackknifing on a wet motorway — that represent the highest-stakes failure modes.

This is the fundamental problem that synthetic data solves. When the data you need is rare, dangerous, ethically impermissible, or simply doesn’t exist in sufficient volume, you generate it. And the ability to generate it has improved so dramatically in the last five years that synthetic data is no longer a second-best alternative to real data. For a growing number of AI applications, it is the primary training data source.

What synthetic data actually is — and the common misconception

Synthetic data is artificially generated data that mirrors the statistical properties and patterns of real data, without containing actual records of real events or real individuals. The critical phrase is ‘mirrors the statistical properties’ — synthetic data is not random noise, and it is not fabricated content. It is mathematically generated data that should be statistically indistinguishable from real data in the properties that matter for model training.

The common misconception is that synthetic data is inherently less reliable than real data. This is not universally true and is often wrong. Synthetic data generated by well-designed generative models, calibrated to match the real data distribution and validated against real-world benchmarks, can produce models that are as accurate — and sometimes more accurate — than models trained on real data. The reason the ‘more accurate’ case arises is that synthetic data can be generated to perfectly balance class distributions, include sufficient examples of rare edge cases, and exclude the noise and errors that are endemic in real-world data collection.

Five situations where synthetic data is not a compromise — it’s the only path

Autonomous systems. Waymo, Tesla, Cruise, and every serious autonomous vehicle company uses simulation environments — CARLA, NVIDIA DRIVE Sim, Waymo’s own internal simulation — to generate billions of miles of synthetic driving experience. The real-world miles their vehicles accumulate are dwarfed by the synthetic miles, and it is in the synthetic environment that the rare edge cases — the scenarios that would take years to encounter at real-world driving frequencies — are trained on intensively.

Healthcare and medical imaging. Generating a training dataset of 50,000 labelled pathology slides showing rare cancers requires patients who have those cancers, pathologists to label the slides, and regulatory compliance for every step. Generating synthetic medical images — using generative adversarial networks calibrated to match the statistical properties of real pathology data — can produce similar quantities of training data in a fraction of the time, without any patient privacy implications. The synthetic images are not photographs of real patients. They are statistically realistic representations of what real pathology looks like.

Fraud detection under privacy constraints. A bank that wants to train a fraud detection model needs examples of fraudulent transactions. Sharing actual fraud data between organisations — which would allow industry-wide model training — creates regulatory and competitive concerns. Synthetic fraud scenarios, generated to match the statistical characteristics of real fraud without containing actual customer data, can be shared freely and used to train models that benefit from the pattern knowledge embedded in the real data without the privacy exposure.

Rare event modelling. The value of AI-based early warning systems — for equipment failure, medical deterioration, structural failure — depends on having sufficient examples of the failure events the system is trying to predict. By definition, rare events are rare, and historical datasets may contain too few examples for reliable model training. Synthetic data generation can amplify the rare class — creating additional synthetic examples of failure events with realistic characteristics — to provide the training signal the model needs.

Digital twins. Industrial organisations including Siemens, Honeywell, and GE Digital operate digital twin environments — high-fidelity simulations of physical assets — that can generate synthetic operational data at will. Training an AI system on synthetic data from the digital twin, then validating it against real sensor data from the physical asset, allows the AI to learn complex operational patterns without instrumenting the physical asset for the years it would take to accumulate equivalent real-world data.

The most sophisticated AI systems in the world — autonomous vehicles, protein structure prediction, physics simulation — are trained primarily on synthetic data. Real data validates. Synthetic data educates.

Data augmentation: the lightweight version of the same idea

Data augmentation is synthetic data’s smaller, faster sibling. Where full synthetic data generation creates entirely new examples, data augmentation creates variations of existing examples to increase dataset size and diversity.

In image recognition, augmentation typically includes rotation, scaling, flipping, brightness adjustment, colour shift, and cropping. The original image of a cat is used to generate ten additional images — the same cat, slightly rotated, slightly darker, slightly cropped — each of which is a distinct training example. This is the reason that image classification models trained on modest datasets can achieve strong generalisation: the effective training dataset is ten to twenty times larger than the original through augmentation alone.

In text-based models, augmentation includes synonym substitution, sentence paraphrasing, back-translation (translating to another language and back), and controlled noise injection. A training example like ‘the product was delivered late and the packaging was damaged’ generates multiple augmented versions with the same sentiment label, teaching the model that the sentiment of disappointment is expressed in many different ways.

Large language models are now being used to generate augmented training data for smaller, specialised models — an approach sometimes called data distillation. A GPT-4 class model is prompted to generate thousands of examples of a specific type of customer interaction, question-answer pair, or document summary, which are then used to fine-tune a smaller, domain-specific model. The large model’s broad language capability is transferred to the small model’s narrow domain via generated data. This is one of the most productive current applications of generative AI in enterprise AI development.

The critical validation requirement — and where teams fail

Synthetic data’s power comes with one non-negotiable condition: it must be validated against real data before the models trained on it are trusted for real decisions.

The validation process compares the statistical properties of the synthetic dataset with the real dataset — distributions, correlations, rare event frequencies, class balances. It uses real data to benchmark the performance of models trained on synthetic data. And it explicitly tests whether the model trained on synthetic data generalises correctly to real-world inputs.

The failure mode is using synthetic data as a way to avoid the difficult work of real data collection and validation, rather than as a complement to it. A model trained on synthetic medical images that has never been validated against real pathology will fail in ways that are difficult to detect without that validation — because the simulation will be internally consistent but miss the specific characteristics of real-world data that the simulation doesn’t capture.

Human supervision of synthetic data generation is essential precisely because generative models can produce data that looks realistic but encodes systematic biases or statistical anomalies that only a domain expert would recognise as wrong. The AI that generates the synthetic training data needs a human who understands the domain to review what it produces.

Synthetic data doesn’t solve the data problem. It reshapes it — from a problem of collection and labelling to a problem of generation and validation. Both are hard. But the second is, for many of the most important AI applications, significantly more tractable.