AI in 1 minute

When AI Learns the Wrong Lessons

OpenAI’s ChatGPT and Sora reportedly show caste bias in India

LLM showing Brahmin and Dalit perspectives jobs: Image Source: MIT Tech Review

A new MIT Technology Review investigation found something troubling: OpenAI’s ChatGPT and Sora show signs of caste bias.

The timing is awkward. At the GPT-5 launch in August, Sam Altman called India OpenAI’s second-largest market. So the country helping drive AI adoption may also be the one most hurt by its hidden prejudices.

The Problem with Learning from Us

Large language models train on vast amounts of internet content. That content reflects human history, including our biases.

When caste stereotypes show up in AI responses, the system isn’t being malicious. It’s being reflective. It learned from what we wrote and published.

But AI doesn’t just reflect bias. It amplifies it.

When a hiring tool favours certain names, it’s not inventing discrimination. It’s learning from decades of biased data. When an image generator associates professions with specific backgrounds, it’s mirroring patterns it found online.

Why This Matters

Caste bias in AI isn’t just an Indian problem. It’s a preview of how AI can perpetuate any social hierarchy.

Facial recognition systems show racial bias. Hiring tools show gender bias. Criminal justice algorithms discriminate against certain neighbourhoods.

The pattern is the same: AI learns from biased human data and replicates it at scale. A prejudiced person can harm individuals. A prejudiced algorithm can affect millions.

Intelligence Without Empathy

Building smarter AI is one challenge. Building fairer AI is another.

The tech industry focuses on making models bigger and faster. But if they’re better at reflecting human prejudice, that’s not progress.

The real work isn’t just technical. It’s ethical. It requires recognising that training data carries historical baggage and building systems that resist those patterns.

The Bigger Question

Can we build AI that’s better than the data it learns from?

Right now, AI is a mirror showing us our flaws. But we can build systems that recognise bias and push back against it.

That requires diverse teams, transparency, and acknowledging that technical excellence isn’t enough.

Because intelligence without empathy risks scaling the very inequities we hope to solve.

The caste bias in ChatGPT isn’t a bug. It’s a feature of the world the code learned from. Fixing it means confronting uncomfortable truths about the data we create and the systems we build.

And that work has barely begun.