AI 1.07 — Machines and Learning Languages:

What Generative AI Built, What It Broke and What It Could Not Do

Attention is all you need:

For the better part of the 2010s, the field of NLP was dominated by Recurrent Neural Networks and their variants like LSTMs, GRU and NMTs. They were popular but had capabilities to process words only one at a time. This made them slow and prone to issues. And one among the major flaws which we noticed among them was that the data being read from the beginning of the long sentences were forgotten by the end ie. The vanishing gradient problem. Earlier attempts had tried to fix this, like Sutskever, Vinyals and Le’s sequence-to-sequence models in 2014 but they were not very successful.

To tackle this problem, in June 2017, eight researchers at Google Brain and Google Research published a paper titled Attention Is All You Need. At its core, the paper made a bold claim: you do not need recurrence, you do not need convolution, you do not need sequence, you do not need memory gates, you do not need hidden states, you do not need any of any of it — all you need is… Attention.

https://medium.com/machine-intelligence-and-deep-learning-lab/transformer-the-self-attention-mechanism-d7d853c2c621

Instead of reading a sequence word by word, the introduced transformer architecture processed every word in parallel. Positional encodings whic helped preserved order. “Self-attention”, computed through queries, keys and values, let the model determine which words mattered most to each other. The result was a model that could look at an entire context simultaneously, everything everywhere all at once, in parallel, on a GPU hardware. Training times plummeted. Performance soared. The paper set new records on English-to-German and English-to-French translation benchmarks, achieving a then-unprecedented BLEU score of 28.4.

It was elegant. It was fast. And it was about to change the research world.

Technicalities of the GPT revolution and a callback to Turing Test from the 1950s:

In June 2018, OpenAI published a paper titled Improving Language Understanding by Generative Pre-Training. The idea was simple, but deceptively so. It took taking the decoder half of the Transformer, pre-train it on a massive corpus of text scraped from the internet. This was done by having it predict the next word in a sequence, after which it was fine-tuned it on specific tasks. The pre-training gave the model a broad understanding of language. The fine-tuning made it useful. The first iteration of GPT built by OpenAI had a whopping 117 million parameters. But this was only tip of the iceberg.

GPT-2, released in February 2019 with 1.5 billion parameters, generated text so coherent that OpenAI initially withheld the full model out of concern for misuse (They used to do that sort of thing!). GPT-3, in June 2020, scaled to 175 billion parameters and demonstrated something few had anticipated: emerging abilities. Without being explicitly trained on tasks like arithmetic, translation or code generation, GPT-3 could perform them simply by being given a few examples in the prompt. This was called few-shot learning, and it caught the reseachers off guard with its customizability.

What the Transformer gave AI was not just a better architecture. It gave it a scaling law: make the model bigger, feed it more data, and it gets better. Reliably. Predictably. And almost unsettlingly so.

But there was a gap between GPT-3 the research marvel and ChatGPT the phenomenon which came later. The gap was Reinforcement Learning from Human Feedback (RLHF). In a 2022 paper, OpenAI’s Ouyang et al. showed that you could take a powerful but erratic language model and fine-tune it using human preferences. Humans evaluated and ranked model outputs. The model learned which responses people actually wanted and the model actually got better!

RLHF turned research model GPT-3.5 into the consumer product “ChatGPT”, and in November 2022, ChatGPT would go on to became the fastest consumer application to reach 100 million users. In a whopping Two month’s time. It was able to write essays, debug code, explain quantum mechanics to a ten-year-old and hold what truly felt like a genuine conversation, but it was not perfect. It had a major recurring problem of hallucination. It could be confidently wrong. But it was undeniably extraordinary.

And then, in March 2025, after a whopping 75 years after Dr. Alan Turing posed his question, GPT-4.5 would become the first model to pass the Turing test. The test was simple, and had been the benchmark for Artificial Intelligence in general and GPT4.5 would become the first model to pass.

Extended GenAI and the Lawsuits which followed:

In 2014, Ian Goodfellow, then only a PhD student at the University of Montreal, proposed Generative Adversarial Network or simply GANs. The concept was elegant and slightly mad: train two neural networks against each other. One generates fake images. The other tries to detect them. You were able to essentially avoid “Reinforcing” human efforts.

https://medium.com/sigmoid/a-brief-introduction-to-gans-and-how-to-code-them-2620ee465c30

The two neural networks improve together, locked in an arms race, until the generator produces images that the discriminator couldn’t distinguish from the training set.

GANs were able to produce increasingly realistic faces, landscapes and objects, but they were notoriously difficult to train and prone to mode collapse where the generator learns to produce only a narrow set of outputs. The architecture that eventually overtook them was the diffusion model, which worked on a completely different principle: take a real image, gradually add noise until it becomes static, then train a neural network to reverse the process — to reconstruct an image from pure noise, guided by a text prompt.

https://developer.nvidia.com/blog/improving-diffusion-models-as-an-alternative-to-gans-part-2/

OpenAI released DALL-E in January 2021 and DALL-E 2 in April 2022. Stability AI open-sourced Stable Diffusion in August 2022. Midjourney, accessible through a Discord bot, became the tool that put AI art into the hands of millions. In September 2022, a Midjourney-generated image won first place at the Colorado State Fair’s fine art competition. The artist, Jason Allen, had typed a text prompt and the judges did not know then.

https://openai.com/index/dall-e-2/

The lawsuits followed. In January 2023, Getty Images sued Stability AI for training on millions of copyrighted photographs without permission or compensation. That same month, three artists, Sarah Andersen, Kelly McKernan and Karla Ortiz, filed a class-action lawsuit against Stability AI, Midjourney and DeviantArt, arguing that the models had been trained on their work without consent. We had a new legal question in order. If a model learns from copyrighted art and generates new images in a similar style, who owns what?

Then came video. In February 2024, OpenAI demonstrated Sora, a model that could generate coherent 60-second videos from text descriptions. Runway and Pika had been producing shorter clips, but Sora’s fidelity — consistent physics, coherent camera movement, characters that persisted across frames — was a step change.

https://www.bbc.co.uk/bitesize/articles/z4rsmbk

And it was not just the Western labs. China’s DeepSeek, already making waves in language and reasoning, pushed into multimodal generation — proving that the race to teach machines to see, hear and create was global from the start.

Audio had its own lineage. DeepMind’s WaveNet in 2016 produced speech so natural that listeners could barely distinguish it from a human recording. By 2024, tools like Suno and Udio were generating full songs — lyrics, melody, instrumentation, vocals — from a text prompt. The music industry, still recovering from the disruptions of streaming, found itself staring at another one.

In April 2023, a track called “Heart on My Sleeve”, generated using AI-cloned voices of Drake and The Weeknd. It went viral across streaming platforms before being pulled down. It sounded authentic enough that listeners could not tell.

https://www.youtube.com/watch?v=dKgZj1R1I1A

By June 2024, the RIAA, on behalf of Universal Music Group, Sony Music and Warner Music, sued both Suno and Udio for copyright infringement, alleging the models had been trained on copyrighted recordings without licence. The music industry had seen Napster and Spotify reshape distribution, but this was different. This was reshaping creation of music itself.

Voice assistants also had their own battleground. In May 2024, OpenAI demonstrated a voice mode for GPT-4o using a voice called “Sky” that bore a striking resemblance to Scarlett Johansson — who had previously declined OpenAI’s request to use her voice. Johansson issued a public statement saying she was “shocked, angered” and hired legal counsel. OpenAI had paused the voice but the incident raised the same creative yet legal question again: who owns what?

https://www.theguardian.com/technology/article/2024/may/27/scarlett-johansson-openai-legal-artificial-intelligence-chatgpt

Coding was no exception. OpenAI’s Codex, released in 2021 and built on top of GPT-3 was trained on billions of lines of public code from GitHub. It powered GitHub Copilot, which by 2024 was writing 46% of all code on files where it was enabled. The implication was not that programmers would vanish. It was that the profession was being compressed. A junior developer with Copilot could produce output that previously required an experienced architect. Vibe coding was the new normal and everyone started getting used to it. We would not stop seeing new models released every other month which outperformed the previous one.

What worked and what didn’t:

For all the astonishment, the models had clear limitations. They hallucinated. In early 2023, a New York lawyer used ChatGPT to prepare a court filing and submitted it citing six legal cases that did not exist. The model had fabricated case names, citations and judicial opinions, all of which read convincingly. The lawyer was sanctioned by the court.

They inherited bias. A 2023 study by researchers at Stanford and McMaster University found that GPT-4, when evaluating resumes, consistently rated candidates with names perceived as White or male more favourably than equally qualified candidates with names perceived as Black or female. The bias was not programmed in. It was inherited from the text of the internet.

Google’s Bard, in its first public demo in February 2023, incorrectly stated that the James Webb Space Telescope had taken the first pictures of an exoplanet outside our solar system. It had not. The error wiped $100 billion off Alphabet’s market capitalisation in a single day.

https://www.reuters.com/technology/google-ai-chatbot-bard-offers-inaccurate-information-company-ad-2023-02-08/

And the copyright question was not going away. In December 2023, The New York Times sued OpenAI and Microsoft. They argued that the models had been trained on millions of copyrighted articles and could reproduce them nearly verbatim.

https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html

The models were fluent. They were not honest. Fluency without honesty is a dangerous combination.

The grim prognostications piled up — that the models would erode trust, displace professions and flood the internet with plausible nonsense. And yet, the patches came fast. Retrieval-augmented generation, introduced by Lewis et al. in 2020, let models pull in verified documents before responding. Tool use and code interpreters allowed models to call calculators, query databases and run code rather than guessing. Reasoning models internalised deliberation. Each iteration was measurably better. The patches were becoming part of the architecture itself.

No more AI Winters in sight:

Research was abundant. But it was expensive. Training GPT-4 reportedly cost over $100 million and required 25,000 NVIDIA A100 GPUs running for roughly 100 days. Google’s Gemini Ultra used tens of thousands of TPU v5p chips across multiple data centres. The energy consumption of these training runs rivalled that of small cities.

The data centres, the chips, the cooling systems or the energy grids are as much a part of the AI story.

https://www.scientificamerican.com/article/china-powers-ai-boom-with-undersea-data-centers/

We don’t have any more winters in sight. No previous wave of AI had moved this fast, no previous wave had this much enthusiam or participation and one of the key reasons for the success is the abundance of cloud and infra we have built in the last couple of decades.

Up Next: AI 1.08 — The Silicon Beneath: Big Orgs and Compute and AI-zation of Everything

References and further reading:

1. Attention Is All You Need — Vaswani et al. (2017): https://arxiv.org/abs/1706.03762

2. Improving Language Understanding by Generative Pre-Training — Radford et al. (2018): https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

3. Language Models are Few-Shot Learners (GPT-3) — Brown et al. (2020): https://arxiv.org/abs/2005.14165

4. Training language models to follow instructions with human feedback — Ouyang et al. (2022): https://arxiv.org/abs/2203.02155

5. Generative Adversarial Networks — Goodfellow et al. (2014): https://arxiv.org/abs/1406.2661

6. Denoising Diffusion Probabilistic Models — Ho et al. (2020): https://arxiv.org/abs/2006.11239

7. WaveNet: A Generative Model for Raw Audio — van den Oord et al. (2016): https://arxiv.org/abs/1609.03499

8. Evaluating Large Language Models Trained on Code (Codex) — Chen et al. (2021): https://arxiv.org/abs/2107.03374

9. BERT: Pre-training of Deep Bidirectional Transformers — Devlin et al. (2018): https://arxiv.org/abs/1810.04805

10. Constitutional AI: Harmlessness from AI Feedback — Bai et al. (2022): https://arxiv.org/abs/2212.08073

11. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Wei et al. (2022): https://arxiv.org/abs/2201.11903

12. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al. (2020): https://arxiv.org/abs/2005.11401

13. Sequence to Sequence Learning with Neural Networks — Sutskever et al. (2014): https://arxiv.org/abs/1409.3215

14. Neural Machine Translation by Jointly Learning to Align and Translate — Bahdanau et al. (2014): https://arxiv.org/abs/1409.0473