IT WAS ABOUT LEARNING LANGUAGE FIRST.

GPT-1: A Retrospective on the Idea That Started It All

Learnings from the GPT paper

Read it here….

GPT-1 was the first language model built on a decoder-only Transformer architecture, introduced in the 2018 paper “Improving Language Understanding by Generative Pre-Training” by Radford et al.

Author-created visual based on the GPT-1 paper (Radford et al., 2018)

The authors showed that language shares common structure across many NLU tasks. When a model learns this structure first, it can reuse that knowledge across tasks instead of starting from scratch each time. To make this work, they used a semi-supervised learning approach.

Let’s step into the paper to see what was proposed and why it worked so well.

Previous Explorations of Pretraining Prior to GPT-1

Before GPT-1 popularized generative pretraining with Transformers, researchers had already explored how learning from unlabeled text could improve downstream NLP tasks.

An early example is the 2015 paper “**Semi-supervised Sequence Learning” by Dai and Le**. The authors showed that pretraining sequence models on large unlabeled corpora, either through standard language modeling or sequence autoencoders, leads to better initialization for supervised tasks. This approach improved training stability and generalization for text classification, demonstrating that :

Unsupervised sequence learning can capture transferable language structure.

Another relevant line of work appeared in 2018 with “**Generating Wikipedia by Summarizing Long Sequences” by Liu et al**. While not focused on transfer learning, this paper introduced a decoder-only Transformer capable of handling very long contexts and generating coherent multi-sentence text. It demonstrated that large, generative, decoder-only architectures could scale effectively, anticipating architectural choices later adopted by GPT-style models.

Illustrative timeline showing major milestones in the evolution of neural NLP, from early language models to pretrained architectures.

Together, these works established two key ideas that GPT-1 would later unify: learning general language structure from unlabeled text and using generative sequence models that scale with data and model size.

The Core Idea of GPT-1

GPT-1 was built to address a fundamental limitation in natural language processing:

Language understanding did not scale.

At the time, NLP systems were trained separately for each task. They relied heavily on labeled data, used task-specific architectures, and often depended on hand-engineered features. As a result, learning was fragmented, knowledge was not reused across tasks, and progress depended on costly human annotations.

At the same time, large amounts of raw text were already available. However, most models could not take advantage of it effectively. Supervised, discriminative training optimized directly for task labels rather than learning the underlying structure of language, which limited how well models generalized beyond the tasks they were trained on.

GPT-1 proposed a different learning order.

Instead of learning tasks first, the model learned language first. The breakthrough was not architectural innovation, but a change in training philosophy. GPT-1 adopted a semi-supervised training approach split into two stages: unsupervised pretraining followed by supervised fine-tuning.

During pretraining, the model was trained with a single language modeling objective, learning the probability P(wt∣w1,…,wt−1) from large amounts of unlabeled text.

This reframed the core problem in NLP. The limitation was not model architecture, but the heavy dependence on labeled supervision. Even when labeled data was available, unsupervised pretraining dramatically improved learning speed and performance. All of this was achieved using a decoder-only Transformer.

The intuition behind GPT-1 mirrors how humans learn language. We do not begin by solving tasks one at a time. A child does not study sentiment analysis, translation, or question answering in isolation. Instead, they first absorb language by listening, reading, and recognizing patterns.

Only after this shared understanding is formed do specific tasks emerge naturally.

GPT-1 followed the same learning order:

It learned how language works first, and only later was adapted to use that knowledge for particular tasks.

Architecture

GPT-1 uses a Transformer-based, decoder-only architecture.

Comparison of the Transformer architecture vs. decoder-only Transformer . (Image from the Transformer and GPT paper)

If you’re unfamiliar with Transformers, you can read here:

[embed]The Transformer Architecture What the “Attention Is All You Need” paper actually taught mepub.towardsai.net

The architecture is intentionally simple. Since the training objective was language modeling, the model only needed to generate the next token based on previous ones. A decoder does exactly that.

GPT-1 uses causal self-attention, so each token can look only at the tokens before it. This enforces a strict left-to-right flow and prevents the model from seeing the future during training. The same mechanism is repeated across 12 decoder blocks, letting the model gradually build deeper language understanding.

The architecture made generation possible, but training is what taught the model how to use language.

Causal self-attention

Causal self-attention simply means that a token is allowed to look only at itself and the tokens that came before it, not at any future tokens.

Masking in transformers (Source: Gemini)

In practice, causal self-attention is implemented by applying a causal mask inside multi-head attention, ensuring each token can attend only to itself and past tokens. It matters because:

In practice, this is done using a causal mask (a triangular mask) that blocks attention to positions on the right. It matters because:

• It preserves the left-to-right generation property of language models.
• It prevents information leakage from future tokens during training.
• It makes training and inference consistent with how text is generated.

Training Setup

Training followed a semi-supervised setup, consisting of two stages: first, unsupervised pretraining on large-scale unlabeled text, followed by supervised fine-tuning on specific downstream tasks.

Unsupervised Pretraining

Pretraining was performed on **BooksCorpus, a dataset of over 7,000 unpublished books. The key advantage of this corpus was its long, contiguous text**, which allowed the model to learn long-range dependencies.

The model was first trained using standard language modeling, with the objective of maximizing the likelihood of the next token given previous tokens:

Language modeling objective used during pretraining, where the model maximizes the log-likelihood of each token given its preceding context.

Here:

U=(u1​,u2​,…,un​): sequence of tokens (words, subwords, or characters).

*ui: token the model is trying to predict at position i.*

*ui−k,…,ui−1(context): previous tokens the model is allowed to look at. This enforces causality: no future tokens are used.*

*Θ: model parameters.*

At every position i, the model answers one question:

Given the past tokens, how likely is the next token?

Do not get confused by the idea of maximizing here. The model is simply trained to make the real text likely. At each step, it tries to assign a high probability to the correct next token based on the tokens it has already seen. This goal is written mathematically as maximizing the log-probability of the correct token. In practice, training algorithms minimize a loss, so we flip the sign and minimize the negative log-probability instead. This is what we commonly call cross-entropy loss.

Supervised Fine-Tuning

After pretraining, the same model was fine-tuned on supervised NLP tasks such as natural language inference, question answering, semantic similarity, and classification.

Crucially, no architectural changes were made. Tasks were adapted using task specific input transformations (earliest form of **prompting)**, reusing the pretrained language model as-is. Fine-tuning converged quickly, often within just a few epochs, showing that most of the learning had already happened during pretraining.

Input transformations for fine-tuning on different tasks (Image from GPT paper)

This figure illustrates how GPT-1 solves many different NLP tasks using one single transformer model.

Instead of designing a new architecture for each task, GPT-1 rewrites every problem as a text sequence. Special tokens such as Start, Delimiter, and Extract are used to structure the input in a consistent way.

Once the sequence is formed, it is passed through the same transformer. The model builds a contextual representation of the entire input, regardless of whether the task is classification, entailment, similarity, or multiple choice.

The final prediction is made by reading the representation at the Extract position and passing it through a small linear layer. This is the only task-specific component.

For tasks involving two texts, like entailment or similarity, both texts are placed in the same sequence separated by a delimiter. In multiple-choice tasks, the same context is paired with each possible answer, and the model scores each option independently. In essence :

Do not change the model for each task change the input format and let the language model generalize.

To understand GPT-1 fully, it is important to note what it did not attempt. It used no instruction tuning, no RLHF, and no human preference alignment. By today’s standards, it was limited in scale, and despite unsupervised pretraining, it still relied on labeled data for supervised fine-tuning. These constraints are not flaws, but the starting point of modern language models.

GPT-1 defined the blueprint that GPT-2, GPT-3, and modern foundation models simply scaled. Pretraining became representation learning, and language became the interface for tasks, instructions, and reasoning.

GPT-1 was not powerful because it solved many tasks. It was powerful because it learned one thing well: language itself.

Reference

Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI. Available at: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf