GPT-3: 175 Billion Parameters and the Death of Fine-Tuning

By Ramprasad | NLP Series — Part 5

If you’d like, you can take a look at **GPT-1 , BERT , GPT-2 & BART **before moving on to read about GPT-3

Reference Paper GPT-3: “Language Models are Few-Shot Learners” — Published on May 2020 by Open AI

Core Idea

GPT-3 showed that if you make a language model large enough, it stops needing task-specific training — a few examples in the prompt are enough, which is both the breakthrough and the risk.

Zero / one / few-shot in-context learning. Give examples in the prompt itself. Model adapts reliably with no weight updates.

What they built:

GPT-3 — a 175 billion parameter language model. 10× bigger than anything built before.

What they discovered:

Make the model big enough and it can perform tasks with just a few examples in the prompt — no fine-tuning, no labeled dataset, no weight updates. Just show it the task in text and it figures it out.

What it could do:

Translation, question answering, arithmetic, unscrambling words, using new words in sentences — all from prompts alone.

Architecture

GPT-3 uses the same decoder-only Transformer architecture as GPT-1 and GPT-2. The design principle: don’t change what works — just scale it.

Corpus — 570 billion tokens

GPT-3 was trained on a mixture of five datasets, each weighted differently. CommonCrawl was sampled below its raw proportion because raw web text contains a lot of noise — quality filtering reduced it significantly.

Architecture parameters

Vocabulary size — 50,257 tokens

Context window — 2,048 tokens

Layers (stacked Transformer decoder blocks) — 96

Attention heads (Independent attention computations run in parallel within each layer) — 96

d_model (hidden size) — 12,288 = 96 heads × 128 dims per head

Feed-forward size — 49,152

Total parameters (all weight matrices) — ~175B

Conclusion

GPT-3 didn’t learn translation, coding, or arithmetic — nobody taught it those things. It learned language at such enormous scale that everything else came along for the ride.

It was brilliant but dangerous, powerful but unpredictable.— but is it safe enough to actually use? Wait for GPT-3.5**