How LLMs actually work — the beginner’s guide

Photo by Markus Spiske on Unsplash

You’ve probably heard that a language model just predicts the next word. That’s true, and it doesn’t tell you much.

The explanations out there come in two sizes. There are the analogies: autocomplete on steroids, a parrot, a well read intern. They’re easy to remember and they leave you none the wiser. Then there are the papers, which are exact and assume you already know what a scaled dot product is.

I wanted to actually see the middle bit. So I built a city.

TokenTown is an island where each district is one stage of a language model. A convoy of trucks drives the data around. Your prompt gets chopped up at the docks, turned into numbers at the foundry, stamped with its position, then driven around a ring road once per layer. At the far end a stadium works out how likely every possible next word is, a sampler picks one, and the winner drives back up a flyover so the whole thing can run again.

The city isn’t a drawing of a model. It runs one. The beams you see arcing over the warehouse are real attention weights, worked out in your browser while you watch, and they add up to exactly 1.

Open the city: https://laurentiugabriel.github.io/token-town/

Source: https://github.com/LaurentiuGabriel/token-town

Here is the walk round.

The docks: your prompt stops being text

The first thing that happens to your prompt is that it stops being writing.

A model doesn’t read letters. It reads numbers that stand for chunks of text, taken from a fixed list of somewhere between 50,000 and 200,000 chunks. Common words get a chunk to themselves. Unusual ones get cut into pieces. The space in front of a word normally travels with it, which is why tokenizer tools show you things like ·the rather than the.

That matters more than it sounds like it should. It’s why models miscount the letters in a word, and why the same sentence can cost you more in one language than another.

The foundry: numbers instead of words

Each chunk number is used to look up one row in a very large table. That row is a list of numbers, and from here on it is the only thing the model has.

The spelling is gone. The word is now a spot in a space with a few thousand directions to it. TokenTown uses 12 numbers so you can watch them move around on the back of a truck. Real models use four thousand or more.

The beacon: the model has no idea what order you wrote things in

Attention, which is the next stage, can’t tell word order by itself. Shuffle the words and it sees the same input. “dog bites man” and “man bites dog” would be identical to it.

So the position gets mixed into the numbers. Older models add a wave pattern. Most current ones rotate the numbers by an angle that depends on where the word sits, which people call RoPE.

The plaza: attention

This is where most of the work happens.

Each word gets turned into three things: a query, a key and a value. The query is what this word is looking for. The key is what each earlier word advertises about itself. The value is what that word hands over if you decide to use it.

The model checks the query against every earlier key, which gives it a score per word. Those scores get squashed so they add up to 1. Then it blends all the values together, using the scores as the recipe.

So a word looks back over everything before it, works out which of it matters, and pulls a bit of each one into itself.

It works well for two reasons. The matching is done on content rather than distance, so a word can reach back forty words if that’s where the useful thing is. And the whole process runs many times side by side with different settings. Real models run 32 or more at once, and they end up specialising in different jobs.

Watching real numbers throws up habits that trained models have too. One is strange. The very first word in the text often soaks up most of the attention, no matter what it is. People call this an attention sink, and the current guess is that heads dump attention there when they have nothing better to do with it. In a normal run of the city the first word takes about 87 percent and everything else shares the rest.

The warehouse: why long chats get expensive

Behind the plaza there is a row of silos, one per word, and a new one goes up every time the model writes something.

Keys and values don’t change once you’ve worked them out, so the model keeps them. Without that, writing word 500 would mean shoving all 499 earlier words back through every layer first.

The catch is memory. The store grows with the length of the conversation, then again with the number of layers, and it all sits on the GPU. That’s the real reason a long context window costs money, and why so much effort goes into shrinking it.

The bridge: one long thread

Every stage adds its result onto what it was handed rather than replacing it. So there is a single unbroken line of numbers running from the docks to the exit, and each block writes a small correction onto it.

That sounds like housekeeping. It isn’t. Adding instead of replacing is what lets anyone stack a hundred layers and still train the thing.

The mill: where the facts live

After attention comes an ordinary little neural network. It stretches the numbers wider, bends them, then squashes them back. Usually four times wider in the middle.

Two things stand out about it. It never looks at other words, so each one goes through on its own. And it holds roughly two thirds of the model’s weights. When a model knows a fact, the fact is mostly sitting in here.

The arch: now do it eighty more times

Then the whole ring runs again. Small models go round 12 times, big ones 80 or more, and every lap uses a completely separate set of weights. It isn’t a loop. It’s a stack of different blocks that happen to be the same shape.

Roughly speaking, the early blocks sort out grammar and local structure, and the later ones deal with meaning.

The stadium: no answer, just odds

At the end, the model compares its final list of numbers against every word in its vocabulary. That gives one score per word, and squashing the scores turns them into probabilities.

This is the part most explanations skip. The model never picks an answer. It works out how likely every possible next word is, all of them at once, and then something else has to choose. In the city that is a stadium full of towers, one per candidate, each as tall as its odds.

The sampler: where the randomness comes in

Temperature divides the scores before the squashing. Below 1, the favourite pulls further ahead and the model plays it safe. Above 1, the field flattens out and unlikely words start getting a look in.

Top-p, sometimes called nucleus sampling, takes the best words until their odds add up to p and bins the rest.

Then one word is drawn at random from whatever is left. That’s why you get a different answer the second time you ask the same question.

The flyover: one word, then start again

The chosen word gets stuck onto the end of the input and the model runs again from scratch.

Everything above happened to produce one word. Quite often not even a whole word.

It also explains two numbers you see in benchmarks. The first pass, called prefill, pushes your whole prompt through together. Every pass after that, called decode, pushes one word through and reads everything else out of the store. Prefill is why a long prompt is slow to start. Decode is why the rest arrives at a steady rate, and why you can’t make it faster by throwing more machines at it.

What is real and what is scenery

I’d rather say this up front than have someone find it out later.

Worked out live in your browser: the chopping up, the table lookup, the position code, the normalisation, attention with a proper mask over a store that really does grow, the additions, the little network, and the sampling. The bars on the truck are the real numbers. The beams are the real weights.

Shrunk down: 12 numbers per word instead of thousands, 2 attention heads instead of dozens, 2 to 12 layers instead of 80.

Faked on purpose: the weights are random. Nobody trained this. A random model writes gibberish, so the final word choice is nudged by a simple table of word pairs to keep the output readable. The attention scores are sharpened a little as well, so the picture looks like what a trained model produces.

So don’t read anything into the sentences it writes. The plumbing is the part that’s real.

Go and have a look

The thing I got wrong first time round was the speed. My early version ran a word through in about four seconds and taught nobody anything. The version I kept stops at each district for long enough to read the panel, which means the first word takes about four and a half minutes. It’s a slow walk round, deliberately.

It’s a plain static site. No build step, no dependencies, no sign in. Open the link, or clone it and open index.html.

Watch it run: https://laurentiugabriel.github.io/token-town/

Read the code: https://github.com/LaurentiuGabriel/token-town

The whole model is under 400 lines in js/toy-model.js if you want to check my working. The attention part is about 40 of them.

If I’ve explained something badly, or got something wrong, open an issue. I’d rather fix it.