Day 0: What Is a Language Model, Actually?

Notes from Chapter 0 (“Preflight”) of Under the Hood: Build Every Layer of a Large Language Model from Scratch, by Ramchand Kumaresan

I’m working through Under the Hood, a book that promises to teach LLMs by having you build one, layer by layer, from a blank file. Rather than quietly finish it and post a highlight reel at the end, I’m going to write up what I learn after every chapter — partly to hold myself accountable, partly because explaining something is the fastest way to find out whether you actually understood it.

Chapter 0 isn’t a project. There’s no code to run yet. It’s a calibration chapter — the author’s attempt to make sure every reader starts from the same mental model before the actual building begins. Here’s what stuck with me.

1. A language model does one very narrow thing

Strip away the demos and the benchmarks, and a language model has exactly one job: given some text, predict what comes next. Not “understand” it, not “know” it — predict it.

Concretely: feed it “The capital of France is” and it doesn’t look up a fact in a database. It outputs a probability distribution over every token it knows — “Paris” gets assigned something like 82% probability, “Lyon” gets a sliver, “Wednesday” gets almost nothing. Generating text is just sampling from that distribution, appending the result, and repeating. One token, one distribution, over and over.

What got me was the framing that the model holds opinions, not records. That’s a genuinely different shape of object than I’d been picturing.

2. Tokens are not words

I’d always assumed tokens with words, and apparently that’s the single most common wrong intuition beginners carry in. Tokens are subword pieces produced by a tokenizer’s compression algorithm, and they have nothing to do with linguistic meaning — they’re statistical artifacts.

Two details reframed this for me:

• “cat” and “cats” get completely unrelated integer IDs. The model has to learn that they’re related from data; it doesn’t start out knowing.
• Code isn’t a special case — print(x) splits into four tokens, so the model absorbs programming syntax at the same subword level as prose.

3. Training is a three-move loop, repeated billions of times

This is the part that made the whole “mystery” of neural networks click for me. Every neural network — from a toy classifier to a 70-billion-parameter model — learns via the same three moves:

1. Guess (forward pass): push input through the model’s math and get a prediction.
2. Measure the wrongness (loss): compare the prediction to the correct answer and collapse it into a single number.
3. Assign blame and adjust (backward pass): work backward through the computation to figure out which parameters caused the error, and nudge each one a little in the direction that reduces it.

Repeat on billions of token pairs, and the loss slowly goes down. The book is emphatic that the backward pass isn’t mysterious — it’s the chain rule from calculus applied mechanically, in reverse, to every step of the forward pass. Nothing more exotic than that.

4. Four Python patterns you need before touching the code

This isn’t a Python course, but the author flags four idioms that show up constantly in the book’s code:

• Closures — functions that “remember” variables from the scope they were created in. This is exactly how a backward pass works under the hood: every node in the computation graph stores a closure that knows how to push gradients to its parents.
• Dunder methods (__add__, __mul__, __repr__, etc.) — how a custom class intercepts ordinary operators like + and *. This is the mechanism that lets a hand-rolled Value class secretly build a computational graph while looking like normal arithmetic. PyTorch tensors do the exact same trick at production scale.
• Classes, for bundling data and behavior — every neuron, layer, and model in the book is a class instance.
• List comprehensions, for building parameter lists and batches concisely.

None of these are exotic Python, but knowing to watch for them before diving into unfamiliar code saves a lot of “wait, what is this doing” moments.

5. Everything is a tensor, and shape is the thing to track

A tensor is just a list of numbers generalized to multiple dimensions — a 1D tensor is a list, a 2D tensor is a grid, a 3D tensor is a stack of grids. What makes them worth replacing plain Python lists with is that operations move through every value in lockstep, which is what makes GPU parallelism possible.

The shape that shows up constantly in language model code is [B, T, d] — batch size, sequence length, and embedding dimension respectively. I have a feeling I'll be squinting at shape mismatches for the rest of this book, so getting this notation into muscle memory early feels worthwhile.

6. “Large” just means parameter count and data volume — the mechanics don’t change

A small model might have 7 million parameters; a large one might have 70 billion. A model trained on a trillion tokens has seen roughly the text of a million books. But scaling those two numbers up doesn’t change what the model is doing — it’s still the same forward pass, same loss, same backward pass. “Large” is a statement about capacity, not a different mechanism.