DECODING AI MODEL SPECS.

How to Read AI Model Specs When You’re the One Running Them

Every model you find online shows you the same two things. A parameter count, like 21B or 7B. And a format, like BF16 or FP8 or INT4.

I ignored both for longer than I should have. They looked like metadata. They are actually the entire answer to “will this run on my machine”, and once you can read them you can size a model in your head in about four seconds.

Here is what each one is.

The first number: weights

When someone says “we’re running Qwen 7B”, they mean there is a file on disk containing roughly seven billion numbers. Those numbers are called weights or parameters; the two words mean the same thing and get used interchangeably. The “7B” in the name is literally the count.

When the model starts, those weights are loaded from your drive to the GPU, where actual calculation happens.

Note: Always check memory bandwidth (GB/s). That tells you roughly how quickly data can be moved between VRAM and the GPU’s compute hardware.

Coming back to the topic, those “7B” numbers were created during training, which almost none of us will ever do. That’s the expensive part Meta, Alibaba, and OpenAI run on thousands of GPUs for months. What we do is the much simpler part: inference.

Inference is basically just multiplication at massive scale. Your text gets turned into numbers, those numbers get multiplied through all the weights in a fixed sequence, and the results get turned back into text. No clever branching, no rules engine, no lookup tables, just a lot of arithmetic.

Which leads to the thing that surprised me most: model size is a memory problem before it is an intelligence problem. Twenty-one billion numbers have to physically fit in your graphics card’s memory before anything else can happen.

So the next question is how big each number is.

Picture an enormous mixing desk in a recording studio, but instead of 48 faders it has seven billion, and every one has already been set to a precise position by someone else. You feed audio in one end, it passes through every fader, and something comes out the other end. You are not adjusting the faders. You are pushing signal through a configuration someone else spent millions of dollars discovering.

The second number: how many bytes per weight

That is what BF16 is telling you. It is the format each of those twenty-one billion numbers is stored in, and it decides the file size directly.

A bit is one binary digit, 0 or 1. Eight bits make a byte. BF16 means 16 bits per weight, so two bytes. Twenty-one billion weights at two bytes each is about 42 GB.

If your card has 24 GB, that model does not fit. Not “runs slowly”. Does not load.

Inside the 16 bits

Now the part that took me a while, and that nobody seems to explain: what those sixteen bits are actually doing.

A floating-point number is stored in three parts. A sign bit for positive or negative. An exponent, which is the scale, roughly how many zeros. And a mantissa(the significant digits). It is scientific notation in binary. The exponent decides how big or small the number can get; the mantissa decides how precisely you can state it.

Every format is just a different way of dividing the bits between those three:

Look at FP16 and BF16 for a second, because that comparison is the whole idea.

They are the same size. Sixteen bits each. What changed is the split. FP16 spends five bits on the exponent and ten on the mantissa. BF16 spends eight on the exponent and only seven on the mantissa.

BF16 is deliberately less precise than FP16, at identical cost.

That sounds like a bad trade until you know what it was built for. BF16 means “brain float”, and Google invented it specifically for machine learning after finding that neural networks care much more about range than about decimal places. A weight that overflows because the format could not represent a large enough number is a disaster. A weight that is accurate to seven digits instead of ten is fine. So BF16 keeps all eight exponent bits from FP32, giving it FP32’s full range, and pays for them out of the mantissa. That is why almost every model you download is published in BF16.

Going smaller: quantization

Back to the 42 GB model that does not fit on your 24 GB card. The fix is quantization, which means re-storing the same weights using fewer bits.

Halve the bits, halve the file. The honest analogy is the JPEG quality slider: same photograph, you are choosing how much information to discard for a smaller file. At 90% nobody can tell. At 60% you notice artefacts in the sky. At 20% it is blocky.

FP8 is roughly the 90% setting. INT4 is roughly the 60% setting.

What it costs, and where that bill lands

Every weight ends up slightly off the position it was trained to. Individually those errors are tiny. Whether they add up to anything depends entirely on what the model is doing, and this is where I had to change how I was thinking.

Ask a model to summarise something at 4-bit and you might get a slightly clumsier sentence. You read it, you notice, you re-run it. The error is loud, local, and self-correcting, because a person is standing right there looking at the output.

Now point a quantized model at a scanned table and let it read a 3 as an 8. What comes back is a perfectly well-formed number. Right column, right units, right number of digits. Nothing about it looks wrong. It goes into the database, gets indexed, and months later comes back out as a fact. There is no fluency signal to catch it, because fluency was never damaged.

Embedding models are the same one layer down. Quantize one and every vector shifts slightly off position. Nothing errors. Search still returns results. They are just quietly worse, forever.

So the allocation that follows looks backwards, and is not. We run our largest model, twenty-one billion parameters, at four bits, because a human reads its output and would catch a bad sentence. We run our embedding model, forty times smaller, at full precision, because nobody will ever notice its mistakes.

The model a person reads gets the cheap setting. The model that only ever feeds other software gets the expensive one.

The short version

• 21B is a count of numbers in a file. Twenty-one billion of them.
• BF16 is how many bits each one takes, and how those bits are split. Sixteen bits, arranged for range instead of precision, which is what neural networks turn out to need.
• Multiply the two for your memory requirement. 21B at two bytes is 42 GB.
• Quantization shrinks that by storing each weight in fewer bits. FP8 halves it, INT4 quarters it.
• Spend your precision where mistakes are silent and permanent. Save it where a person would catch them anyway.

I spent years building backend systems, then became responsible for running models on hardware instead of calling somebody’s API. The difficulty was never the concepts. It was that every explanation assumed I already knew forty terms, and each of those forty was explained using the other thirty-nine. So, I planned to start a short Medium article series to clear up a few concepts I stumbled over while learning.