Parameters & Weights — The Numbers That Define What AI Knows

Series 2, Part 3: Technical AI Terms, Simply Explained

You’ve probably heard someone say:

“GPT-4 has over a trillion parameters.”

Or: “This model has 7 billion weights.”

These numbers get thrown around like they mean something obvious. But what actually is a parameter? What is a weight? And why does the number of them matter so much?

This post answers all of that — simply and clearly.

What Does an AI Model Actually Store?

When an AI model is trained, it doesn’t store facts the way a database does. It doesn’t have a giant table somewhere that says:

“Paris is the capital of France” “Water boils at 100°C”

Instead, everything an AI model “knows” is encoded as numbers — millions or billions of them — stored inside the model’s structure.

These numbers are called parameters or weights.

They are the memory of the model. They are what learning actually produces. And they are what makes one AI model different from another.

What Is a Weight?

Let’s build up to this with a simple analogy.

Imagine you’re hiring a candidate for a job. You evaluate them on three factors:

• Technical skills
• Communication skills
• Cultural fit

But not all three factors matter equally to you. You decide:

• Technical skills → 50% importance
• Communication skills → 30% importance
• Cultural fit → 20% importance

These percentages are essentially weights — numbers that determine how much each factor influences your final decision.

Now imagine an AI doing the same thing — but instead of 3 factors, it’s evaluating thousands of input signals. And instead of you manually deciding the importance of each, the AI learns the right weights automatically from data.

That’s what weights are:

Weights are numbers that determine how much influence each input has on the model’s output.

What Is a Parameter?

Parameters is the broader term. It includes weights — and also another type of number called biases (not the same as the bias we discussed in Part 1).

So:

Parameters = Weights + Biases

In practice, when people say “this model has 7 billion parameters” — they mean the total count of all the weights and biases inside the model combined.

A Simple Neural Network Example

Let’s make this concrete with a tiny example.

Say you’re building a simple AI to predict house prices based on two inputs:

• Size of the house (in sq ft)
• Number of bedrooms

Your tiny neural network might look like this:

Input → Multiply by weights → Add bias → Output (predicted price)

Predicted Price = (Size × W1) + (Bedrooms × W2) + B

Where:

• W1 = weight for size
• W2 = weight for bedrooms
• B = bias (a baseline adjustment)

This tiny model has 3 parameters — W1, W2, and B.

During training, the model starts with random values for W1, W2, and B. It makes a prediction. It checks how wrong it was. It adjusts W1, W2, and B slightly. It tries again.

After thousands of iterations, the weights settle at values that produce accurate predictions — something like:

Predicted Price = (Size × 4,500) + (Bedrooms × 2,00,000) + 10,00,000

The model has “learned” that each square foot adds ₹4,500, each bedroom adds ₹2,00,000, and there’s a base price of ₹10,00,000.

That learning is stored entirely in those three numbers.

What Is Bias (In This Context)?

Quick clarification — because this word appears twice in AI with different meanings:

Bias Difference

The bias parameter is just a number that helps the model fit data better — like saying “even if size and bedrooms are both zero, a house still has some base value.”

Why Do Large Models Have Billions of Parameters?

Our house price example had 3 parameters. Real models have billions. Why?

Because real problems are incomparably more complex.

When a language model reads a sentence, it isn’t looking at 2 inputs — it’s processing thousands of tokens, each with hundreds of dimensions of meaning, across dozens of layers of neural networks.

Each connection between neurons in those layers has its own weight. Each neuron has its own bias. Add them all up across billions of connections and hundreds of layers — and you get billions of parameters.

A rough comparison:

Comparison of models

More parameters = more capacity to learn complex patterns. But more parameters also means:

• More data needed to train
• More compute needed
• More memory to run
• Higher risk of overfitting (sound familiar from Part 2?)

Do More Parameters Always Mean a Better Model?

Not necessarily — and this is an important .

A model with more parameters has more capacity — but capacity only helps if:

• You have enough data to fill it meaningfully
• You train it correctly
• You use it for the right task

A 7-billion parameter model trained well on the right data can outperform a 70-billion parameter model trained poorly. Parameters are potential — training is what realizes it.

This is why smaller, well-trained models like Mistral 7B often compete with much larger models on specific tasks. Size matters — but it’s not everything.

How Weights Change During Training

Here’s the full picture of what training actually does to weights:

1. Initialization Weights start as small random numbers. The model knows nothing yet.

2. Forward Pass Data goes in, the model makes a prediction using current weights.

3. Loss Calculation The model checks how wrong the prediction was. This “wrongness” is called the loss (we’ll cover Loss Functions in Part 6).

4. Backward Pass (Backpropagation) The error is sent backwards through the network. Each weight gets a signal: “you contributed this much to the error — adjust by this much.”

5. Weight Update Weights are nudged slightly in the direction that reduces error.

6. Repeat This cycle runs thousands or millions of times until weights stabilize at values that produce accurate predictions.

When training is complete, the weights are frozen. The model is now ready to use — and everything it learned lives in those numbers.

Why This Matters

Understand why AI models are expensive: Storing and running billions of parameters requires significant memory and compute. That’s why running large AI models costs money — and why smaller, efficient models are increasingly valuable.

Understand model updates: When a company says they’ve “retrained” or “fine-tuned” their AI — they’ve adjusted the weights. The architecture might be the same, but the numbers inside have changed — and so has its behaviour.

Next up → **Series 2, Part 4: Activation Functions** — the hidden switch inside every neuron that decides whether a signal moves forward or stops — and why without it, even the deepest neural network would behave like a simple calculator.

This series is built for curious people who want to understand AI at a deeper level. Follow along and share with someone who’d find it valuable.