From GPT-2 to DeepSeek: What’s Actually Inside a Language Model

Most explanations of LLMs either skip the architecture entirely or go so deep into matrix math that they lose everyone by paragraph two. This is neither of those.

If you’ve stared at a DeepSeek V3 architecture diagram and wondered what you’re actually looking at — what a “MoE layer” is, why there are 61 blocks, what RoPE does — this should help. I’ll start from GPT-2 not because it’s relevant in production today, but because it’s the cleanest skeleton version of what everything else built on.

GPT-2: The Skeleton

GPT-2 shipped in 2019 with 1.5 billion parameters. OpenAI was nervous enough about it that they initially sat on the release. In retrospect, that’s funny. A 1.5B model now runs on a phone.

But the architecture is still the architecture. Most large decoder-only language models — GPT-4, Llama, Mistral, DeepSeek — are variants of what GPT-2 established.

Here’s the flow when text goes in:

Token embeddings. Text gets split into tokens — roughly word fragments — and each token becomes a high-dimensional vector. GPT-2 used 768 dimensions. DeepSeek uses 7,168. That vector is the model’s representation of what the token means before it has seen any context.

Positional encoding. Transformers have no built-in sense of order. Without explicit position information, “the cat sat on the mat” and “the mat sat on the cat” look identical. GPT-2 added learned position embeddings. Modern models use RoPE (Rotary Positional Embedding), which handles longer sequences better.

The transformer blocks. This is the actual work. GPT-2 stacks 48 of them. DeepSeek stacks 61. Each block does two things:

1. Self-attention — looks at how each token relates to every other token in the sequence and updates its representation. When the model processes “mat” in “the cat sat on the mat,” attention lets it look back at “sat” and “cat” and decide what’s relevant to bring forward.
2. 1. Feed-forward network (FFN) — after attention mixes representations across tokens, the FFN runs each token through a two-layer MLP independently. This is where most of the factual knowledge stored during training actually lives, not in the attention layers.

Between these steps, layer normalization (RMSNorm in modern models) keeps activations stable. Residual connections — adding the input back to the output at each step — let gradients flow during training without vanishing.

Output head. After all 48 blocks, the final representation gets projected to vocabulary size. The model outputs a probability distribution over every possible next token. Sample from that, or take the argmax. That’s how you get text.

Clean. That’s GPT-2.

The Jump to Modern Models

The core loop hasn’t changed. What changed is scale, and then efficiency at scale.

Scale. Going from 1.5B to 70B or 700B means more blocks, wider embeddings, more attention heads. GPT-2 had 12 attention heads. DeepSeek has 128. More heads lets the model attend to different aspects of context simultaneously — one head might track syntactic structure, another entity references, another temporal relationships. Or at least that’s the intuition; the actual learned representations are harder to interpret cleanly.

Grouped-Query Attention. Original multi-head attention is expensive at inference because you cache key-value states for every head. Grouped-query attention (GQA) shares KV caches across groups of heads, cutting memory without hurting quality much. Most modern models use some version of this.

Context length. GPT-2 had a 1,024-token window. Current models run at 128k or more. Extending context isn’t just making the attention matrix bigger — there are real extrapolation problems with positional encodings at lengths not seen during training. RoPE handles this better than learned absolute embeddings, which is part of why it became standard.

Open Weights vs. Open Source: The Distinction That Matters

Popular coverage collapses this constantly, and it creates real confusion.

Open weights means the trained parameters are publicly released. You can download them, run them, fine-tune them. You may not have the training data, the full training code, or the infrastructure details. Llama 3 is open weights.

Open source means all of that — weights, data, training code, the full picture. Almost no frontier model is truly open source. BLOOM from BigScience is probably the closest, and it’s no longer competitive at the frontier.

With open weights you can run locally (no API dependency, no data leaving your machine), fine-tune on your own data, study behavior directly, and build without commercial restrictions depending on the license. What you can’t do: reproduce the training run, audit what was filtered during alignment, or verify the safety story independently. That gap matters more than most people admit.

The current open-weights landscape worth knowing: Llama 3 (Meta), Mistral and Mixtral (Mistral AI), Qwen (Alibaba), Gemma (Google), DeepSeek (DeepSeek AI). The quality gap between these and closed frontier models has been closing faster than anyone predicted two years ago.

DeepSeek V3/R1: Where the Numbers Get Weird

671 billion parameters. Only about 37 billion active per forward pass.

That’s Mixture of Experts (MoE), and it changes the tradeoffs in ways that aren’t obvious at first.

Standard dense FFN: every token goes through the same network. 671B parameters means every inference step uses all 671B. MoE replaces the FFN with a set of independent “expert” networks plus a router that picks which ones fire per token. DeepSeek has 256 experts per MoE layer. For each token, the router picks 1 shared expert plus 8 specialized ones. The other 247 sit idle.

The result is a 671B model that costs roughly as much to run (in compute FLOPs) as a ~37B dense model. Memory footprint for serving is still large — all weights need to stay loaded — but the actual computation per forward pass is much cheaper. It’s an unusual tradeoff: expensive to host, cheap to run.

The architecture layer by layer:

• Token embedding at 7,168 dimensions
• • 61 transformer blocks, each running: RMSNorm → Multi-head Latent Attention (128 heads) → residual, then RMSNorm → MoE → residual
• • Final RMSNorm → linear output to vocabulary (129k tokens)

The first 3 blocks use a dense FFN with hidden size 18,432 instead of MoE. Early layers handle basic token-level processing — syntax, token identity, surface structure — before the higher-level expert specialization makes sense to kick in.

Multi-head Latent Attention (MLA) is DeepSeek’s solution to the KV cache problem. At 128k context length, the KV cache becomes the real memory bottleneck in serving. MLA compresses keys and values into a lower-dimensional latent space, significantly reducing memory requirements at long context. The math is more involved, but the practical effect is that you can serve longer contexts without blowing up memory.

Tool Use: The Layer That Lives Outside the Architecture

The tool use loop looks like this: foundation model outputs a plan, tools execute against an environment, the environment feeds observations back, the model incorporates them. Humans sit in the middle providing instruction and course-correction.

None of this is in the transformer. There’s no tool block in those 61 layers. What the model learned during training is how to generate structured outputs — JSON, function call syntax, a specific format — that an external runtime can parse as instructions. The loop lives in the scaffolding around the model. The architecture itself doesn’t know it’s using a tool. It’s predicting tokens.

This matters because people conflate model intelligence with agent capability. A model that calls tools effectively isn’t necessarily reasoning better. It learned a pattern: when context looks like X, output something in format Y. The web search, the code execution, the database query — all of that happens outside the model entirely.

Agent behavior is a runtime property. The architecture just predicts the next token.

What This Changes Practically

A few things this architecture knowledge actually affects in day-to-day work:

The model has no memory between calls. Every forward pass is stateless. Context window is all the state that exists. Systems that feel “memory-aware” are reconstructing context from external storage on every call.

Bigger isn’t always better for specific tasks. A 7B model fine-tuned on your domain will often beat a 70B general model at that task. Fine-tuning a MoE model is more complex than fine-tuning a dense one — the routing behavior interacts with the learned representations in ways that need more care.

At 128k context, attention is the bottleneck, not the FFN. Latency and memory both scale with context length. MLA helps, but it doesn’t eliminate the problem.

Early layers handle structure, later layers handle meaning and reasoning. This is why targeted fine-tuning approaches like LoRA work at all — you’re not teaching the model to read again, you’re adapting the representations in the layers that matter for your specific task.

GPT-2 is still in there, somewhere under 61 layers and 128 attention heads and 671 billion numbers. The shape of the thing hasn’t really changed since 2017. What’s changed is how many of those things we stack, how cleverly we route computation through them, and what infrastructure we’ve built around them to make any of it useful.