DeepSeek V4 Is Not a Better Model. It's a Different Class of Model.

Everyone's been talking about V4 since the technical report dropped on April 24. And the benchmark numbers are wild — V4-Pro-Max matching Claude Opus 4.6 on agentic tasks, Codeforces rating of 3206 (the first open model to rival a closed-source model on competitive programming), 1M token context windows as the default.

But if you focus on the benchmarks, you'll miss the actual story.

DeepSeek V4 isn't a bigger V3. It's a ground-up rethink of how to train a frontier model at scale — one that makes certain previously expensive things (like million-token context inference) so cheap they become routine. This post breaks down exactly how.

The Problem V4 Was Built to Solve

Here's the wall every long-context LLM hits: attention is quadratic.

As context grows, the KV cache explodes. At 1 million tokens, you're not just paying 10x more than at 128K — you're paying closer to 60x more in memory. That makes truly long-context reasoning either comically expensive or practically impossible.

DeepSeek V3 had a 128K context window. V4 needed to jump to 1M. The math didn't work — until they changed the architecture.

The Four Big Technical Bets

1. CSA + HCA: A New Attention Stack

V4 replaces standard attention with a hybrid of Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). The core idea is deceptively simple: not all tokens deserve equal attention.

Instead of comparing every token against every other token in the context window, V4 compresses older tokens into a denser representation and keeps recent tokens at full resolution. Think of it like how your brain handles a conversation — you remember the gist of what was said an hour ago, but your last few sentences are crystal clear.

The result is striking:

V4-Flash is even more aggressive — 10% compute, 7% memory. This is the number that changes the economics of long-context AI entirely.

2. Manifold-Constrained Hyper-Connections (mHC)

Deep networks are unstable. Stack 100+ transformer layers and the residual connections that were supposed to help can start causing gradient instability. This is well-documented and usually patched over with careful tuning and warmup schedules.

mHC (from a December 2025 paper, arXiv:2512.24880) replaces plain residual addition with connections that are constrained to lie on a manifold — essentially guiding signal propagation through geometrically consistent paths across layers. The paper has been cited over 33 times since release and is confirmed in V4's final architecture.

In practice, this allowed DeepSeek to train V4-Pro at 1.6 trillion parameters with better stability than V3 at 671B. That's a 2.4x scale jump without blowing up training.

3. Muon Optimizer at 1.6T Scale

Most frontier labs still use AdamW. DeepSeek ships V4 with the Muon optimizer — a second-order method that uses curvature information to take better steps. This had been tried at smaller scales, but deploying it at 1.6T parameters is a first.

The engineering lift here is non-trivial. Muon requires computing matrix norms that don't fit neatly into distributed training setups. DeepSeek's kernel team built custom TileLang DSL kernels with bitwise-reproducible determinism across runs to make it work. It's the kind of detail that doesn't show up in benchmark tables but matters for anyone trying to replicate this at scale.

4. On-Policy Distillation Instead of RLHF

This is the post-training decision that surprised everyone.

V4 doesn't use RLHF in the traditional sense. Instead, it runs a two-stage pipeline:

Stage 1 — Train domain specialists separately. DeepSeek trains independent expert models for math, code, agent tasks, and instruction-following — each via SFT + domain-specific RL. These specialists reach peak capability in their domain without compromising on each other.

Stage 2 — Distill them into one generalist. Ten+ specialist teacher models are consolidated into V4 using on-policy distillation with a reverse-KL divergence loss. The student (V4) doesn't just mimic outputs — it learns the distribution of each expert's reasoning.

This "many specialists → one generalist" recipe is cleaner than mixed RLHF pipelines and sidesteps reward hacking across domains. The quality of each specialist's training data directly caps the unified model's ceiling. It also explains why V4 feels sharp across domains rather than averaged-out.

The Training Data Story (Often Overlooked)

Architecture gets the headlines. Data is where the actual decisions live.

V3 trained on 14.8 trillion tokens. V4-Pro trained on 33 trillion tokens — more than double. But scale isn't the story here. How those tokens were curated is.

Long-document emphasis. V4's corpus specifically prioritizes scientific papers, technical reports, and academic material. Content that's only valuable when read in full — the kind of text where a model needs to track arguments across tens of thousands of tokens. Short, repetitive web content gives diminishing returns at million-token training lengths.

Anti-model-collapse filtering. DeepSeek explicitly filters out batched auto-generated and templated content from web data. They cite model collapse research as an engineering constraint, not a theoretical worry. A model trained on recycled AI text loses exactly the diversity that makes 33 trillion tokens worth something.

Agentic data during mid-training. Tool-use patterns, multi-step workflows, and environment-interaction sequences are baked into the base model before post-training. This is different from layering agent behavior on top through fine-tuning — it's structural.

What V4 Can Actually Do

Let's be concrete about where V4 lands in practice.

Competitive Programming: Codeforces rating 3206 for V4-Pro-Max — first open model to match a closed-source model here.

Agentic Coding: Optimized for Claude Code, OpenClaw, and CodeBuddy. Internal survey of 85 experienced developers: 90%+ included V4-Pro in their top model choices for coding.

Long-Context Reasoning: 1M token context as default, not an upsell. An AI coding assistant that can read an entire codebase without losing the thread. A research agent that can hold a full archive in working memory.

Benchmarks vs. SOTA:

• V4-Pro beats GPT-5.2 and Gemini-3.0-Pro on reasoning
• Approaches Claude Opus 4.5 on internal agent evals (200+ real R&D tasks from 50+ engineers)
• Leads all open-source models on coding, math, and STEM

Pricing:

• V4-Pro: $1.74/M input tokens, $3.48/M output tokens
• V4-Flash: $0.14/M input tokens, $0.28/M output tokens

For reference: comparable closed-source API access is often 5–20x more expensive per token at this capability tier.

How to Run V4 Locally (Quick Start)

V4 weights are available on HuggingFace. For the Flash variant (more manageable at ~284B total, 13B active):

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "deepseek-ai/DeepSeek-V4-Flash"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
 model_id,
 torch_dtype=torch.bfloat16,
 device_map="auto",
 trust_remote_code=True
)

messages = [{"role": "user", "content": "Explain the CSA/HCA attention hybrid in V4."}]
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)

output = model.generate(input_ids, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0][input_ids.shape[1]:], skip_special_tokens=True))

For production inference, vLLM with ROCm (MI300X) or SGLang on NVIDIA Blackwell are the recommended serving backends. The KV cache reduction means V4-Flash fits comfortably on configurations that would have OOM'd with V3 at equivalent context lengths.

Key Takeaways

• 1M context at 10% of the memory cost of V3.2 is the engineering breakthrough. CSA/HCA makes this possible — not brute-force hardware.
• mHC enables stable training at 1.6T parameters. This is the new residual connection standard for extreme-depth networks.
• On-Policy Distillation replaces RLHF. Train specialists first, merge via distillation. Cleaner, harder to game, domain-coherent.
• Data diversity matters as much as scale. 33T tokens of curated long-document data beats 50T tokens of low-quality web text.
• Use V4-Flash for cost-sensitive pipelines. At $0.14/M input tokens, it's one of the cheapest frontier-class models ever released.
• Don't use V4-Pro-Max as a drop-in replacement for smaller models. The 1.6T parameter scale means inference infrastructure requirements are real.

Why This Matters Beyond Benchmarks

DeepSeek V4 makes a specific argument: that open-weights models can match closed-source frontier capability and be significantly cheaper to run — if you rethink the architecture from scratch rather than scaling what already exists.

The CSA/HCA attention hybrid doesn't just make V4 cheaper. It makes a whole category of applications economically viable: full-codebase reasoning, long-document research agents, persistent memory across sessions, multi-turn planning over hour-long context windows. These weren't practical before not because the ideas were wrong, but because the compute cost made them non-starters.

Now they're just features.

The race isn't about who has the biggest model anymore. It's about who can make the right compute cheap. V4 just moved the finish line.