Every AI Training Pipeline Has a Ceiling Problem

How SFT, RL, and distillation shape what your model can and can’t learn.

There’s a tradeoff baked into every model training strategy

The SFT-then-RL pipeline isn’t just convention. There’s a real compounding logic behind it and once you see it, the new wave of self-distillation methods becomes much easier to evaluate.

Why does the order of training actually matter?

Here’s something that bugged me when I first dug into post-training pipelines: almost every team does SFT first, then RL. But when you ask why, the answers tend to be vague, “SFT gets you a baseline,” “RL needs somewhere to start from.” That’s true but it misses the real reason.

The actual argument is about compounding. When you run SFT, you’re training on a fixed dataset examples produced by some teacher model, captured at one point in time. As the student model improves, that dataset doesn’t improve with it. Once your model gets close to the teacher’s level, you’ve hit the ceiling. More SFT is mostly just memorization at that point.

RL is structurally different. The model samples its own outputs, gets graded, and the gradient pushes it forward. Each improvement feeds back into the next batch. The ceiling isn’t the dataset anymore it’s the grader. And a well-designed grader can be much harder to max out than a static teacher.

So really the SFT → RL order is saying: “learn the easy stuff cheaply first, then switch to the mode that can actually compound past where your teacher was.” Makes sense as a resource allocation argument more than an algorithmic one.

One thing worth knowing: rejection-sampled SFT — where you filter outputs for correctness before training — is better than vanilla SFT, but it doesn’t escape the ceiling. Once the filter saturates, you’re stuck in the same trap. The curve shifts up, the shape stays the same.

Choosing a teacher — the axis nobody talks about enough

If you’re doing SFT, you need a teacher. And the relationship between your teacher and student model turns out to matter enormously in ways that aren’t obvious upfront.

Same-family teacher

Same vocabulary, similar training recipe. The signal is almost purely about capability gap the student is learning what the teacher knows, not how it talks.

Different-family teacher

Mismatched tokenizer, mismatched training style. A big slice of what the student learns is surface artifacts — how the teacher formats, hedges, and structures — not its actual reasoning.

Cross-family distillation carries a real hidden tax. Think about what happens when you try to learn from someone who not only knows more than you, but thinks in a completely different style. You end up half-learning their habits instead of their knowledge. Same thing here — the student absorbs the teacher’s formatting quirks and stylistic fingerprints alongside (and sometimes instead of) the actual capabilities.

So what does this mean practically?

If you’re evaluating vendors or open-weight models distilled from frontier APIs, this is worth probing. “Distilled from GPT-4” sounds good — but if the tokenizers don’t match and the training recipes diverge, a lot of that supposed capability transfer may be style transfer instead.

On-policy distillation, what happens when you combine both?

Once you lay out the previous two sections, on-policy distillation (OPD) feels almost inevitable. What if you could get RL’s self-generating, compounding sampling loop but instead of a sparse binary reward, each token in the rollout gets a rich, dense signal from a same-family teacher?

That’s exactly what OPD does. The student generates its own outputs. The teacher (same family, same vocab) assigns a per-token score: how much does it prefer this token over what the student would have predicted? Dense signal, on-policy rollouts, no hard ceiling from a frozen dataset.

The efficiency numbers reported are striking roughly 9–30× less compute than pure RL to reach comparable quality on hard benchmarks. Most of the saving comes from the fact that teacher forward passes over student tokens are essentially prefill, which is far cheaper than generation.

There’s a catch though, and it’s harder than it looks: OPD requires tokenizer match just to compute the loss. Without it, “how much does the teacher prefer this token” doesn’t have a well-defined answer at the per-position level. So OPD is only available in the same-family setting — which is not always where you are.

So the honest framing is: OPD gets you to the teacher’s level faster and cheaper than RL would. RL’s ceiling is higher it’s bounded only by the verifier, not by any teacher. But if the teacher is already very good, and you have limited compute, OPD is usually the move. The question is what to do when you don’t have a same-family teacher at all.

The self-distillation shortcut and where it breaks

No same-family teacher? Recent research has reached for an appealing workaround: use the model as its own teacher, but give the “teacher” version information the “student” version doesn’t have. Two main approaches have shipped:

• SDFT: Teacher version sees an expert demonstration of a similar task. Student generates without it. Teacher’s per-token preferences over the student’s output become the training signal.
• OPSD: Even more aggressive teacher is conditioned on the ground-truth answer. It knows where the rollout should end up. Student doesn’t.

Tokenizer problem: solved (same model). But here’s what I find genuinely interesting — and concerning — about these approaches. The privileged information doesn’t just help the teacher. It shifts the teacher’s distribution, sometimes dramatically. And that shift is where the failure mode lives.

Picture a long math proof. The student goes wrong because it fails to make one key observation — a pivot substitution that unlocks the rest of the solution. For the student, that token has maybe 1% probability. For the teacher, conditioned on the answer, it’s obvious — maybe 60% probability.

The per-token loss between “student says 1%” and “teacher says 60%” is huge roughly 100× larger than a typical well-agreed-upon token. The gradient doesn’t average out. It’s one enormous tug, in one direction, toward territory the model has barely visited.

The paper ships with KL clipping as a required defense. Without it, performance collapse happens within ~100 training steps. That’s not an edge case — that’s the method acknowledging its own instability in the default setting.

Compare this to RL, where most per-token gradients are noise that cancel out leaving only the small, consistent signal that actually correlates with reward. Or vanilla SFT, where the bias is real but spread diffusely across thousands of examples the model mostly half-understands already. Self-distillation with privileged hints is the uniquely risky combination:

The deeper issue is that the reasoning traces become incoherent for the student. The teacher is thinking with information the student can’t see. The traces refer, implicitly or explicitly, to a context that simply isn’t there. That’s not just a stability problem — it’s a reasoning quality problem that compounds over time.

What’s actually the right framing here and what comes next?

Here’s the way I’ve started thinking about all of this. Every method : SFT, RL, OPD, self-distillation is trying to answer the same underlying question: how do you move the model toward higher reward without destabilizing what it already knows?

There’s a Pareto tradeoff running through all of them. On one axis: how much reward improvement does the training signal produce per step? On the other: how aggressively does it shift the model’s distribution? The ideal teacher produces large reward improvement with minimal distributional disruption. RL sits at one end — nearly no distributional push, but honest and composable. OPD sits closer to the middle — large signal, bounded by teacher quality. OPSD overcorrects.

What This Actually Means

• RL → truth-seeking but slow
• SFT → safe baseline, bounded upside
• OPD → efficient transfer, controlled bias
• OPSD → aggressive optimization, fragile equilibrium

What if you could construct the optimal teacher dynamically, per task, rather than picking from a fixed menu? That seems like where the interesting work is heading.

A few directions are starting to look promising:

• Per-task prompt optimization — search for the minimal hint that maximizes reward gain while keeping the teacher’s distribution close to the student’s
• Training a separate hint-writer — a model that learns to give surgical nudges rather than large privileged-context dumps, trained on the reward-gain vs. KL tradeoff directly
• Online co-evolution — hint policy and student policy improve together, each adapting to the other in real time

The goal in all 3 cases is the same: a teacher that’s calibrated to the current student rather than fixed externally. High reward signal, low distributional disruption, no tokenizer dependency.

Pure RL probably still wins in the infinite-compute limit for the hardest problems — the long tail where any teacher adds more bias than it contributes. But for the large middle of real-world use cases, the gap between distillation efficiency and RL’s ceiling seems genuinely bridgeable. And bridging it cleanly, without requiring a pre-existing same-family teacher, is the most interesting open question in post-training right now.

The SFT → RL pipeline is not convention — it’s a compounding argument, and the compounding logic extends directly to which fine-tuning and distillation approaches you should trust. Methods that rely on same-family teachers tend to transfer cleanly. Methods that rely on self-distillation with privileged hints are promising but require careful stability engineering. The right question to ask any vendor pushing a distillation-based approach: what is the teacher, and how close is it to the student’s family?