Distilling a Teacher into a Student with Doc-to-LoRA Training

Large language models are mostly feed-forward weight. In a transformer, the attention layers do the routing, but the feed-forward layers hold the bulk of the parameters, and with them the bulk of the knowledge. That imbalance is an opportunity. If you could shrink the feed-forward part while keeping everything else, you could fit a much larger model’s behavior onto hardware that could never hold the original. This is the story of doing exactly that, and of arriving at a method that distills a large teacher into a small student by generating the student’s feed-forward weights on demand from a hypernetwork — a doc-to-LoRA style of training — rather than storing them.

Start with an observation: a non-gated feed-forward layer is already a mixture of experts in disguise. Each of its hidden neurons is a tiny, independent unit, a vector that reads the input and a vector that writes to the output. A layer with twelve thousand neurons is twelve thousand single-neuron experts. And on any given input, the model only really uses a small fraction of them. We measured this directly on a three-billion-parameter model: the effective rank of what a feed-forward layer actually computes on real text is low, a few hundred out of several thousand, and in the final layer just a handful. The weights are full-rank, but the work they do on real data is not. That redundancy is the room to compress.

The plan follows from it. Freeze everything worth keeping — the attention layers, the embeddings, the overall structure — and replace only the feed-forward layers with something much smaller, then train that smaller thing to reproduce what the original produced. Because we inherit the expensive, already-trained machinery and only relearn the compressed part, recovery is far cheaper than training a model from scratch.

The first real question was how to train the replacement. We compared three objectives at a matched token budget. The first was ordinary next-token prediction. The second added a term matching the student’s output distribution to the teacher’s. The third matched, layer by layer, each compressed feed-forward block’s output to what the original block produced on the same input, a feature-level distillation. The third won decisively, and by a wide margin. Matching the teacher’s internal computations recovered far more than matching only the final prediction, and the distribution-matching term actually made things worse. The lesson underneath the numbers is that when you compress, you damage the model’s precise stored knowledge first, the specific facts and the exact word completions, and the most direct way to protect that is to pin each layer to what it used to compute.

That gives a good student, but it still stores its experts. The more interesting move is to stop storing them and start generating them. Instead of a fixed bank of compressed neurons, a small network, the hypernetwork, reads the input and produces the feed-forward weights for it, as a low-rank adapter on top of a shared base. The experts never sit in memory as a table. They are synthesized as needed and discarded. The model’s footprint becomes the size of the generator, not the size of everything the generator can produce. This is the doc-to-LoRA idea: condition on the content, emit a small specialized adapter, apply it.

A natural worry is whether this is even trainable. You generate the weights, then you run the input through those generated weights, and the error has to travel back through both. It does, cleanly. The generated weights are simply activations in the middle of the computation, and the chain rule passes through them exactly as it passes through any intermediate value. Two differentiable steps compose into one. The only thing that breaks it is choosing experts by a hard, discrete selection, so you generate them smoothly instead of picking them.

With a generator and generated experts, a budget question appears. At a fixed size, would you rather have a larger generator making smaller experts, or a smaller generator making larger ones? We swept it. A larger hypernetwork making smaller experts won, monotonically. The reason is that a generated expert cannot carry more information than the thing that generated it. A large expert squeezed out of a small generator is mostly illusion. Capacity has to live in the generator. The familiar mixture-of-experts intuition, where big experts and a small router work well together, inverts the moment the experts are generated rather than stored.

Then came the result that mattered most. At convergence, generating the experts costs nothing. Trained to the same token budget, the hypernetwork that synthesizes its experts reached the same quality as the version that stored them outright, slightly better in fact. The earlier gap, where the generated model looked far behind, was entirely a matter of training length. Give it the tokens and it catches up exactly. So you can replace a large stored set of experts with a small generator and pay no quality tax, only the compute of running the generator, which you can amortize by generating once per sentence instead of once per token.

And it runs light. The compressed student, at a third of the teacher’s parameters, used less than half the memory and ran more than twice as fast in the throughput-bound regime, on a single consumer graphics card.

None of the individual pieces is unprecedented, and it is worth being honest about that. Shrinking a model’s feed-forward width and retraining to recover is structured pruning with distillation, a well-worn path. Matching internal features is classic layer-wise distillation. What is less explored is the combination aimed squarely at one end: turning a large model’s stored experts into a generator small enough to run a much bigger teacher’s behavior on hardware you already own, and doing the whole recipe on a single card cheaply enough that anyone can repeat it. The destination is a frontier-quality model compressed to a footprint that fits on a desktop, generated rather than stored.

That is the next step, now underway: applying the validated recipe to a real twenty-six-billion-parameter teacher, and measuring how small its generator can get while keeping what made it worth compressing in the first place.