What If the Experts Don’t Have to Exist?

HyperPEER: replacing a stored expert bank with a hypernetwork that writes the experts on demand. One day of experiments on a single consumer GPU, with all the numbers, including the ones that went wrong.

DeepMind’s PEER architecture answers a question every transformer asks at every token: which little piece of the network should handle this? Its answer is a lookup. PEER keeps a bank of up to a million tiny experts, each one just a pair of vectors, and uses a clever product-key index to retrieve a few hundred of them per token. The retrieved experts are assembled into a small MLP, applied, and discarded. It works, and it scales, and it rests on an assumption so natural that it is easy to miss: the experts have to be stored somewhere before they can be retrieved.

HyperPEER asks whether that assumption is necessary. Instead of retrieving experts from a bank, train a small hypernetwork that takes the token’s hidden state and generates the expert matrices directly. No bank, no index, no retrieval. The experts become a function of the input rather than entries in a table. A bank can only ever return what was stored in it. A generator might, in principle, synthesize sensible experts for inputs it has never seen.

The bet, in one sentence: the function PEER computes is much smaller than the bank PEER stores, and a generator can learn the function without carrying the table.

I ran the first full test of this idea in one day, on one RTX 5070 Ti. This article reports everything: the result, the instrument failures, the training divergence, and the experiment where the published warning failed to reproduce.

The parts list

None of the components are mine, and I want the lineage explicit. The teacher is a PEER language model I trained for an earlier project: six layers, d_model of 384, about 99.5 million parameters, with a bank of 16,384 experts in every layer and 256 experts retrieved per token, trained on TinyStories. The PEER implementation descends from lucidrains’ open-source PEER-pytorch, with my modifications. The method for generating adapters comes from Sakana AI’s Text-to-LoRA and Doc-to-LoRA papers: a small Perceiver-style hypernetwork that emits adapter weights, trained student-teacher by matching the base model’s output distribution. I had previously reimplemented Doc-to-LoRA from their paper and verified it works, which matters, because Sakana’s headline claims have not been independently replicated, and now at least the core mechanism has been, once, by me.

The synthesis is the only part that is mine: point the Doc-to-LoRA machinery at PEER’s expert bank. PEER experts are rank-one adapters, which makes them little LoRAs. If a hypernetwork can generate a LoRA from a document, maybe it can generate a token’s worth of experts from a hidden state.

The training recipe is logit distillation, not weight regression. The student is the same frozen language model with its expert retrieval replaced by generated experts. The loss is the KL divergence between the teacher’s next-token distribution and the student’s. We never ask the generator to reproduce the bank’s weights; we ask it to reproduce the bank’s behavior. Sakana’s own ablation, in an appendix I will come back to, found that matching the output distribution transfers more than matching tokens, and my results are consistent with that.

One scope note. The teacher uses one head with 16,384 experts per layer rather than PEER’s million-expert design point, because that is what trains overnight on one consumer GPU. Everything below should be read at that scale.

The morning: a generator barely trying

The first generator was deliberately small: 3.8 million parameters, a Perceiver trunk with eight latents, generating all 256 expert pairs for all six layers from each token’s hidden state. Distillation ran for 5,000 steps at batch size two. The whole run took 17 minutes.

Three numbers anchor everything that follows. The teacher, with real retrieval over its full bank, scores a validation perplexity of 10.2 on TinyStories. The same model with its expert layers deleted outright scores about 20.5. That spread is everything the expert bank contributes. After 17 minutes of distillation, the model running on generated experts scored 11.9.

In other words, a 3.8-million-parameter generator, trained for less time than it takes to eat lunch, recovered about three quarters of what the expert layers do, without storing a single expert.

The make-or-break check was generalization. If the generator only fits the inputs it trained on, it has memorized a lookup table and the premise fails. So every evaluation tracked the distillation loss on held-out data against the same loss on training data. The gap stayed at zero, within noise, for the entire run. I will be honest about why that test was easy to pass: 5,000 steps saw only about 2.5 million tokens out of a hundred-million-token corpus, with essentially no repeats, so memorization was nearly impossible. The gap staying at zero was necessary, not sufficient. But the generator was unmistakably doing real functional work on inputs it had never seen.

The question that produced the best evidence

At this point I asked a skeptical question: did we even train on all of the experts? I doubted it.

We measured it exactly. Because the training loop seeds its data sampler, we could replay the identical token stream through the teacher and count which expert IDs were ever retrieved. Over the full 2.56 million training tokens, the teacher never touched 62 percent of its expert bank. Per layer, somewhere between 5,300 and 8,500 of the 16,384 experts were ever used at all. And inside the used third, the skew was severe: about six percent of the bank accounted for half of all retrievals, and around a fifth of the bank covered ninety percent.

I had asked the question as an objection, and it is one: whatever we distilled is the TinyStories restriction of the teacher, not the full function, and nobody should claim otherwise. But notice what the measurement says about banks. The expert tables hold about 75 million of the model’s 99.5 million parameters, and on real data, half the layer’s behavior concentrates in six percent of them. The function PEER computes is dramatically lower-dimensional than the bank that stores it. That is precisely the gap a generator lives in. The skeptical question turned out to be the strongest argument for the whole idea.

The afternoon: scaling up, and a divergence

The obvious next move was a bigger generator: sixteen latents at width 512, about 17 million parameters, with a 20,000-step budget. The first attempt out-of-memoried a 16-gigabyte card, because a per-token Perceiver multiplies its activations by every token in the batch; the fix was to recompute the generator’s internals during the backward pass rather than storing them. The second attempt trained beautifully for 5,000 steps and then detonated: between one evaluation and the next, the distillation loss went from 0.10 to 2.6 and stayed there. The learning rate that a 3.8-million-parameter hypernetwork tolerated was too hot for a 17-million-parameter one, and gradient clipping slowed the explosion without preventing it.

I am including this because hypernetworks that generate weight matrices have a reputation for instability, the reputation is deserved, and pretending otherwise would make this report less useful. The recovery was mundane: a checkpoint from minutes before the spike, restarted at a third of the learning rate, ran the remaining 15,000 steps without incident. The only reason a good checkpoint existed is that the save cadence happened to outrun the divergence. Stamp your checkpoints.

The scaled run finished with the precise evaluation at 2.371 nats of cross-entropy against the teacher’s 2.323, a gap of 0.048 nats. To make that concrete, my earlier PEER project produced a frontier of fixed-retrieval models: retrieving 4 experts per token scores 2.367, 8 experts scores 2.339, 16 scores 2.323. The generated experts, after three hours of distillation, were statistically tied with real retrieval at k equals 4. The held-out gap stayed at zero through all 20,000 steps.

So by dinner: a 17-million-parameter generator stood in for a 75-million-parameter bank at a cost of about five percent perplexity, tied with genuine retrieval of four experts, with no sign of memorization.

The evening: the forgetting that wasn’t

The final experiment was the one I was most worried about. The plan for this system was always two-stage: distill the generator against the teacher, then switch to ordinary next-token training so the generated experts can become what the task wants rather than what the teacher had. I remembered a warning about exactly this switch, something I had read or heard from the Sakana orbit, to the effect that moving from distillation to next-token training causes catastrophic forgetting, with freezing the model and training a projection first as the fix.

Before testing it we chased down the provenance, and the trail ends somewhere real but different. Sakana’s Doc-to-LoRA paper, in its appendix on training objectives, reports that training the hypernetwork with next-token loss instead of KL distillation gives substantially worse results, with recall on their hardest generalization split dropping from 0.385 to 0.235. That is a finding about which objective to train with from scratch. It is not a finding about whether next-token training after distillation destroys what distillation built. As far as I can tell, nobody had run that experiment.

So we ran both arms. The naive arm took the distilled generator, unfroze everything, and trained on pure next-token loss with no protection whatsoever. The protected arm did the freeze-then-unfreeze procedure, which turns out to be the published LP-FT recipe: freeze the trunk, train only the output projections, then unfreeze with a small annealed anchor back to the teacher.

The forgetting did not happen. The naive arm showed a clear transition signature, with the KL to the teacher roughly doubling in the first 300 steps as the student drifted off the teacher’s exact distribution, but the actual task performance never degraded at all, let alone collapsed toward the no-expert baseline. It just improved, straight through the switch. And the two arms finished at endpoints identical to four decimal places: 2.3573 nats for the naive arm, 2.3572 for the protected one. The insurance cost nothing and bought nothing.

Next-token training improved both to 2.357, which moves the generated experts clearly past retrieval at k equals 4 and partway to k equals 8. It did not beat the teacher. Five thousand gentle steps closed about thirty percent of the remaining gap.

The honest caveat: this was next-token training on the same distribution as the distillation. The setting where I would still expect trouble is adaptation to genuinely new data, where the pull away from the distilled solution is much stronger. That arm is the natural next experiment, and if the forgetting shows up there, the LP-FT machinery is already built and tested.

Why I care, and why you might

The practical case is memory. A PEER-style bank must be resident to be retrievable; at the architecture’s intended scale that is a vast table held in RAM so that a few hundred entries can be gathered per token. A generator replaces residency with computation, and computation is the resource a local machine actually has spare, since consumer-GPU inference spends most of its time waiting on memory anyway. In this first test, 17 million generator parameters stood in for 75 million bank parameters at five percent perplexity cost, and the coverage measurement explains why that exchange rate is possible: most of the bank is air.

There is also a research-method point. Every result in this article was produced in one day on one consumer GPU, including the failure modes. The largest labs cannot cheaply run twenty experiments before lunch at their scale, so they don’t, and ideas like this go untried. I test on tiny corpora first; if an idea fails there, it is dead, and if it survives, it has earned a bigger rung. This idea has earned the bigger rung: WikiText next, which means retraining the teacher, and a task-shifted version of the forgetting experiment.

What is proven so far is exactly this, no more: at small scale, on a narrow corpus, a hypernetwork can learn to write a transformer’s experts on demand, generalizes to held-out inputs while doing it, matches modest real retrieval, survives the switch to task training, and improves from there. Whether generation can match retrieval at the scales where PEER actually shines is open. But the assumption that the experts have to exist before you can use them is now, at least at this scale, demonstrably optional.

Code, logs, checkpoints, and every number in this article: https://github.com/MikeyBeez/HyperPEER. The teacher comes from my earlier PEER adaptive-k project, the method from Sakana AI’s Text-to-LoRA and Doc-to-LoRA, the original architecture from Xu Owen He’s Mixture of A Million Experts at DeepMind, and the stability recipe from Kumar et al.’s LP-FT, which turned out, this time, to be insurance against a fire that never started.

Appendix: HyperPEER — Configurations, Numbers, and Reproduction Notes

Companion to “What If the Experts Don’t Have to Exist?” Everything here comes from the logs and metrics files in the repository at https://github.com/MikeyBeez/HyperPEER. All experiments ran on June 10, 2026, on a single RTX 5070 Ti with 16 GB of VRAM, under Pop!_OS 24.04 with PyTorch 2.11.0 and CUDA 12.8. Weights and Biases runs live in the project mikeybee/hyperpeer.

A1. The teacher

The teacher is the fixed-k 256 checkpoint from my earlier peer-adaptive-k project (checkpoint p0_matched_k256.pt). It is a decoder-only transformer with six layers, model width 384, eight attention heads, RoPE positions, pre-norm RMSNorm, and a PEER layer as the feed-forward block in every layer. Each PEER layer holds 16,384 single-neuron experts, stored as two embedding tables (a down-projection vector and an up-projection vector per expert, each of width 384), addressed by product keys with 128 keys per axis. Every token retrieves exactly 256 candidate experts per layer via the product-key cartesian top-k, and the expert outputs are combined with softmax weights renormalized over the retrieved set. The vocabulary is GPT-2 BPE, 50,257 tokens, with weight tying between the embedding and the LM head. Total parameters: 99,519,744, of which the expert tables hold about 75.5 million (16,384 experts times 384 dimensions times two vectors times six layers).

Training data is TinyStories: 500,000 stories for training and 20,000 for validation, packed into uint16 memmaps, a bit over 100 million training tokens, context length 512. The teacher was trained with plain language-model cross-entropy, not distillation.

The retrieval frontier referenced throughout the article comes from sibling checkpoints of the same model trained at fixed k of 4, 8, 16, 32, 64, 128, and 256. Their final validation losses, in nats, are: k of 4 scores 2.3673, k of 8 scores 2.3387, k of 16 scores 2.3233, k of 32 scores 2.3165, k of 64 scores 2.3076, k of 128 scores 2.3052, and k of 256 scores 2.3114. Note the frontier is not monotonic at the top end; k of 256 lands slightly above k of 128. The k-256 model is the teacher everywhere in this work, and on the 50-batch evaluation protocol described in A4 it measures 2.3227.

A2. The capture harness

The file src/harness.py wraps the teacher for distillation. Forward hooks on each block’s FFN capture the pre-norm hidden state entering the PEER layer; the collect path of the PEER forward returns retrieval scores, expert IDs, and the gate mask, from which the harness recomputes the exact renormalized gate weights the teacher used. The down and up expert matrices are never stored; they are gathered on the fly from the teacher’s embedding tables by expert ID. The harness includes a verification routine that reconstructs each layer’s FFN output from the captured pieces (apply the teacher’s own RMSNorm, dot with the gathered down vectors, GELU, scale by gate weights, sum against the up vectors) and compares it to the hooked output. On the k-256 teacher this reconstruction is exact, with zero maximum absolute error on all six layers, which establishes that the captured pairs are precisely the function the generator is asked to learn.

A3. The generator

The generator (src/generator.py) follows the Perceiver hypernetwork from Sakana’s Doc-to-LoRA reference design, adapted from per-document to per-token conditioning. Each token’s normalized hidden state forms a one-element key-value set. A bank of learned latents cross-attends to it, passes through self-attention blocks, and a head pools the latents with one query per generated expert row, offset by a layer embedding so a single generator serves all six layers. Two linear projections emit the down row and the up row of each expert. The up projection is initialized to zero, Doc-to-LoRA’s stability trick, so generated experts begin as an exact no-op and the student starts from the model-with-FFNs-disabled baseline. The teacher’s per-token gate weights are folded into the up rows during training, so the generator’s target is a clean pair of 256-by-384 matrices per token per layer, and the student’s FFN is simply: GELU of the down products, summed against the up rows.

The small generator uses eight latents at width 256, two cross blocks and two self blocks: 3.8 million parameters. The scaled generator uses sixteen latents at width 512, two cross blocks and three self blocks: 17.1 million parameters. (An earlier draft of the article said 11 million; the training log’s count of 17.09 million trainable parameters is the correct figure, and the bank-to-generator ratio is therefore about 4.4 to one, not 7 to one.)

The student is the frozen teacher with each FFN wrapped so it can run in either mode: teacher mode executes the original PEER retrieval; generated mode calls the generator on the layer’s normalized input. This mirrors Doc-to-LoRA’s install-and-clear adapter injection. Wrapping only the generator call in PyTorch activation checkpointing (recompute in backward, non-reentrant) is what makes the 17-million-parameter version fit in 16 GB; checkpointing the whole transformer block fails with a saved-tensor-count mismatch, presumably interacting with the capture hooks.

A4. Distillation runs and evaluation protocol

The loss everywhere is KL divergence from the teacher’s next-token distribution to the student’s, computed over the full vocabulary at every token position, temperature 1. The optimizer is AdamW with betas 0.9 and 0.95, gradient clipping at 0.5, linear warmup then cosine decay, batch size 2 at context length 256. Non-finite losses skip the step, with a hard stop after 200 consecutive skips; no run ever skipped a step.

The small run (W&B stage1_distill_k256): 5,000 steps at peak learning rate 1e-4, warmup 200. Wall time about 17 minutes at 4.9 steps per second.

The scaled run, first attempt (stage1_distill_k256_big): same learning rate, 20,000-step schedule. Healthy through step 5,000 with held-out KL at 0.098 by step 4,750, then diverged between steps 5,000 and 5,250: held-out KL jumped from 0.10 to 2.6 and per-step KL pinned near 4, far above the 1.1 of a zero-initialized student. The run was killed and resumed (stage1_distill_k256_big2) from the step-5,000 checkpoint, which had been saved minutes before the next scheduled save would have overwritten it with the diverged state. The resume ran 15,000 further steps at peak learning rate 3e-5 and finished in 2 hours 56 minutes at 1.4 steps per second, slowed by the checkpoint recomputation. Held-out KL ended at about 0.040 and was still slowly declining.

Generalization tracking: every 250 steps, both runs evaluated the distillation KL on eight held-out validation batches and eight fresh training batches. The val-minus-train gap stayed within plus or minus 0.02 nats around zero for the entire small run and all 20,000 steps of the scaled run, with no trend.

The precise evaluation protocol behind every headline cross-entropy number (experiments/eval_student.py): 50 fixed-seed validation batches of 4 sequences at context length 512, teacher and student scored on identical batches. Results: teacher 2.3227 nats, perplexity 10.20. No-FFN baseline, approximately 3.0 nats, perplexity near 20.5. Small generator 2.4801, perplexity 11.94, a gap of 0.157 nats. Scaled generator 2.3710, perplexity 10.71, a gap of 0.048 nats. Note the distillation context was 256 but evaluation is at 512; the per-token generator transfers across context lengths without adjustment.

A5. Expert coverage replay

Because the distillation loop seeds its sampler, the exact training token stream is replayable. The script experiments/expert_coverage.py re-ran the identical 5,000 batches (2.56 million tokens) through the teacher with the collect path on and counted retrieved expert IDs per layer. Experts ever retrieved, by layer from bottom to top: 5,388, then 5,341, then 5,839, then 5,699, then 6,809, then 8,493, out of 16,384 each, which is 38.2 percent of the overall bank; 61.8 percent of experts were never retrieved once. The coverage curve flattens quickly: 32.8 percent of the bank after the first 50,000 tokens, 34.3 percent after a quarter million, 35.1 percent after half a million, 36.6 percent after 1.28 million, 38.2 percent after the full 2.56 million. Within the used set, the number of experts needed to cover half of all retrievals, per layer: 856, 970, 1,278, 937, 1,096, 1,200, which is roughly six percent of each bank; covering ninety percent takes between 2,287 and 3,605 experts per layer, between 14 and 22 percent. The replay took 212 seconds.

A6. The stage-two switch experiment

Both arms start from the scaled distilled checkpoint and train 5,000 steps of next-token cross-entropy on the same TinyStories stream, batch 2, context 256, with evaluation every 100 steps on eight held-out batches reporting student cross-entropy, teacher cross-entropy on the same batches, and the KL between them.

The naive arm unfreezes the entire generator immediately and trains at learning rate 2e-5 with no anchor and no probe phase. The LP-FT arm first freezes the Perceiver trunk and trains only the head’s two output projections (0.20 million parameters) for 500 steps at 1e-4, then unfreezes everything at 2e-5 with a KL-to-teacher anchor weighted 0.5 and annealed linearly to zero over the remaining steps.

Pre-switch baseline on the eval protocol: student 2.3523, KL to teacher 0.0387. The naive arm’s KL to teacher rose to a peak of about 0.074 within the first 300 steps and settled near 0.055; the per-100-step cross-entropy evaluations never showed degradation relative to the matched teacher numbers at any point in either arm, and no excursion toward the no-FFN baseline occurred. Final precise evaluations on the 50-batch protocol: naive arm 2.3573 nats, perplexity 10.563; LP-FT arm 2.3572 nats, perplexity 10.561. The arms are identical to within a tenth of a millinats, and both improve on the distilled 2.3710 by about 0.014 nats, landing between the k-4 and k-8 points of the retrieval frontier.

The forgetting claim this tests, precisely: Sakana’s Doc-to-LoRA paper (arXiv 2602.15902, appendix on training-objective ablation) reports that a hypernetwork trained from scratch with next-token loss underperforms one trained with KL distillation, 0.763 versus 0.819 normalized F1 on SQuAD and 0.235 versus 0.385 recall on their swapped split. That result concerns the choice of from-scratch objective. The sequential question, whether next-token training after distillation destroys the distilled solution, is what the two arms here address, and at this scale, in distribution, the answer is no. The untested case remains next-token training on a shifted distribution, where the pull away from the distilled solution should be strongest.

A7. Reproduction

Clone the repository and its sibling peer-adaptive-k, whose checkpoints directory must contain p0_matched_k256.pt and whose data directory must contain the TinyStories memmaps (its own README covers data preparation). The harness smoke test is python -m src.harness, which loads the teacher, captures a batch, verifies exact FFN reconstruction, and prints a validation loss. The small distillation is python -m experiments.distill_stage1 — steps 5000. The scaled run adds — latent-n 16 — latent-d 512 — n-self 3 — lr 3e-5. Coverage replay is python -m experiments.expert_coverage, the precise evaluation is python -m experiments.eval_student — ckpt with the checkpoint path, and the stage-two arms are python -m experiments.ntp_stage2 with — probe-steps 0 — kl-anchor 0 for the naive arm and — probe-steps 500 — kl-anchor 0.5 for the protected one. Every run writes a metrics JSONL into results, and the figures in the article come from those files unmodified.

Peak GPU memory was about 7 GB for the scaled distillation with generator-only activation checkpointing; the same configuration without it does not fit in 16 GB. If a scaled run diverges, resume from the last good checkpoint at one-third the learning rate via — init-from; this recovered cleanly here, with the loss returning immediately to its pre-divergence value.