Train Once, Reuse Everywhere: Rethinking Implicit In-Context Learning

An ICML 2026 paper argues that shift-vector methods are patching the wrong place — and proposes routing instead.

In-context learning (ICL) has become the workhorse for getting large models to adapt to new tasks without touching their parameters. But it isn’t free of problems. Packing examples into the prompt makes sequences longer and inference more expensive, and the whole thing is sensitive to the order and the formatting of those examples.

That’s the motivation behind “implicit ICL” — the idea of paying zero-shot costs while getting few-shot-level results. The mainstream recipe today is to compute a shift vector from a set of labeled examples and add it to the residual stream at some intermediate layer, simulating the latent state the model would be in if the examples were actually there.

This ICML 2026 paper argues that this whole direction has too low a ceiling. A fixed-size vector simply can’t hold much information, and the moment you switch tasks or switch models, you’re starting over. More fundamentally, the vector is added after the softmax — by the point where it takes effect, the model has already finished computing its attention. It’s essentially a patch applied after the fact.

Which is why these methods only really help on tasks where explicit ICL was already working well. Hand them a task with no labeled examples and they struggle.

The authors also noticed something stranger. If you mix in examples from different domains — sometimes ones with no relevance to the target task at all — performance sometimes beats both zero-shot and the best single-task few-shot setup. And sometimes it tanks. The implication is that ICL ability might be a latent mode that already lives inside the model: at least part of it doesn’t come from the content of the examples at all.

Their method is called In-Context Routing (ICR). The first step is to run explicit ICL across several domains and collect the Q and K projections of the last token of each prompt. Stack those, run PCA over them, and you get a set of layer-wise Principal ICL Directions (PID). What they find is that the structure shared across domains accumulates into the principal direction, while each domain’s idiosyncratic differences cancel each other out.

From there, a frozen lightweight model encodes the current query, which then passes through a lightweight dual-branch MLP router — and this router is the only trainable piece. It outputs a weighting over the PIDs plus gating weights for each attention head. The weighted PIDs get assembled into projection matrices that stand in for the original Q and K, and the result is added to the attention logits. Only the router is updated during training; the backbone stays frozen the whole way through.

This is what the title means by train once, reuse everywhere. You obtain the PIDs and the router a single time, and from then on you apply them zero-shot to new tasks. Across 12 datasets — 5 in-domain, 7 out-of-domain — ICR on Qwen2.5–7B beats the SOTA implicit baseline by 6.5% on average, and even edges out few-shot prompting by 2.7%. The vector-based methods collapse outright on some of the out-of-domain tasks; ICR holds its ground without dropping.

| Find papers faster on arXivSub with AI summary (CVPR/ICCV/ICML/ICLR/NeurIPS/AAAI/MICCAI)