Steering the KV Cache at Inference Time

🧠 KV Cache = a dictionary of Key & Value tensors, stored in GPU RAM during generation.

🎯 You can steer this cache using special steering vectors.

🤖 Ask GPT-4o to solve a reasoning problem step-by-step — this is your teacher trace.

📋 Create: A positive prompt (with GPT’s reasoning trace), A negative prompt (just the question + answer), And feed both to your Small Language Model (SLM).

🔍 Extract the Key and Value vectors (from the KV cache) for both prompts — especially at the final token. ➖ Compute the difference: K_diff = K_pos — K_neg, V_diff = V_pos — V_neg

🔁 Do this for ~500 prompt pairs, then average the differences → 🎁 This gives you: Sk_l = steering vector for Keys Sv_l = steering vector for Values

🛠️ Now modify the cached vectors like this: K_l = K_l + c_k Sk_l V_l = V_l + c_v Sv_l Where c_k and c_v are scalar strength values 💪

🎉 Ta-da! Your SLM now thinks more logically — 💡Reasoning unlocked at inference time (No fine-tuning, no retraining, just cute little cache magic ✨)

Paper- https://arxiv.org/pdf/2507.08799v1