KV Cache Diet Plans 🍽️ for Hungry Transformers

KV Cache is great. It speeds things up by not recomputing keys and values again and again. But it comes with a major downside: it eats up a…