Reduce AI/LLM cost using Semantic Caching

Semantic Caching — Reduce AI/LLM Cost

Hi folks, today I am going to share one of the most effective ways through which you can reduce your LLM cost drastically depending on how you use it. And this method is known as Semantic Caching.

Before you understand semantic caching, you need to understand how normal and typical caching works.

What is Caching

Imagine that there is a restaurant, where customers give orders of different cuisine. but there is one specific cuisine that gets ordered more often. Now every time going to chef and demanding to cook it from scratch is kind of a process. If we know that there is this dish that is being ordered more often. we would cook that in advance and keep it close to the counter. And now customers don’t have to wait for that specific dish. as it’s already there near the counter ready to be served, this thing is called caching.

In developer language, what happens every time when a user makes a request we process that request and check our database (hardware storage-slow) and then do some processing and send it back to the user, this whole thing takes time, depending on type of operation. So caching is a mechanism that allows you to store that data in caching db (memory like storage), these are very fast as compared to a typical hardware style storage database.

So the first time the request will hit and get processed how it usually happens but from next time when the user will try to access the same data they will be able to retrieve it 90% faster. because instead of going back to the database it does quick check if this data exists in our cache.

The normal caching basically stores the data in the key value pair means: You have this specific order with this specific OrderID that is being retrieved more often — so here we basically cache this data where we know OrderID will always be constant.

{orderId: <value_of_data>}

Semantic Caching

Now when we use LLM queries then this specific format doesn’t work well and Let me tell you why.

It happens because our keys will never be the same, because the user asks questions in Natural Language, e.g:

1. One user might say “What is refund policy”
2. Second user might say: “How to get my refund”

Now if you would closely at this then you can see — the intent of both statements are similar but words are not. These types of queries we can’t cache using normal caching.

So we perform semantic caching here — In short, semantic caching is the way to convert these into vector embedding and storing it then, so the next when the user comes and tries to lookup for something then instead of matching word by word it does vector similarity search to find whats the closest query that was being asked, and if it’s present then it’s return that response directly.

How does it reduce cost?

Let’s run it through simulation where we have LLM + RAG pipeline, what happens when user asks question.

Without semantic caching:

when user asks a question(prompt), we take that then we convert it into embedding(using OpenAI embedding) then we try to find relevant context from our vector database — and whatever the context we get in return we use that to generate an answer and then we will send it back to user. If you see we did these many actions.

• Vector DB Lookup: To retrieve matching context.
• Postgres/Database Lookup(optional): but if you are storing your text of context into this(good habit).
• LLM Generation: using context + prompt you generate response.

This entire process would cost you time(because looking through searches and then generating LLM response itself takes time) and money (obviously you are generating response using LLM like openai or whatever that costs money).

• And now next time when user comes and asks a question with same intent but in different tone then what you do? — you do all that from scratch.

With semantic caching:

• User asks you question, if it’s first time you will go through same process but before you return response. -> you will take user prompt and convert that into embedding and store it with generated LLM response.
• Now next time when user comes -> you convert that prompt into embedding -> then match it inside your semantic cached db, if there is match that is close enough(like that refund policy example), then skipping all the process — you could directly take that response and send it back to user. (10 seconds to 50ms). You saved your time and cost both.

Things to Keep in Mind:

• Semantic cache is useful and effective only if you know there are keywords that are being searched and its global like all the users are referring same db, if it’s individual and isolated queries then it won’t help much. (Refund policy was an example that those kind of question are being asked more often). But imagine user asking about himself on his document, you wouldn’t like that he gets someone else’s cached information.
• If you wanna implement semantic caching where you have different users to serve make sure add separation using userId meta tag or something, so that you can filter while doing match.
• Tune your simiarity threshold: what it means, when you do similarity search for semantic caching depending on embedding model you used, you might have to tune the number, for example if closeness score reaches 0.95 only then consider, if its less reject that. It’s important because else you would endup returning wrong and useless response.
• Cache Invalidation: It can be that your company’s refund policy got changed but your cache still has old data, when you have normal caching then you know this is exact keyword that I need and I will delete that and it will get solved. But here things get trickier so you have to manage it smartly, there are different mechanism like TTL, Event based invalidation(for that you gotta tag your cache with specific id and link it with source doc), etc.

So, I hope you understand how this works.