Making Native Semantic Search Cheaper to Run in ODC

A Question After the Talk

Someone asked me a good question after my talk in Amsterdam: does the semantic search have any caching built in?

It didn’t. Every search hit the embedding API. The same query a hundred times a day meant a hundred API calls.

That one stayed with me on the flight home. The RAG Knowledge Base Forge component proved native vector search works in ODC. Proving it works and being worth running in production are two different things, and the question was pointing straight at the gap.

So I added a cache. Here is how it works and why each part is built the way it is.

What We Cache

There are two things worth caching in a semantic search pipeline: the query vector and the search result. The query vector is the embedding generated from the query text. The search result is the ranked list of matching documents.

SearchVectorWithCache caches both, controlled by a single parameter: IsCacheResult (Boolean, default False).

The query vector is always cached. On a repeat query, the action skips the embedding API call entirely and uses the stored vector to run cosine similarity against the current document set. Fresh results, no API cost.

When IsCacheResult is True, the search result is also cached. On a repeat query with a result hit, the action skips both the embedding call and the vector scan. It deserialises the cached result and returns it directly.

The developer controls which mode to use per call. IsCacheResult = False for fresh results without the embedding cost. IsCacheResult = True for maximum speed when result freshness is not critical. SearchVector directly when you want no caching at all.

The Cache Entity

QueryEmbeddingCache has nine attributes.

Id (Identifier, AutoNumber). Primary key.

QueryHash (Text 64). SHA-256 hash of the query text. This is the lookup key. Hashing keeps the index tight and avoids comparing long query strings on every read.

QueryText (Text 2000). The original query, stored for audit. When you want to see what is actually being cached, it’s here.

CachedQueryVector (Text 50000). The query embedding as a serialised JSON float array. Always stored on every run, regardless of IsCacheResult.

CachedResult (Text 100000). The full serialised search result. Only stored when IsCacheResult is True. Blank otherwise.

EmbeddingModel (Text 100). The model used when the entry was created. Swap models later and this field drives invalidation. You do not want results from text-embedding-3-small quietly mixing with results from something else.

CacheMaxAgeMinutes (Integer). The per-record expiry age. Not a global setting. Each entry knows its own TTL.

CreatedOn (DateTime). When the entry was written. Used by the purge timer.

HitCount (Integer, default 0). Incremented on every hit. Handy for spotting which queries are worth optimising elsewhere.

The design follows the same pattern as DocumentChunk in the original component. Named, typed, one clear job per attribute. The separation between CachedQueryVector and CachedResult is intentional. The vector is always worth keeping. The result is optional.

QueryEmbeddingCache entity in ODC Studio

SearchVectorWithCache

SearchVectorWithCache wraps the search with a cache lookup. Same inputs as a plain search, plus two: CacheMaxAgeMinutes (Integer) and IsCacheResult (Boolean, default False).

When a query comes in, the action lowercases the query text, hashes it, and checks QueryEmbeddingCache for a matching record with the same hash and embedding model. Three paths from here.

If a match is found and CachedResult has data, the cached result is deserialised and returned. No embedding call. No vector scan.

If a match is found but CachedResult is blank, the stored query vector is used to run cosine similarity against the current document set. No embedding call. Fresh results.

If no match is found, the action embeds the query, runs the search, and writes the record. The query vector is always stored. The result is stored only when IsCacheResult is True.

Expired records are not checked on read. The purge timer handles deletion. Until a record is deleted, it remains a valid cache hit.

SearchVectorWithCache flow in ODC Studio

CacheMaxAgeMinutes can be set per call. Pass a value above zero and that value wins. Pass zero and the action falls back to the DefaultCacheMaxAgeMinutes site property, which defaults to 1440 minutes (24 hours). Either way, the resolved value is stored on the record itself, so each entry manages its own expiry. Changing the site property later has no effect on records already written.

On every path, HitCount is incremented when a record is found. The record is always upserted. Never duplicated.

SearchVectorWithCache parameters in ODC Studio

HashText in C

ODC has no built-in SHA-256 function.

So I added HashText to SemanticEngineV2Service in C# External Logic. Plain text in, SHA-256 hex string out. It follows the existing layer pattern: ISemanticEngineV2 exposes the contract, SemanticEngineV2Facade delegates, SemanticEngineV2Service implements.

This is the same path ChunkHash already takes in the DocumentChunk entity. The component had a hashing pattern. HashText extends it.

I could have stitched something together in ODC with text manipulation and HMAC logic. I didn’t. The External Logic layer was cleaner and easier to test.

Three Cache Management Actions

There are three: ClearCache, ClearCacheByQuery, and ClearCacheByHash.

ClearCache deletes everything in QueryEmbeddingCache. An admin action. You reach for it to force a full rebuild after a big ingestion batch or a model change.

ClearCacheByQuery takes the query text, hashes it internally, and deletes the matching record. For developers who know which query went stale. You pass the string the way a user would type it.

ClearCacheByHash takes a pre-computed QueryHash and deletes the record directly. For programmatic callers who already have the hash and don’t need to compute it twice.

Cache management actions in the RAG folder

The Purge Timer

CleanupQueryEmbeddingCache timer deletes records that have outlived their expiry.

Expiry filter in CleanupQueryEmbeddingCache

Each record’s own CacheMaxAgeMinutes drives this, not a global setting. A 10-minute record expires in 10 minutes. A 1440-minute record expires in 24 hours. They live in the same table without stepping on each other.

Here is something I found while setting this up. The ODC console timer UI has a minimum schedule interval of 5 minutes. If you want the purge to run more often, the normal workflow won’t take you there.

For production, I’d treat that 5-minute floor as the safe default. In my test environment, I asked Mentor to schedule the timer as frequently as possible. It set ScheduleConfiguration = "* * * * *" directly, and the timer ran every minute. You can see it active in the console with the * * * * * schedule, last run and next run one minute apart.

Useful for validation. Beyond that, I’d reach for it carefully and on purpose.

CleanupQueryEmbeddingCache timer in the ODC console

When to Use Cache, When to Go Direct

Use SearchVectorWithCache with IsCacheResult = False for most queries. You skip the embedding call on repeats and always get fresh results. This is the safe default.

Use SearchVectorWithCache with IsCacheResult = True when the knowledge base is stable and speed matters more than freshness. Support FAQs, internal search tools, anything where the documents don't change between requests and the same questions repeat.

Use SearchVector directly when you want no caching at all. Right after ingesting new documents. In a verification flow. In admin tooling where accuracy is the whole point.

ClearCacheByQuery is the bridge. When a document behind a known query changes, invalidate that one entry instead of nuking the whole table.

The Tradeoffs You Should Know

With IsCacheResult = True, the cache trades freshness for speed. New documents won't appear in cached results until the entry expires or gets cleared. If your knowledge base changes constantly, use IsCacheResult = False instead. You still skip the embedding call, and results are always current.

The cache key is a hash of the query text, lowercased first so case differences still match. Punctuation and wording don’t. “Reset password steps” misses “How do I reset my password?” even though they mean the same thing. Semantic caching is a known pattern in RAG pipelines: match incoming queries against cached ones by vector similarity rather than exact hash, and return the cached result if they’re close enough. Bringing that natively into ODC without an external vector store is the next step. Not in this version.

CachedResult is capped at 100000 characters. A normal topK fits easily. If a result set overflows that limit, the search still runs and returns normally, but the result isn’t stored. You get a message saying so. The query vector is still cached, so the next repeat skips the embedding call regardless.

HitCount is a counter, not an analytics suite. It tells you how often a query hit the cache. It does not tell you whether the answer was any good.

The timer runs every minute on the schedule Mentor set, so records expire close to their TTL. Close, not exact. A 10-minute record gets purged on the next cycle, not on the dot.

QueryEmbeddingCache holds two large Text fields: CachedQueryVector at 50000 characters and CachedResult at 100000 characters. You could move them to a child entity and fetch them only on a confirmed hit. Aurora PostgreSQL stores large text out-of-line via TOAST. Keep the lookup Aggregate scoped to QueryHash and EmbeddingModel, and the large fields stay out of the way. At this scale, one entity is simpler and the difference is not measurable.

QueryEmbeddingCache grows with every unique query. In a high-volume deployment with a wide query spread, the table gets large. That’s what ClearCache is for.

Why It’s Worth Adding

Native vector search in ODC works. This layer makes it cheap enough to leave running.

If 100 users ask the same support question in a day, only the first one pays the embedding cost. The other 99 get their answer from a single row lookup. That is the difference between a demo and a system you’re happy to see on the bill.

Proving it works was step one. Making it worth keeping is step two.

If you’re building on the RAG Knowledge Base, the latest version on Forge already includes all of this.

Are you caching semantic search results in ODC? Or solving the freshness problem some other way? I’d like to compare notes in the comments.