You Pay OpenAI Five Times for One Answer

Your users ask the same question five ways. Semantic caching answers four of them for free.

Your users ask the same question five different ways. You pay the provider five times. Semantic caching answers four of them for free.

I found it in the token dashboard, not the code. Same answer, billed sixty times in an hour.

The questions were not identical. How do I cancel my plan. Where is the cancel button. Can I stop my subscription. Four strings, four cache misses, four full calls to the model, four times the output tokens. Our exact-match cache saw four different keys and shrugged. The user saw the same answer every time. We paid for all four.

That is the gap semantic caching for LLM responses in Rails closes. An exact-match cache keys on the bytes of the question. A semantic cache keys on the meaning. When the meaning has been answered before, you return the stored answer and never call the provider at all. No output tokens. No latency. The four rephrasings collapse into one paid call and three free ones.

1. Add pgvector to Postgres (enable_extension "vector") and the neighbor gem to your Gemfile. It is the ActiveRecord bridge to pgvector's similarity search.
2. Create an llm_cache_entries table with an embedding vector column, a question_digest string with a unique index, and the stored answer.
3. On every question, check the exact digest, then the semantic neighbor within a distance threshold, and only call the model on a true miss.

That is the whole shape. The rest of this piece is the two parts that bite. The write path that fills the cache. The read path that decides a hit, where one inverted number turns your cache into a machine that returns the wrong answer confidently.

Why OpenAI’s Own Cache Does Not Save You Here

You might think OpenAI’s prompt caching already handles this. It does not, and the reason is worth being precise about. Prompt caching discounts token prefixes. It matches when the leading tokens of your prompt are identical to a recent call, it requires prompts of 1024 tokens or longer, and the discount applies only to cached input tokens.

The user’s rephrased question sits at the end of the prompt, in the suffix. That is exactly the part prompt caching does not match on. And even on a hit, the output tokens still bill at the full rate.

Run the arithmetic at current pricing (July 2026, gpt-5.5 standard tier, short context). Input is $5.00 per million tokens, cached input is $0.50, output is $30.00. Prompt caching, at its best, shaves the input side of a repeat call. Semantic caching removes the entire call, input and output both, for every rephrasing after the first.

• Prompt caching discounts input-token prefixes. It never matches a reworded question and never discounts output.
• Exact-match caching keys on bytes, so four phrasings of one intent are four misses.
• Semantic caching keys on meaning, so the second through fifth phrasings cost zero provider tokens.

The Write Path: Embed the Question, Store the Answer

Here is the flow that fills the cache. It runs once per genuinely new question, on a miss.

new question ──> embed (text-embedding-3-small) ──> call the model
                                                          │
                                                          ▼
                              store {digest, question, answer, embedding} row

The migration is where you commit to a metric. Cosine distance, all the way through, because embeddings are compared by angle. The text-embedding-3-small model returns 1536 dimensions, which sits under pgvector's 2,000-dimension index ceiling.

class CreateLlmCacheEntries < ActiveRecord::Migration[8.0]
  def change
    enable_extension "vector" unless extension_enabled?("vector")
​
​
    create_table :llm_cache_entries do |t|
      t.string   :question_digest, null: false
      t.text     :question,        null: false
      t.text     :answer,          null: false
      t.string   :model,           null: false
      t.vector   :embedding, limit: 1536
      t.integer  :hit_count, null: false, default: 0
      t.datetime :last_hit_at
      t.datetime :expires_at
      t.timestamps
    end
​
​
    add_index :llm_cache_entries, :question_digest, unique: true
    add_index :llm_cache_entries, :embedding,
              using: :hnsw, opclass: :vector_cosine_ops
  end
end

The unique index on question_digest is not decoration. It is what lets you handle the concurrent-miss race later, when two requests for the same new question both try to insert. And the :vector_cosine_ops opclass on the HNSW index has to match the cosine distance you query with later. Mismatch it and Postgres silently ignores the index, so every lookup becomes a sequential scan.

The model declares the vector column with has_neighbors, from the neighbor gem (current stable at the time of writing, check RubyGems for the version you install). One line wires pgvector similarity search onto the model.

class LlmCacheEntry < ApplicationRecord
  has_neighbors :embedding
​
​
  scope :live, -> { where("expires_at IS NULL OR expires_at > ?", Time.current) }
end

Embedding the question is one call. I use RubyLLM.embed because it keeps the provider swappable, but a raw OpenAI client works the same way. The return is a vector of floats.

# RubyLLM.embed returns a result; .vectors is the Array<Float> for a single input.
result = RubyLLM.embed(question, model: "text-embedding-3-small")
embedding = result.vectors

• The embedding column limit (1536) must equal the dimension count the model returns, or inserts fail on a size mismatch.
• The HNSW index opclass must be:vector_cosine_ops to match cosine queries, or the index is dead weight.
• text-embedding-3-large returns 3072 dimensions, over pgvector's 2,000-dim vector index ceiling. It needs a halfvec index, so do not swap it in without changing the column.

The Read Path: One Inverted Number Away From Disaster

This is the part that broke first when I built it. The read path runs on every question, hit or miss. Exact digest first, because it is free. Then the semantic neighbor. Only then the model.

question ──> digest hit? ──> return stored answer
                │ no
                ▼
          semantic neighbor within threshold? ──> return stored answer
                │ no
                ▼
          call the model ──> (write path stores it)

The trap is the threshold. In neighbor, nearest_neighbors(:embedding, vector, distance: "cosine") orders by cosine distance, and the threshold: keyword is a distance ceiling, the maximum distance a row may have and still be returned. Lower distance means more similar.

Use the gem’s own threshold: keyword to apply that ceiling. Do not hand-roll it with .where("neighbor_distance <= ?"). neighbor_distance is an alias the gem adds to the SELECT list, and Postgres evaluates WHERE before SELECT, so a WHERE that references the alias either errors on an unknown column or silently filters nothing. The keyword pushes the bound into the query correctly.

Cosine similarity s and cosine distance d relate as d = 1 - s. So a return a hit only if the questions are at least 95% similar rule is not threshold: 0.95. It is threshold: 0.05.

Write threshold: 0.95 here and you have built a machine that treats nearly every question as a hit. How do I cancel matches how do I upgrade matches what is your refund policy, and every user gets the first answer that happened to be cached. That is worse than no cache. It is a confident wrong answer at scale.

class SemanticCache
  # 0.05 cosine distance == 0.95 cosine similarity. Lower is more similar.
  SIMILARITY_DISTANCE_CEILING = 0.05
​
​
  def fetch(question, model:)
    digest = Digest::SHA256.hexdigest(question.strip.downcase)
​
​
    if (exact = LlmCacheEntry.live.find_by(question_digest: digest))
      return record_hit(exact)
    end
​
​
    embedding = RubyLLM.embed(question, model: "text-embedding-3-small").vectors
​
​
    neighbor = LlmCacheEntry.live
      .nearest_neighbors(:embedding, embedding,
                         distance: "cosine",
                         threshold: SIMILARITY_DISTANCE_CEILING)
      .first
​
​
    return record_hit(neighbor) if neighbor
​
​
    answer = yield # caller makes the provider call on a true miss
    store(digest:, question:, answer:, embedding:, model:)
    answer
  end
​
​
  private
​
​
  def record_hit(entry)
    entry.update_columns(hit_count: entry.hit_count + 1, last_hit_at: Time.current)
    entry.answer
  end
end

The .live scope is doing quite work in both branches. A cache entry that has expired must not be returned as a hit, so the TTL check lives in the lookup scope, not in a column you forgot to read. An answer that was correct in March is a liability in July if your product changed underneath it.

• The threshold is a cosine distance ceiling, not a similarity floor. A 0.05 distance is 0.95 similarity, and inverting it silently returns garbage.
• Run the exact-digest check before embedding. The digest lookup is free, and the embed call is not.
• Enforce TTL in the query scope (.live), not as an afterthought. A stale answer returned as a hit is a support ticket.

The Failure Mode Nobody Warns You About: Negation

Here is the one that bit me after the threshold was already correct. Embeddings measure topical similarity, and they are bad at negation. How do I cancel my subscription and how do I stop my subscription from being canceled sit close together in cosine space, because they share almost every token and the same topic, even though the correct answers are opposites.

A tight distance ceiling does not save you. These two questions can land inside 0.05 of each other, so the cache returns the cancel instructions to the user who was trying to avoid cancellation. That is not a rare edge. Negation, not, without, instead of, except, is exactly where a support cache turns a confident wrong answer into a real ticket.

I do not have a clean fix, and I distrust anyone who claims one. What I do now is keep a short deny-list of intents where a wrong hit is expensive (billing, cancellation, data deletion) and route those past the semantic layer to a fresh call, exact-match only. The cache still absorbs the long tail of harmless how-to phrasings, which is where the volume is anyway.

And the 0.05 is not gospel. I did not derive it, I tuned it. Before you ship, build a small labeled set, a few dozen pairs hand-marked same-intent or different-intent, and sweep the ceiling until it separates them. Your model, your domain, and your users' phrasing all move the number. A threshold copied from an article is one you have not tested on your own traffic.

• Cosine similarity collapses negation. Antonym questions sit close in embedding space, so a distance ceiling alone will not separate cancel from do not cancel.
• Route high-cost intents (billing, deletion, cancellation) past the semantic layer. Serve those from exact-match or a fresh call.
• Pick the distance ceiling with a labeled same-intent/different-intent set, not by copying a number. The right value is model- and domain-specific.

Storing the Answer Without a Race

On a true miss, two requests for the same new question can arrive at once. Both miss, both call the model, both insert. The unique index turns the second insert into a RecordNotUnique you catch and resolve to the row the first request just wrote.

def store(digest:, question:, answer:, embedding:, model:)
  LlmCacheEntry.create!(
    question_digest: digest,
    question: question,
    answer: answer,
    embedding: embedding,
    model: model,
    expires_at: 30.days.from_now
  )
rescue ActiveRecord::RecordNotUnique
  LlmCacheEntry.find_by!(question_digest: digest)
end

You still paid for both provider calls in that race. The cache saves you on the third request onward, not the concurrent pair. That is the honest limit. If brand-new questions arrive in thundering herds, a per-digest lock before the provider call is the next lever, and it is more machinery than most teams need on day one.

• A unique digest index plus a RecordNotUnique rescue is the whole race fix for the common case.
• The concurrent-miss pair still pays twice. Semantic caching pays off on repeat volume, not on the first two simultaneous askers.

When a Semantic Cache Is the Wrong Call

I would not put a semantic cache in front of every LLM call, and it took one bad incident to learn where it does not belong.

Personalized answers are the clearest no. If the response depends on the user’s account, their data, their permissions, then the same question from two users is not the same request. A shared cache leaks one user’s answer to another. Key the cache per-user, or do not cache these at all.

Authored or high-stakes answers are the second no. Legal, medical, anything a human reviewed. A false-positive semantic match returns a reviewed answer to an unreviewed question. Tighten the distance ceiling hard, or skip the semantic layer and keep only exact-match.

And it does not replace OpenAI’s prompt caching. If your prompts are large and static at the front and users send identical prompts within minutes, that input discount is free and needs no table. Semantic caching kills whole calls when the wording varies. I run both.

One more cost that hides until you hit it. Vectors from one embedding model are not comparable to vectors from another. The day you upgrade text-embedding-3-small to a newer model, every cached vector was written in the old space, and new queries land in the new one, so similarity scores go quietly meaningless. That is why the row stores its model. When the model changes, you either scope lookups to rows written by the current model, or you re-embed the cache, or you let the old entries expire out. Plan the swap as a migration, not a config change.

• Never share a semantic cache across users when the answer depends on user data. Key per-user or do not cache.
• For reviewed or high-stakes answers, tighten the distance ceiling or drop the semantic layer entirely.
• Changing the embedding model invalidates every stored vector. Scope by model, re-embed, or expire out. Never mix vectors from two models in one similarity search.

The Bill, Traced

Say a common question gets asked 100 times a day in ten phrasings, each answer a 500-token reply. No cache, you pay 100 output-token bills. Exact-match, you pay 10, one per unique phrasing. Semantic, you pay one, plus one cheap embedding per question.

The embedding you add is a rounding error. text-embedding-3-small runs about $0.02 per million tokens at current pricing (July 2026), tens of thousands of pages of text per dollar. The call you avoid is gpt-5.5 output at $30.00 per million tokens. A fraction-of-a-cent embedding for a full generation you never make. That four of five for free is not marketing, it is the shape of natural-language traffic, where every phrasing after the first is a call you already answered.

Your users were never asking five questions. They were asking one, five times, and you were paying full price for the echo. The lever you now hold is a distance ceiling and a vector column. One paid call, four free answers, and a bill that finally tracks the questions you actually answered.

Want the follow-up? Tell me in a response and I will write the eviction and freshness strategy, expiring cache entries when the underlying answer changes, so you never serve a stale support answer for a month.

Related reads

• pgvector Semantic Search in Rails: Catch What Keyword Misses
• Your Rails RAG Pipeline Is Not Hallucinating. It Is Retrieving Garbage. Here Is the Diff.

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!