pgvector Semantic Search in Rails: Catch What Keyword Misses

How keyword search drops cancel my plan when your docs say terminate subscription, and the embedding plus ranking fix that closes the gap

pgvector’s hybrid search bridges the semantic gap — exact matches and contextual intent now coexist in Rails’ query results.

If You Do Three Things Today

A user types cancel my plan. Your search returns nothing. The answer is sitting in a doc titled How to Terminate Your Subscription. Your search box does not know those mean the same thing. It never will.

Semantic search in Rails with pgvector solves this by comparing meaning instead of tokens. A user’s query and your doc get converted into embeddings — lists of 1536 numbers that encode meaning. Words that mean similar things land close together in that space. Cancel and terminate end up near each other. The search finds the doc.

1. Enable the pgvector extension in a migration and add an embedding column to your searchable model.
2. Add a background job that calls your embedding API and stores the result after each record saves.
3. Replace your current ILIKE '%query%' with nearest_neighbors(:embedding, query_vector, distance: "cosine").limit(10).

The rest of this article covers why each step matters and where the setups you have read before cut corners.

Why Keyword Search Has a Ceiling You Cannot Engineer Around

I spent more time than I want to admit trying to fix keyword search before accepting it cannot be fixed. Synonyms table. Stemming. Trigram indexes. Each one moved the line a few percent and introduced a new class of miss.

The problem is structural. ILIKE '%cancel%' matches the string cancel. It does not match terminate, end, close, discontinue, or stop billing — even though every one of those phrases means the same thing to a user typing into a support search box. You can add synonym expansion, but synonym tables rot the moment your product language shifts. Someone renames a feature, the synonym table goes stale, and you are back to zero results.

Trigram indexes (pg_search’s :trigram mode) help with typos and partial matches. They do not help with semantic distance. Cancel and terminate share zero trigrams.

The gap is not a bug in your implementation. It is the ceiling of the approach.

• Token-matching treats text as a bag of characters. Two documents that share no characters share no relevance score, no matter what they mean.
• Synonym tables scale with vocabulary, not with meaning. Every new phrasing requires a manual entry.
• You cannot enumerate all the ways users say a thing. They will always find a new one.

The Three-Layer Semantic Search Pipeline

Before writing a line of code, it helps to see what you are wiring:

User query ("cancel my plan")
    ↓
Embed query → 1536-dimension vector via text-embedding-3-small
    ↓
pgvector HNSW index (cosine distance) on Documents table
    ↓
Top-K nearest neighbor documents ranked by vector distance
    ↓
Optional: re-rank by keyword score + vector score (hybrid)
    ↓
Results

The index layer is where most setups I have seen cut the first corner. They use IVFFlat (Inverted File with Flat compression), which requires you to know the list count before you have data and degrades badly if you add records without reindexing. HNSW (Hierarchical Navigable Small World) builds incrementally and handles a growing corpus without manual rebuilds. For anything that gets new records regularly — support docs, help articles, product pages — HNSW is the right index from day one.

Setting Up pgvector and the neighbor Gem in Rails

The pgvector PostgreSQL extension (v0.8.0) adds the :vector column type and the distance operators. This is PostgreSQL-only — it does not work with SQLite or MySQL. The neighbor gem (v1.1.1, Andrew Kane) wraps pgvector in ActiveRecord so you get familiar scope syntax instead of raw SQL.

Two migrations. First, enable the extension:

class EnablePgvector < ActiveRecord::Migration[7.1]
  def change
    enable_extension "vector"
  end
end

Second, add the embedding column and the HNSW index. The column’s limit: must match your embedding model's output dimensions exactly. OpenAI's text-embedding-3-small outputs 1536 dimensions by default:

class AddEmbeddingToDocuments < ActiveRecord::Migration[7.1]
  def change
    add_column :documents, :embedding, :vector, limit: 1536
    add_index :documents, :embedding,
              using: :hnsw,
              opclass: :vector_cosine_ops
  end
end

opclass: :vector_cosine_ops tells pgvector which distance metric the index is optimised for. Build an HNSW index with vector_l2_ops and then query with cosine distance, and the index does not accelerate your queries — it just costs you disk space. Match the opclass to your query metric or the index is unused.

In your model:

class Document < ApplicationRecord
  has_neighbors :embedding
end

has_neighbors registers the column with the neighbor gem and makes nearest_neighbors available on the class.

• The limit: on the migration column and your embedding model's output dimensions must match exactly — a mismatch raises a PostgreSQL dimension error at query time, not at migration time.
• HNSW index builds incrementally; IVFFlat requires a lists: parameter and degrades without periodic reindexing as your corpus grows.
• opclass must match the distance metric you use in queries or the index is bypassed entirely.

Generating and Storing Embeddings Without Slowing Down Writes

The embedding call is the part most tutorials skip. They show the query side and assume you have already populated the column. You have not.

The wrong way is a before_save callback that calls the embedding API synchronously. Your users wait for a network round trip on every doc save. One slow OpenAI response and the save request times out.

The right way is an after_save job. The record saves immediately. The embedding backfills asynchronously:

class Document < ApplicationRecord
  has_neighbors :embedding
 after_save :enqueue_embedding_job, if: :content_changed?
  private
  def enqueue_embedding_job
    new_hash = Digest::SHA256.hexdigest(content)
    return if new_hash == content_hash
    update_columns(content_hash: new_hash)
    EmbedDocumentJob.perform_later(id)
  end
end

class EmbedDocumentJob < ApplicationJob
  queue_as :embeddings
  retry_on StandardError, wait: :polynomially_longer, attempts: 5
  def perform(document_id)
    document = Document.find(document_id)
    response = OpenAI::Client.new.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: document.content
      }
    )
    vector = response.dig("data", 0, "embedding")
    document.update_columns(embedding: vector)
  end
end

update_columns bypasses callbacks and validations. Without it, the embedding update triggers another after_save, which queues another job, which triggers another after_save. The content_hash check in the model stops the same content from re-embedding even if the record saves for an unrelated reason.

retry_on with polynomially_longer gives you five attempts with increasing backoff — critical when you hit OpenAI rate limits during bulk indexing of an existing corpus. Without it, one rate-limited request fails permanently and leaves a gap in your index.

There is a window between a document saving and its embedding being written where a semantic search will miss the new doc. For most apps — knowledge bases, support docs, help centers — that window is acceptable. If freshness matters in seconds, a new record with no embedding silently returns zero vector score. In a hybrid ranking setup it gets buried behind well-scored results anyway, which is usually the right behaviour.

• Never embed synchronously in a save callback — the network latency lands on the user’s write path.
• update_columns prevents the embedding update from re-triggering after_save and creating a job loop.
• retry_on is mandatory — OpenAI rate limits will hit during bulk indexing.

Querying: rails vector search From Input to Results

At query time, embed the user’s search string with the same model you used for documents, then ask pgvector for the nearest neighbors:

class SearchService
  def self.call(query_text, limit: 10)
    response = OpenAI::Client.new.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: query_text
      }
    )
    query_vector = response.dig("data", 0, "embedding")
    Document
      .nearest_neighbors(:embedding, query_vector, distance: "cosine")
      .limit(limit)
  end
end

distance: "cosine" is a string, not a symbol. Pass :cosine and you get a NoMethodError. I got that wrong the first time. The returned records include a neighbor_distance attribute — a float where 0.0 is identical and 2.0 is maximally different. For most doc retrieval use cases, anything above 0.4 is noise:

Document
  .nearest_neighbors(:embedding, query_vector, distance: "cosine")
  .where("neighbor_distance < 0.4")
  .limit(limit)

That threshold is not universal. Run a sample of real queries against your corpus, pull neighbor_distance for the results, and find the cutoff where relevant and irrelevant results split in your data. 0.4 is where I have landed on support doc corpora — a single-project measurement, not a benchmark. A broader corpus may need 0.5 or 0.6.

• distance: "cosine" is a string — passing a symbol raises NoMethodError.
• neighbor_distance is available on every result record without a separate query.
• Calibrate the distance threshold against real queries on your actual corpus.

Why Pure Semantic Search Is Not Enough — and How pgvector Ranking Fixes It

I shipped pure semantic search on one project and thought I was done. Then a user searched for a specific product SKU — PRO-2024-X — and got back thematically related results with no SKU match. Exact string matches were scoring no better than vague semantic neighbors.

The fix is hybrid search rails: combine the vector score with a keyword score, rank by the weighted sum. Semantic search handles meaning. Keyword search handles exact matches, product codes, proper nouns, and anything where token identity matters.

With pg_search (v2.3.7, Grant Hutchins), you can get a keyword score from PostgreSQL’s built-in full-text search and combine it with the neighbor distance. Add the scope to the model:

class Document < ApplicationRecord
  has_neighbors :embedding
  pg_search_scope :keyword_search,
                  against: :content,
                  using: { tsearch: { prefix: true } }
end

Then merge both result sets in your service:

class HybridSearchService
  VECTOR_WEIGHT = 0.7
  KEYWORD_WEIGHT = 0.3
  def self.call(query_text, limit: 10)
    response = OpenAI::Client.new.embeddings(
      parameters: { model: "text-embedding-3-small", input: query_text }
    )
    query_vector = response.dig("data", 0, "embedding")
    vector_results = Document
      .nearest_neighbors(:embedding, query_vector, distance: "cosine")
      .limit(limit * 2)
      .map { |d| [d.id, d.neighbor_distance] }
      .to_h
    keyword_results = Document
      .keyword_search(query_text)
      .limit(limit * 2)
      .map.with_index { |d, i| [d.id, i] }
      .to_h
    all_ids = (vector_results.keys + keyword_results.keys).uniq
    scored = all_ids.map do |id|
      vector_score = vector_results[id] ? (1.0 - vector_results[id]) : 0.0
      keyword_rank = keyword_results[id]
      keyword_score = keyword_rank ? (1.0 / (keyword_rank + 1)) : 0.0
      combined = (VECTOR_WEIGHT * vector_score) + (KEYWORD_WEIGHT * keyword_score)
      [id, combined]
    end
    top_ids = scored.sort_by { |_, score| -score }.first(limit).map(&:first)
    Document.where(id: top_ids).index_by(&:id).values_at(*top_ids).compact
  end
end

The vector score converts neighbor distance to similarity: 1.0 - distance. The keyword score uses reciprocal rank — position 0 scores 1.0, position 1 scores 0.5, and so on. At 0.7/0.3 weighting, semantic meaning dominates but exact SKU matches get a meaningful lift into the top results.

The weights are not magic numbers. On a support doc corpus, 0.7/0.3 worked well on the three projects I measured this on. On a product catalog where SKU matching is critical, 0.5/0.5 is more appropriate. Run a set of representative queries, compare results at different weight ratios, and measure precision at the top three results. That is the only calibration that matters for your data.

Fetch limit * 2 from each source before merging. Candidates that appear in both lists need to be present in both maps to receive the combined score boost. A limit(10) on each pass risks excluding a highly-ranked candidate from one side before the merge sees it.

• Pure semantic search scores SKUs and proper nouns no better than vague semantic neighbors — hybrid ranking fixes this.
• Reciprocal rank (1 / position + 1) is a stable keyword scoring function that does not require pg_search to expose a raw score.
• Fetch more candidates than you need from each source before merging — the best results often appear in both sets.

What Breaks in Production That No Tutorial Covers

Two things caught me off-guard that nothing I read prepared me for.

The first is the embedding API rate limit on bulk indexing. When you first run the job across an existing corpus of 10,000 documents, you will hit OpenAI’s rate limit within minutes. The retry_on in the job handles transient failures, but you also need to process in batches with a brief sleep between them. A single Document.find_each with no throttle will exhaust your rate limit and leave a partially-indexed corpus.

The second is the write path impact at scale. Every content update now requires an async embedding call. At low-edit-volume apps this is invisible. At high-edit-volume apps — wikis, CMS platforms, anything with frequent bulk imports — the embeddings queue can grow faster than it drains if your job concurrency is too low or your embedding API tier is too slow. Monitor queue depth on the embeddings queue separately from your general job queue. A backed-up embeddings queue means stale vectors and degrading search quality, with no error in the logs to tell you.

The semantic gap between what users type and what your docs say is not something you can close with better SQL. Cancel my plan and terminate subscription are two tokens with zero overlap and identical intent. Semantic search in Rails with pgvector closes that gap with one embedding column, one background job, and a neighbor scope that your existing ActiveRecord queries compose with naturally. Add hybrid ranking and you keep the exact-match precision that keyword search was good at. That is the setup. The rest is tuning.

If this matched a problem you are sitting on, the next level is cross-encoder reranking — when the HNSW top-K is not precise enough and you need a second model pass over the candidates before showing results.

Related reads

• Your Rails RAG Pipeline Is Not Hallucinating. It Is Retrieving Garbage. Here Is the Diff.