How Semantic Search Works Behind the Scenes

From raw text to vector databases — a visual, step-by-step guide to the technology powering modern AI search.

1. What is a Vector Database?

A traditional database stores data in rows and columns — it’s great at exact lookups. But when you want to search by meaning rather than keywords, you need something fundamentally different: a vector database.

A vector database stores numerical representations of data (called vectors or embeddings) and is optimized to find items that are nearest to a query vector in high-dimensional space. Think of it as a map where semantically similar things live close together.

Key insight: The word “dog” and “puppy” might be miles apart alphabetically, but their vector representations sit only millimeters apart in embedding space — because they share similar meaning.

2. What is an Embedding?

An embedding is the process of converting text (or images, audio, etc.) into a list of numbers — a vector — such that semantically similar content maps to numerically similar vectors.

Modern embedding models like OpenAI’s text-embedding-3 or Cohere's embed-v3produce vectors with 768 to 3,072 dimensions. Every dimension captures some latent semantic feature — though no single dimension has a human-readable meaning.

Mental model: Imagine each word occupies a point in 3D space — but with 1,536 dimensions instead of 3. Words used in similar contexts cluster together. That spatial proximity is the semantic similarity.

What exactly is a “dimension”?

When an embedding model encodes a sentence, it outputs a long array of floating-point numbers — each one called a dimension. Think of each dimension as a dial that captures a tiny slice of meaning. No single dial maps cleanly to a human concept like “financial” or “emotional”, but together, hundreds or thousands of dials recreate the full semantic fingerprint of the text.

⚠ These labels are illustrative. Real embedding dimensions are anonymous — the model learns them from billions of sentences without being told what they mean.

The full vector for this sentence would have 384 to 3,072 such numbers depending on the model — not 6. The chart above is a simplified window into what’s actually happening.

How dimensions affect search quality

More dimensions generally means the model can capture finer semantic distinctions — but at a cost. Higher-dimensional vectors take more storage, more memory, and longer to compare. The right choice depends on your use case:

Practical rule: Start with a 384–768 dimension model for prototyping — fast and cheap. Move to 1,536+ when retrieval quality needs to improve in production. Higher dimensions only help if your dataset is large and semantically diverse enough to benefit.

3. What is Semantic Search?

Semantic search finds results based on intent and meaning, not keyword overlap. When you ask “how much can John invest?”, keyword search fails — it doesn’t find “John has $500 in his bank”. Semantic search does, because the meaning of both sentences is close in vector space.

It works by embedding the query into a vector, then finding the stored vectors that are geometrically closest — typically using cosine similarity or dot product.

4. How These Three Are Interlinked

The three concepts form a pipeline. Text gets embedded into vectors, vectors get stored in a vector DB, and semantic search queries that DB using vector similarity.

5. Live Demo: Text → Vectors → Search

Watch how a PDF sentence gets converted to a vector, stored in the database, and retrieved when an LLM asks a semantically related question.

6. Why Semantic Filtering Matters

Pure vector search can be a blunt instrument. Semantic filtering lets you combine vector similarity with structured constraints — ensuring results are both relevant and correct.

Example: A legal AI searches 10M documents. Without filters, results span decades and jurisdictions. With semantic filters — jurisdiction: “US” AND year: 2022–2024 — it returns the 200 relevant chunks, then ranks by similarity.

7. Available Vector Databases & Embedding Models

Vector Databases

Embedding Models

8. Real-World Case Study: PortfoliMosaic

Theory is only half the story. Let’s look at how all these concepts come together in a real open-source project — PortfoliMosaic, an AI-powered financial document assistant built with FAISS, Sentence Transformers, LangChain, and GPT-4.

PortfoliMosaic

AI-powered financial document assistant — analyze brokerage statements, tax forms, and balance sheets through natural language Q&A.

This project is a working example of the full semantic search pipeline described in this article. Here’s how each concept maps to the implementation:

• FAISS
• all-MiniLM-L6-v2 (384 dims)
• LangChain Agent
• GPT-4o-mini
• FastAPI
• SQLite
• Landing AI ADE
• React + Vite

https://www.youtube.com/watch?v=vH5-Af0NfU4

https://github.com/sivasundharam/PortfoliMosaic