101 Vespa|Search Fundamentals

Visaul Mental Model

                USER
                  │
                  ▼
         ┌────────────────┐
         │    QUERY       │
         │   "vespa"      │
         └────────────────┘
                  │
                  ▼
      ┌──────────────────────┐
      │      MATCHING        │
      │ Inverted Index Lookup│
      └──────────────────────┘
                  │
                  ▼
      ┌──────────────────────┐
      │      RANKING         │
      │ Relevance Scoring    │
      └──────────────────────┘
                  │
                  ▼
      ┌──────────────────────┐
      │      RESULTS         │
      │ Top Relevant Docs    │
      └──────────────────────┘

──────────────────────────────────

  DOCUMENTS
     │
     ▼
┌─────────────────────────────┐
│          INDEXING           │
│ Tokenise → Build Index      │
└─────────────────────────────┘

"vespa"  → [1,3,5]
"search" → [1,2,4]
"engine" → [1,3]

Simple Search Engine Mental model

                 SEARCH ENGINE

         ┌───────────────────────┐
         │       INDEXING        │
         │ Build Search Map      │
         └───────────┬───────────┘
                     │
                     ▼
         ┌───────────────────────┐
         │       MATCHING        │
         │ Find Candidates       │
         └───────────┬───────────┘
                     │
                     ▼
         ┌───────────────────────┐
         │       RANKING         │
         │ Order by Relevance    │
         └───────────┬───────────┘
                     │
                     ▼
         ┌───────────────────────┐
         │        RESULTS        │
         └───────────────────────┘

What Really Happens After You Press Enter?

Imagine you're standing inside a massive library.

Not a normal library.

A library containing millions of books, articles, product descriptions, support tickets, blog posts, and research papers.

Now imagine someone walks in and says:

"Vespa search."

That's it.

Two words.

No context.

No explanation.

No indication whether they're looking for documentation, architecture diagrams, installation guides, or performance benchmarks.

Yet somehow, within milliseconds, a search engine returns a list of results that feels almost magical.

How?

This module is about understanding that magic.

Not machine learning.

Not embeddings.

Not neural ranking.

Just the fundamental machinery that every search engine relies on.

By the end, you'll be able to explain exactly what happens between a user typing a query and seeing results on the screen.

The Real Problem Search Engines Solve

Most people think search is about finding documents.

It isn't.

Search is about understanding intent.

The challenge is that users rarely express their intent clearly.

A user might type:

vespa

But what they actually mean could be:

• "What is Vespa?"
• "How does Vespa ranking work?"
• "Install Vespa on Kubernetes"
• "Compare Vespa and Elasticsearch"

The search engine never sees the intent.

It only sees the query.

Its job is to bridge the gap between a few typed words and the documents most likely to satisfy the user's need.

To do that, every search engine solves three core problems:

1. Indexing

How do we organize millions of documents so we can find them quickly?

2. Matching

Given a query, which documents are even candidates?

3. Ranking

Of those candidates, which should appear first?

Everything in search ultimately falls into one of these three categories.

Why Search Doesn't Scan Every Document

Suppose your system contains one million documents.

A naive approach would be:

1. Read document 1
2. Check if it contains "vespa"
3. Read document 2
4. Check again
5. Repeat one million times

This would be painfully slow.

Instead, search engines build a shortcut.

A giant lookup table.

This shortcut is called an inverted index.

The Data Structure That Changed Search

Instead of storing information by document, we store information by term.

Think of it like the index at the back of a textbook.

Rather than asking:

"Which words exist in this document?"

we ask:

"Which documents contain this word?"

Example:

vespa   → [doc1, doc3, doc5]
search  → [doc1, doc2, doc4]
engine  → [doc1, doc3]

Now when a user searches for:

vespa search

the engine doesn't inspect every document.

It simply looks up two lists and combines them.

This is the fundamental reason keyword search is fast.

The heavy work happens before the query arrives.

How Vespa Builds the Index

Vespa’s search core, called Proton, builds and maintains the index through a layered pipeline of in-memory and on-disk structures that get periodically merged together. Here’s the flow shown above, broken down:

When a document is written, it first goes through Proton’s feed handler and gets persisted in the transaction log before updating the document in the document store, with the operation acknowledged to the client and immediately visible in search results. The transaction log exists specifically so that writes are durable without forcing every update to go straight to a slow disk-based index structure.

From there, the document enters the memory index. Vespa keeps a memory index where mutating document operations are applied, and this memory index has a dictionary per index field containing all unique words with their posting lists. This in-memory structure is what makes newly fed documents searchable almost instantly, since updating an in-memory dictionary is far cheaper than rewriting a disk file.

That memory index can’t grow forever, so a maintenance job periodically flushes it. The memory index flush job writes a memory index to disk and then triggers disk index fusion, with the goal of shrinking memory usage by adding to the disk-backed indices. Each flush produces a new disk segment, and disk index fusion merges the primary disk index with these smaller indices generated by flushing — so over time you don’t end up with hundreds of tiny disk segments, just one consolidated primary index plus whatever’s currently in memory.

On disk, each index field gets its own folder containing a dictionary, a compressed posting list with position information for ranking, and optionally a bitvector-style occurrence file for very common terms used in filtering-only queries.

A couple of details worth knowing: this whole index only applies to fields marked as index type in the schema (string fields for text search) — numeric/sortable fields typically live in separate in-memory attribute structures instead, and the raw document itself sits in yet another structure, the document store, used for re-indexing and replication rather than search.