How Search Engines Actually Work — Behind All of It

Hey, it’s been a while.

Honestly, I needed the break. But I didn’t stop learning —I’ve been wanting to explore for a long time: how search actually works under the hood.

Not the UI. Not the algorithm that decides which result shows up first. I mean the thing underneath all of that — how a system manages to search through billions of documents in under a second.

So this is Day 1 of what’s going to be a 2-part series. Today: the fundamentals of search engines and a data structure called the Inverted Index that makes fast search possible. Day 2 will be hands-on — we’ll plug all of this into an actual AI system.

Let’s get into it.

Okay, but how hard can search really be?

That’s what I thought too. Then I sat with the actual problem for a minute.

You have 10 billion documents. Someone types three words. You have less than a second to return the best results.

If you tried to check each document one by one — even on fast hardware — you’d be looking at hours, not milliseconds. That’s just not how it’s done.

The trick is: you do most of the work before anyone even searches.

What a search engine actually does

Three things, in order:

Crawl — Go find content. Webpages, PDFs, docs, whatever. Spiders crawl the web following links, collecting everything.

Index — Organize that content into a structure that can be searched quickly. This is where the interesting stuff happens.

Rank — When someone searches, decide which results are most relevant and in what order.

Most people talk about ranking. But indexing is the part I found fascinating, because without a smart index, ranking doesn’t even matter — you’d never find the candidates fast enough.

The dumb way to search (and why it breaks)

Say you have these 5 documents:

• Doc 1: “the cat sat on the mat”
• Doc 2: “the dog sat on the log”
• Doc 3: “the cat chased the dog”
• Doc 4: “a dog and a cat are friends”
• Doc 5: “machine learning is fun”

Someone searches for cat. The obvious approach — scan every document and check if the word is there. Works fine for 5 docs.

But at 10 billion? You’d need to process terabytes of text for every single query. That’s a non-starter.

There has to be a smarter way to set this up.

The Inverted Index — flip the question, change everything

Here’s the insight that clicked for me:

Instead of asking “does this document contain the word?” — flip it. Ask “which documents contain this word?” And pre-compute that answer for every word, once, ahead of time.

That’s an Inverted Index. It’s basically a dictionary where:

• Each key is a word (called a term)
• Each value is the list of documents that contain it (called a postings list)

For our 5 docs, it looks like this:

"cat"      → [Doc1, Doc3, Doc4]
"dog"      → [Doc2, Doc3, Doc4]
"sat"      → [Doc1, Doc2]
"machine"  → [Doc5]
"learning" → [Doc5]

Now someone searches for cat. You do a single dictionary lookup. Done. No scanning. No looping through billions of documents. Just — here are the docs, instantly.

That’s the whole trick. The hard work happens once when you build the index, not every time someone searches.

Real indexes store more than just document IDs

A basic inverted index gets you pretty far, but production systems store extra info with each entry:

"cat" → [
  (Doc1, positions=[2], frequency=1),
  (Doc3, positions=[2], frequency=1),
  (Doc4, positions=[2], frequency=1)
]

The positions part is what makes phrase search work. If you search for "machine learning" you don't just want documents that contain machine and learning somewhere — you want them next to each other. Positions let the engine verify that.

Before a word gets indexed, it goes through a pipeline

The text doesn’t go straight into the index as-is. It gets cleaned up first, every time a document is added. Here’s the chain:

Tokenization — chop the text into individual words

"The cats are running fast" → ["The", "cats", "are", "running", "fast"]

Lowercasing — so Cat and cat match

→ ["the", "cats", "are", "running", "fast"]

Stop word removal — ditch words like the, a, is, are that don’t add search value

→ ["cats", "running", "fast"]

Stemming — reduce words to their root form, so running, ran, and runs all become run

→ ["cat", "run", "fast"]

This whole pipeline runs when documents are added to the index — not when someone searches. So query time stays fast.

Multi-word searches are just set operations

When you search cat AND dog, the engine looks up both postings lists and finds the overlap:

"cat" → [Doc1, Doc3, Doc4]
"dog" → [Doc2, Doc3, Doc4]

AND  → [Doc3, Doc4]   ← documents that have both
OR   → [Doc1, Doc2, Doc3, Doc4]   ← documents that have either

That’s it. Everything you’ve ever searched is built on intersection and union operations like these.

And how do results get ranked?

The index tells you which documents are relevant. Ranking decides how relevant. The classic way is TF-IDF:

• TF (Term Frequency) — how often the word appears in a document. More = more relevant.
• IDF (Inverse Document Frequency) — how rare the word is across all documents. Rarer = more meaningful.

So a word like the appears everywhere — low IDF, barely contributes to the score. A word like eigenvalue appears in very few documents — high IDF, big signal.

Multiply them together and you get a relevance score. Higher score = shows up first.

Modern search goes well beyond this (BM25, vector similarity, neural re-rankers), but they all build on this foundation.

Let’s build one

Here’s a minimal inverted index in Python. Ugly-simple on purpose:

from collections import defaultdict
import re

documents = {1: "The cat sat on the mat", 2: "The dog sat on the log", 3: "The cat chased the dog", 4: "A dog and a cat are friends", 5: "Machine learning is fun"}

class InvertedIndex:
    def __init__(self, documents):
        self.index = defaultdict(list)
        self.documents = documents
        for doc_id, text in documents.items():
            self._build_index(doc_id, text)

    def _tokenize(self, text):
        tokens = re.findall(r'\b\w+\b', text.lower())
        stop_words = {"the", "a", "an", "is", "are", "on", "in", "and", "of"}
        return [t for t in tokens if t not in stop_words]

    def _build_index(self, doc_id, text):
        for pos, token in enumerate(self._tokenize(text)):
            self.index[token].append((doc_id, pos))

    def search(self, query):
        terms = self._tokenize(query)
        if not terms:
            return []
        results = set(doc_id for doc_id, _ in self.index.get(terms[0], []))
        for term in terms[1:]:
            results &= set(doc_id for doc_id, _ in self.index.get(term, []))
        return [{"doc_id": doc_id, "text": self.documents[doc_id]} for doc_id in results]

idx = InvertedIndex(documents)

cat_results     = idx.search("cat")
machine_results = idx.search("machine")

print("--- cat ---")
for result in cat_results:
    print(result["doc_id"], result["text"])

print("--- machine ---")
for result in machine_results:
    print(result["doc_id"], result["text"])

Output- 

--- cat ---
1 The cat sat on the mat
3 The cat chased the dog
4 A dog and a cat are friends

--- machine ---
5 Machine learning is fun

40 lines. Same core idea as Elasticsearch.

What changes at real scale

A few things look different when you’re running this across billions of documents on thousands of machines:

Sharding — the index gets split across machines. A query fans out to all of them in parallel, results get merged.

Compression — postings lists can have millions of IDs. They get compressed using delta encoding (store the difference between IDs, not the IDs themselves).

Caching — common queries are cached. Nobody’s recomputing "weather today" from scratch a billion times.

Segments — new documents go into small in-memory buffers that merge into the main index in the background. This is how Lucene (the engine under Elasticsearch) handles near-real-time indexing.

Why this matters for AI (Day 2 preview)

Every RAG system — every AI assistant that retrieves context before answering — is doing search. When you ask an LLM a question and it pulls from a knowledge base, something like this is running underneath.

The classic approach uses keyword search (BM25, built on inverted indexes). The newer approach uses vector/semantic search. The best systems use both — which is called hybrid search.

Day 2 is where we build that. BM25 + vector search + an LLM, wired together into a working AI system.

But you can’t really understand hybrid search without understanding both halves. Today was half one.

See you in Day 2.