How Search Engines Evolved: From TF-IDF and BM25 to Transformers and RAG

From Keywords to Transformers: How Search Learned to Understand Meaning

Every day, we search for information.

We search on Google, look through emails, browse documentation, query databases, and increasingly interact with AI-powered assistants.

Most of us rarely think about what happens behind the scenes when we type a question into a search box. We simply expect relevant information to appear within seconds.

While working on a Retrieval-Augmented Generation (RAG) project recently, I became curious about a question:

How did search systems evolve from simple keyword matching to modern AI-powered retrieval?

As I started exploring the history of information retrieval, I realized that many of the technologies behind today’s AI applications are built on decades of research in search and document retrieval.

Modern systems such as semantic search, vector databases, and RAG pipelines may seem new, but they are part of a much longer story.

This article is a beginner-friendly journey through that evolution — from TF-IDF and BM25 to embeddings, transformers, and modern retrieval systems.

What Is Information Retrieval?

At its core, information retrieval is the process of finding relevant information from a large collection of documents.

Whenever you:

• Search on Google
• Search your emails
• Search a PDF document
• Ask questions to a RAG chatbot

You are interacting with an information retrieval system.

The challenge sounds simple:

A user asks a question.

The system finds the most relevant information.

In practice, however, determining what “relevant” means has been one of the central problems in computer science for decades.

The Early Days: Keyword Matching

The earliest search systems were surprisingly straightforward.

They looked for exact words.

Imagine a document containing the sentence:

Machine learning is widely used in healthcare.

If a user searched for:

machine learning

The document would likely be retrieved because the exact phrase appears in the text.

However, consider a different query:

AI techniques in medicine

Although the meaning is very similar, the document might not be retrieved because none of those exact words appear.

The system understood words, but not the meaning.

This approach worked reasonably well for small collections of documents, but it struggled as information grew and user expectations increased.

Researchers needed better ways to determine which documents were actually important.

TF-IDF: Making Search Smarter

One of the most influential developments in information retrieval was TF-IDF, which stands for:

Term Frequency — Inverse Document Frequency

Despite the intimidating name, the idea is surprisingly intuitive.

TF-IDF tries to identify words that are important within a document while ignoring words that appear everywhere.

Let’s break it down.

Term Frequency (TF)

If a word appears many times in a document, it is probably important to that document.

For example, if a research paper repeatedly mentions:

• neural networks
• deep learning
• language models

Those terms likely represent the main topics of the document.

TF measures how frequently a word appears.

Inverse Document Frequency (IDF)

Not every frequently used word is useful.

Words such as:

• the
• is
• and
• of

appear in almost every document.

These words provide little information about the document’s content.

IDF reduces the importance of words that appear everywhere and increases the importance of relatively rare words.

Together, TF and IDF create a scoring system that highlights meaningful terms.

For many years, TF-IDF has become one of the most widely used techniques in search and document retrieval.

Even today, it remains an important concept for understanding how search systems work.

BM25: The Industry Standard

As retrieval systems became more sophisticated, researchers developed improvements over TF-IDF.

One of the most successful was BM25.

Without diving too deeply into the mathematics, BM25 improves document ranking by accounting for factors such as:

• document length
• term frequency saturation
• scoring normalization

In practical terms, BM25 often produces better rankings than traditional TF-IDF.

What makes BM25 particularly interesting is that it remains widely used even today.

Many modern search systems still rely on BM25 because it is efficient, interpretable, and surprisingly effective.

This is an important reminder that newer technologies do not always replace older ones completely.

Sometimes they build upon them.

The Shift Toward Meaning

Although TF-IDF and BM25 were major improvements, they still had a fundamental limitation.

They focused heavily on words.

Humans, however, communicate through meaning.

Consider these two phrases:

Deep learning for language processing

and

Neural networks used in NLP

Most people would recognize that these sentences discuss similar concepts.

Traditional retrieval methods often struggle because the exact words differ.

Researchers needed a way to represent meaning rather than simply counting words.

This challenge eventually led to one of the biggest shifts in information retrieval.

Embeddings and Semantic Search

Modern retrieval systems often use embeddings.

An embedding converts text into a numerical vector that captures semantic information.

Instead of treating a document as a collection of words, the system represents it as a point in a high-dimensional space.

Documents discussing similar topics tend to appear closer together.

For example:

• “Machine learning in healthcare”
• “AI applications in medicine”

may end up near each other even if they share very few exact words.

This allows retrieval systems to perform semantic search.

Rather than asking:

Do these documents contain the same words?

The system asks:

Do these documents have similar meanings?

This change dramatically improved retrieval quality.

It also laid the foundation for many modern AI applications.

Vector Databases Enter the Picture

Once text can be represented as vectors, a new challenge appears.

Where do we store millions of vectors?

And how do we search them efficiently?

This led to the rise of vector databases such as:

• Qdrant
• Pinecone
• Weaviate
• Milvus

Instead of storing traditional rows and columns, these systems specialize in storing embeddings and performing similarity search.

When a user submits a query, the query is converted into an embedding and compared against stored document vectors.

The system then retrieves the most semantically relevant results.

This approach powers many modern recommendation systems, search engines, and AI assistants.

The Transformer Revolution

While embeddings improved retrieval significantly, another breakthrough transformed natural language processing entirely.

In 2017, researchers introduced the Transformer architecture.

The impact of this innovation is difficult to overstate.

Transformers enabled models such as:

• BERT
• RoBERTa
• GPT

Unlike earlier approaches, transformer models can understand context much more effectively.

Consider the word:

Apple

In one sentence, it may refer to a fruit.

In another, it may refer to a technology company.

Transformers use the surrounding context to determine the correct meaning.

This ability to capture context helped improve both retrieval and language understanding.

It also accelerated the development of modern AI systems.

From Search Engines to RAG Systems

Today, information retrieval has entered a new phase.

Many modern AI applications combine retrieval systems with large language models.

This approach is commonly known as Retrieval-Augmented Generation (RAG).

Instead of relying entirely on a model’s training data, a RAG system:

1. Receives a user query.
2. Retrieves relevant documents.
3. Provides retrieved information to the language model.
4. Generates a response grounded in that information.

In many ways, RAG represents the convergence of multiple generations of retrieval research.

A modern system may use:

• BM25 for keyword retrieval
• Embeddings for semantic search
• Vector databases for storage
• Transformers for understanding context
• LLMs for answer generation

What appears to be a simple chatbot often relies on decades of advancements in information retrieval.

What I Found Most Interesting

One thing that surprised me while learning about information retrieval was how many “new” AI technologies are actually built on older ideas.

When people discuss modern AI, conversations often focus on language models and transformers.

However, retrieval systems have been evolving for decades.

Concepts such as document ranking, relevance scoring, and semantic similarity existed long before today’s AI boom.

Understanding this history helped me appreciate that modern AI is not a single breakthrough.

It is the result of many incremental advances building upon one another.

Final Thoughts

The journey from keyword matching to transformer-based retrieval reflects a broader shift in how computers process information.

Early systems focused on words.

Modern systems increasingly focus on meaning.

TF-IDF helped search engines identify important terms.

BM25 improved ranking quality.

Embeddings introduced semantic understanding.

Transformers brought contextual language understanding.

And today, RAG systems combine many of these ideas to create more capable and reliable AI applications.

As AI continues to evolve, understanding the foundations of information retrieval becomes increasingly valuable.

Many of the systems we use every day, from search engines to AI assistants , are built upon principles that researchers have been refining for decades.

Looking back at that evolution provides useful context for understanding where the next generation of retrieval systems might be headed.