The Keyword Search : a detailed look

Keyword search retrieves documents based on word overlap with the user prompt, operating on the principle that documents with a higher frequency of prompt words are more likely to be relevant.

So Key word Search works this way that, you provide the promt, and the documents which have the most keywords from prompt, are selected.

So here no priority word order, and only word presence and frequency matter.

There are different algortithms used in Keyword Search, let see the 2 main of them:-

Keyword Search Algorithms:-

TF-IDF (Term Frquency- Inverse Document Frequency)Framework

Term Frequency (TF): Measures how often a word appears in a document

Inverse Document Frequency (IDF)

• IDF is a weighting metric calculated for every word in the system’s vocabulary to ensure rare, highly descriptive words (e.g., “pizza”) contribute more to the relevance score than common filler words (e.g., “the”).
• Document Frequency:

• Inversion: The document frequency fraction is inverted so that rare words receive higher values than common ones.

• Logarithmic Scaling: To prevent exceptionally rare words from overly dominating the scoring system, the logarithm of the inverted value (log(IDF)) is applied. This dampens the numerical range while preserving a higher weight for rare terms.
• Matrix Compilation: The row for each word in the inverted index is updated by multiplying its Term Frequency by its calculated IDF score, producing a Term Frequency-Inverse Document Frequency (TF-IDF) matrix.

Best Matching 25 (BM25)

BM25 is a retrieval function that ranks a set of documents based on the query terms appearing in each document, regardless of their proximity within the document

• Given a query Q, containing keywords t1,...,tn
• TF(t_i,d) Term Frequency , the number of times that the keyword ti occurs in the document d
• |d| is the length of the document d in words,
• avgdl is the average document length in the text collection from which documents are drawn
• k1 and b are free tunable parameters, usually chosen, in absence of an advanced optimization, as k1∈[1.2,2.0] and b=0.75

Wait wait!!! Didn’t get the formulat let me quote it simply

The formula calculates a relevance score for a single keyword *(t_i in the formula*), relative to a specific document (d in the formula).* The system sums these individual keyword (The sigma function)* scores across all keywords in a prompt to generate a total relevance score for document ranking.

Talking of the free tunanle parameters:

• k1 The degree of term frequency saturation: It handles how rapidly rewards stop for repeated keywords.
• b The degree of document length normalization : It handles how rapidly penalties accumulate for increasing document length.

In production, these parameters are tuned specifically to align with the data stored in the knowledge base.

Just one last thing to mention:

The algorithm penalizes longer documents to account for word volume, but applies diminishing additional penalties as a document grows in length.

What does that means???

Imagine you search for the word “apple”. A 50-page document is naturally going to mention the word “apple” more times than a 1-page document, simply because it contains a massive volume of words.

To prevent long, confusing documents from winning every search just because they are long, BM25 penalizes them. It adjusts the score based on the document’s length compared to the average length of all documents in the database. If a document is unusually long, BM25 dampens its score.

This is the clever part. The penalty isn’t linear. It doesn’t just keep getting harsher and harsher at the same rate forever. Instead, the penalty tapers off (diminishes) as the document gets longer.

Going from a 1-page document to a 5-page document will result in a noticeable penalty because the length jumped significantly, but going from a 100-page document to a 105-page document will barely add any extra penalty at all.