Smoothing (NLP 30-day challenge)

day 7

Smoothing and Backoff are essential techniques used in NLP to handle the data sparsity problem in statistical language models. They prevent models from assigning a zero probability to unseen word combinations, ensuring models generalize to new inputs.

Smoothing

Smoothing (or discounting) shifts a small fraction of probability mass from frequently occurring events and redistributes it to rare or unobserved events so that every possible n-gram has a non-zero probability.

When building language models (like N-gram models), we calculate the probability of words appearing together based on a training dataset. However, language is infinite, and our training data is finite. If your model encounters a phrase it has never seen before, a naive model calculates its probability as exactly zero. Because probabilities are multiplied together, a single zero will completely wipe out the probability of an entire sentence, ruining translation, speech recognition, or text generation tasks.

Smoothing redistributes a small amount of probability mass from frequent words to unseen words, ensuring that no event ever has a probability of zero.

The Core Problem: The Zero-Probability Dilemma

To understand the math, we look at N-gram Language Models. An N-gram is a sequence of N words. A Bigram (N=2) calculates the conditional probability of a word w_i given the previous word w_i-1:

The Breakdown Example

Suppose your vocabulary has 10,000 words. Total possible bigrams = $10,000² = 100 million.

If your training corpus is small, you might only observe 1 million unique bigrams. The other 99 million bigrams will have a count of 0.

If your model encounters the unseen sentence: “I want to eat radioactive sushi.”

Because C(“radioactive”,“sushi”) = 0

Multiplying this into the total sentence probability results in:

The model declares this sentence mathematically impossible. Smoothing fixes this.

Laplace Smoothing (Add-One)

The earliest and simplest smoothing technique is Laplace Smoothing. It acts as if we saw every possible N-gram exactly one more time than we actually did.

The Math

For a vocabulary size V and a unigram count, the smoothed probability becomes:

Example

Imagine a tiny corpus: “cat mat cat rat”.

• Total words N = 4
• Vocabulary V = 3 (cat, mat, rat)
• Actual count of dog = 0.

Without smoothing:

With Laplace smoothing: