Understanding Stemming and Lemmatization in Natural Language Processing

Introduction

In Natural Language Processing, handling textual data is crucial for various applications such as chatbots, search engines, and sentiment analysis. Developers usually follow a path to clean textual data in order to train an algorithm on it. It doesn’t mean that a model cannot be built on raw textual data; however, remember: Garbage In, Garbage Out. This process includes lowercasing words, removing stopwords, converting words into their base forms, tokenizing words, and much more. However, in today’s article, our goal is to understand the need to turn words into their base forms, the available techniques to do this, when to use them, and how to use them.

Example

• running → run
• ate → eat
• cried → cry

Two widely used methods for this are stemming and lemmatization.

Stemming

It is a naive technique to transform a word into its basic form. This method follows heuristic rules and doesn’t always produce valid results.

Example

• happiness → happi
• running → run
• studies → studi

It doesn’t consider the actual meaning of the word and may produce non-dictionary words. The popular algorithms for this are Porter Stemmer, Lancaster Stemmer, and Snowball Stemmer.

Python Implementation

from nltk.stem import PorterStemmer

ps = PorterStemmer()
words = ["running", "happiness", "studies", "better"]
stemmed_words = [ps.stem(word) for word in words]
print(stemmed_words)

['run', 'happi', 'studi', 'better']

Lemmatization

It is an advanced technique that reduces a word to its dictionary root form while considering context and meaning. Unlike stemming, lemmatization produces valid results.

Example

• running → run
• happiness → happiness
• studies → study

Python Implementation

from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet
import nltk

nltk.download('wordnet')
nltk.download('omw-1.4')

lemmatizer = WordNetLemmatizer()
words = ["running", "happiness", "studies", "better"]
lemmatized_words = [lemmatizer.lemmatize(word, pos="v") for word in words]
print(lemmatized_words)

['run', 'happiness', 'study', 'better']

Lemmatization requires linguistic knowledge and is often more computationally expensive than stemming.

When to Use

Use Stemming When

✅ Speed and performance are a priority.

✅ Applications can tolerate minor inaccuracies (e.g., search engines, keyword extraction).

Use Lemmatization When

✅ Accuracy and meaningful word forms are essential.

✅ Context and grammar need to be preserved (e.g., chatbots, machine translation).

Advantages and Disadvantages

Stemming

✔️ Fast and computationally efficient.

✔️ Simple to implement.

❌ May generate incorrect or meaningless words.

❌ Can lead to over-stemming (losing important distinctions).

Lemmatization

✔️ Produces proper dictionary words.

✔️ Context-aware (better for NLP tasks needing semantic understanding).

❌ Slower than stemming.

❌ Requires additional resources (e.g., POS tagging).

Conclusion

Both stemming and lemmatization are essential steps in preparing text for NLP tasks. Stemming is quick, but sometimes it can result in words that don’t quite make sense. On the other hand, lemmatization is slower, but it ensures that the words we work with are meaningful and accurate, which is important for tasks that need a deeper understanding of language. The choice between them ultimately depends on what you’re working on and the computational resources you have available.

So, how about you? Do you use stemming or lemmatization in your NLP projects? I’d love to hear your thoughts in the comments below!