NLP Guide: Stemming & Lemmatization

Lemmatization and Stemming are normalization techniques used in NLP to reduce the word to its’ base form. Stemming uses a heuristic approach to reduce the word to its’ root form. Lemmatization considers the contextual and grammatical properties of the text.

Stemming & Lemmatization on Coporate Lingo

Stemming

Stemming is rules-based process that reduces words to their stem by using predefined rules to systematically strip away prefixes, suffixes or other affixes from words to reduce them to their base form.

Stemming approach tends to reduce the word to a root form without any context. ie the word ‘scheduling’ → ‘schedul’ and ‘replying’ → ‘repli’.

Different stemming algorithms apply different rules and heuristics to strip affixes from words, leading to different results.

Porter stemmer was first proposed by Martin Porter in a 1980 paper titled “An algorithm for suffix stripping.” This has become one of the most common algorithms for stemming in English.

# Porter Stemmer

import nltk
from nltk.stem import PorterStemmer

stemmer = PorterStemmer()
words = ["emailing", "replying", "meeting", "presentations", "reporting", "scheduling"]
stemmed_words = [stemmer.stem(word) for word in words]
print("Stemmed Words:", stemmed_words)
#Stemmed Words: ['email', 'repli', 'meet', 'present', 'report', 'schedul']

Commonly used stemming libraries include: Porter Stemmer, Snowball Stemmer and Lancaster Stemmer.

Lemmatization

Lemmatization relies on determining the intended part-of-speech and the meaning of a word based on its context. It usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.

Lemmatization on the other hand tend to process the word with consideration to the context and grammatical role in a sentence. For example, “scheduling” would be lemmatized to “schedule”, and “replying” would be lemmatized to “reply”.

WordNet is a large lexical database of English that groups words into sets of synonyms called synsets, provides short definitions and usage examples, and records numerous relations among these synonym sets or their members.

#WordNet Lemmatizer
import nltk
from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet

lemmatizer = WordNetLemmatizer()
words = ["emailing", "replying", "meeting", "presentations", "reporting", "scheduling"]
# Function to get the part of speech for lemmatization
def get_wordnet_pos(word):
    """Return the part of speech for a word"""
    tag = nltk.pos_tag([word])[0][1]
    if tag.startswith('J'):
        return wordnet.ADJ
    elif tag.startswith('V'):
        return wordnet.VERB
    elif tag.startswith('N'):
        return wordnet.NOUN
    elif tag.startswith('R'):
        return wordnet.ADV
    else:
        return wordnet.NOUN

lemmas = [lemmatizer.lemmatize(word, pos=get_wordnet_pos(word)) for word in words]
print("Lemmas:", lemmas)
#Lemmas: ['email', 'reply', 'meeting', 'presentation', 'reporting', 'schedule']

Stemming and lemmatization are fundamental linguistic techniques used in NLP to reduce words to their base forms, each serving distinct purposes based on the task requirements.

Stemming is well-suited for scenarios where speed and scalability are prioritized over precise accuracy. It is particularly effective in search systems and text retrieval applications where an approximate matching of words is sufficient. By reducing words to their root forms, stemming enhances the efficiency of these systems while managing large volumes of text.

Lemmatization, on the other hand, is essential for tasks that demand a deeper understanding of language, such as question answering, text summarization, and detailed text analysis. It ensures that words are reduced to their accurate base forms, preserving their meaning and grammatical context.