Text Preprocessing Techniques in NLP

Natural Language Processing (NLP) is a fascinating field that bridges the gap between human language and computers. A crucial step in NLP is text preprocessing, which prepares raw text for analysis by cleaning and structuring it. This blog will delve into three fundamental preprocessing techniques: tokenization, lemmatization, and stemming. Understanding these techniques is essential for anyone looking to work effectively with textual data.

NLP Composition

Why Text Preprocessing is Important

Before diving into the techniques, it’s essential to understand why preprocessing is necessary:

• Noise Reduction: Raw text data often contains noise such as punctuation, special characters, numbers, and irrelevant words that can hinder analysis. Preprocessing helps remove this noise.
• Standardization: Preprocessing ensures that text data is standardized, making it consistent and easier to work with.
• Feature Extraction: Properly preprocessed text allows for more effective feature extraction, which is critical for machine learning and NLP tasks.

Tokenization

What is Tokenization?

Tokenization is the process of splitting text into smaller units called tokens. Tokens can be words, phrases, or even individual characters. Tokenization is the first step in text preprocessing and lays the foundation for further analysis.

Types of Tokenization

• Word Tokenization: Splitting text into individual words.
• Sentence Tokenization: Splitting text into individual sentences.
• Subword Tokenization: Splitting words into smaller units, which is particularly useful for dealing with compound words or words with prefixes and suffixes.

How Tokenization Works

Tokenizers typically use delimiters such as spaces and punctuation to identify token boundaries. More advanced tokenizers can handle complex cases like contractions and hyphenated words.

Tokenization in NLP

As whitespaces often separate the words in a sentence, the easiest way to create tokens from a sentence is to split the sentence by whitespaces. You can traverse the sentence and separate each word by detecting the whitespaces between them.

Python provides a built-in function split() to separate sentences by characters. The function accepts a character argument and uses this character to split the sentences.

#Example 1: Splitting by whitespace:
s = "Hello I am programmer"
lst = s.split()
print(lst)

# Output:
# ['Hello', 'I', 'am', 'programmer']

# ==================================================================

# Example 2: Splitting by comma.
s = "Hello, I am programmer"
lst = s.split(',')
print(lst)

# Output:
['Hello', ' I am programmer']

Example 3: In this example, code reads a text file containing sentences and tokenizes each sentence.

Note that we would use the reviews.txt file for tokenization examples. You can create and save the same file on your local computer. The contents of the file are given below:

reviews.txt ==> " The restaurant has a good staff, good food, 
and a good environment.
It is a good place for family outings. Hospitable staff.
The staff is better than in other places, but the food is okay.
People are great here. I loved this place."

def tokenize(file):
    tok = []
    f = open(file, 'r')
    for l in f:
        lst = l.split()
        tok.append(lst)
    return tok

tokens = tokenize('reviews.txt')
for e in tokens:
    print(e)

output:

['The', 'restaurant', 'has', 'a', 'good', 'staff,', 'good', 'food,', 'and', 'a', 'good', 'environment.']
['It', 'is', 'a', 'good', 'place', 'for', 'family', 'outings.', 'Hospitable', 'staff.']
['The', 'staff', 'is', 'better', 'than', 'other', 'places,', 'but', 'the', 'food', 'is', 'okay.']
['People', 'are', 'great', 'here.', 'I', 'loved', 'this', 'place.']

However, this method has a shortcoming. The split() function does not remove special characters, such as commas, from words, which can reduce the algorithm's efficiency.

Tokenization using the NLTK Library

The NLTK library is yet another popular library for NLP applications. It provides many tools to perform NLP operations, including preprocessing and tokenization. There are two different types of tokenizations that NTLK provides:

• Sentence tokenization: As the name suggests, sentence tokenization breaks the text into meaningful sentences rather than single words. This form can be especially useful in applications such as sentiment analysis, where a paragraph needs to be broken into meaningful sentences to get the sentiment of each sentence.
• Word tokenization: This is the classic form of tokenization, where a text is broken into words. This article mainly focuses on this form of tokenization.

Let us understand each of them using examples. But before writing the code, make sure that the nltk library is installed. Otherwise, you can install it using the pip command from the section above.

Example 1: Sentence tokenization using nltk:

from nltk import sent_tokenize

def tokenize(file):
    tok = []
    f = open(file, 'r')
    for l in f:
        lst = sent_tokenize(l)
        tok.append(lst)
    return tok

tokens = tokenize('reviews.txt')
for e in tokens:
    print(e)

output:

['The restaurant has a good staff, good food, and a good environment.']
['It is a good place for family outings.', 'Hospitable staff.']
['The staff is better than other places, but the food is okay.']
['People are great here.', 'I loved this place.']

Example 2: Word tokenization using nltk:

from nltk import word_tokenize

def tokenize(file):
    tok = []
    f = open(file, 'r')
    for l in f:
        lst = word_tokenize(l)
        tok.append(lst)
    return tok

tokens = tokenize('reviews.txt')
for e in tokens:
    print(e)

['The', 'restaurant', 'has', 'a', 'good', 'staff', ',', 'good', 'food', ',', 'and', 'a', 'good', 'environment', '.']
['It', 'is', 'a', 'good', 'place', 'for', 'family', 'outings', '.', 'Hospitable', 'staff', '.']
['The', 'staff', 'is', 'better', 'than', 'other', 'places', ',', 'but', 'the', 'food', 'is', 'okay', '.']
['People', 'are', 'great', 'here', '.', 'I', 'loved', 'this', 'place', '.']

import nltk 

from nltk.tokenize import word_tokenize, sent_tokenize 

text = "Natural Language Processing (NLP) is a fascinating field." 

word_tokens = word_tokenize(text) 

sentence_tokens = sent_tokenize(text) 

print("Word Tokens:", word_tokens) 

print("Sentence Tokens:", sentence_tokens) 

The shortcoming of this method is that it treats special characters, such as commas and periods, as separate tokens.

Byte-Pair Encoding Tokenizers

The byte-pair encoding tokenizer is a new state-of-the-art tokenizing method. This is a supervised tokenization mechanism, unlike the other rule-based tokenization mechanisms we have seen so far.

The byte-pair encoding algorithm works as follows:

• The algorithm considers all the characters of the text as separate tokens.
• The next step is to find the most common characters that occur together and merge them into a single word.
• This process is repeated until the algorithm finds the number of tokens as needed.

BPE is a data compression algorithm that replaces the most frequent pairs of bytes in a given input with a single, unused byte. It is commonly used to encode text data in natural language processing tasks.

To implement BPE, the algorithm first creates a vocabulary of all the unique bytes in the input. It then iteratively identifies the most frequent pair of bytes in the input that is not already in the language and replaces it with a new, unused byte. This process is repeated until the desired vocabulary size is reached. The resulting vocabulary can be used to encode the input text as a sequence of bytes, with each byte representing a symbol in the vocabulary.

We will use the Hugging Face libraries to implement a Byte Pair Encoding (BPE) tokenizer. If you have not installed libraries, install them by using the pip commands below. Dataset: https://huggingface.co/datasets/mattdangerw/wikitext-103-raw/tree/main

$ pip install huggingface
$ pip install tokenizers

Example: BPE tokenizer

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace
from tokenizers.trainers import BpeTrainer

tk = Tokenizer(BPE(unk_token="[UNK]"))
tr = BpeTrainer()
tk.pre_tokenizer = Whitespace()

f = [f"wikitext-103-raw\wiki.{s}.raw" for s in ["test", "train", "valid"]]
tk.train(f, tr)

tk.save("tokenizer-wiki.json")

Although we can train the tokenizer after we have created a BpeTrainer object, it might produce tokens that have more than one word. To ensure that only a single-word token is generated as output, we used a pre-tokenizer.

After the model is trained, it is better to save it because retraining it each time would take a lot of time and is undesirable.

Now, let us fetch the saved model and tokenize our text data.:

from tokenizers import Tokenizer

tk = Tokenizer.from_file("tokenizer-wiki.json")

f = open('reviews.txt', 'r')
for l in f:
    res = tk.encode(l.strip())
    print(res.tokens)

output:

['The', 'restaurant', 'has', 'a', 'good', 'staff', ',', 'good', 'food', ',', 'and', 'a', 'good', 'environment', '.']
['It', 'is', 'a', 'good', 'place', 'for', 'family', 'out', 'ings', '.', 'H', 'osp', 'itable', 'staff', '.']
['The', 'staff', 'is', 'better', 'than', 'other', 'places', ',', 'but', 'the', 'food', 'is', 'ok', 'ay', '.']
['People', 'are', 'great', 'here', '.', 'I', 'loved', 'this', 'place', '.']

Subword Tokenization

In the previous example, you might have noticed that some of the words in the BPE tokenizer’s output are meaningless. This happens because word pair tokenization operates on the principle of merging frequently occurring words without considering that some words are split, producing meaningless results.

Subword tokenization handles this issue. It works on the principle that frequently occurring words should not be broken into smaller words. Instead, rarely occurring words should be broken down into smaller, more meaningful words.

An advantage of subword tokenization is that the model can efficiently process words it has never seen. Also, the vocabulary size is limited.

The [transformers](https://huggingface.co/docs/transformers/index) library provides the pre-trained [BertTokenizer,](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertTokenizer)which is a subword tokenizer. You can get the model in your code and use it directly. However, first, install the transformers library:

$ pip install transformers

from transformers import BertTokenizer

tk = BertTokenizer.from_pretrained('bert-base-uncased')
f = open('reviews.txt', 'r')
for l in f:
    res = tk.tokenize(l.strip())
    print(res)

output:

['the', 'restaurant', 'has', 'a', 'good', 'staff', ',', 'good', 'food', ',', 'and', 'a', 'good', 'environment', '.']
['it', 'is', 'a', 'good', 'place', 'for', 'family', 'outing', '##s', '.', 'ho', '##sp', '##ita', '##ble', 'staff', '.']
['the', 'staff', 'is', 'better', 'than', 'other', 'places', ',', 'but', 'the', 'food', 'is', 'ok', 'ay', '.']
['people', 'are', 'great', 'here', '.', 'i', 'loved', 'this', 'place', '.']

Sentence Piece Tokenizer

At this point, you are familiar with many different tokenization approaches. However, the problem with all these approaches is that they assume that the words are separated using spaces. However, this might not be true for all languages.

The sentence piece tokenization approach treats the text as a raw stream rather than assuming whitespace to be a separating character. In this way, the whitespace is itself included in the character set.

In the transformers library, you can find a predefined and pre-trained tokenizer named XLNetTokenizer. It is a sentence-piece tokenizer. Let us understand it with an example of code.

Example: Sentence piece tokenizer

from transformers import XLNetTokenizer

tk = XLNetTokenize.from_pretrained('xlnet-base-cased')
f = open('reviews.txt', 'r')
for l in f:
    res = tk.tokenize(l.strip())
    print(res)

output:

['_The', '_restaurant', '_has', '_a', '_good', '_staff', ',', '_good', '_food', ',', '_and', '_a', '_good', '_environment', '.']
['_It', '_is', '_a', '_good', '_place', '_for', '_family', '_out', 'ing', 's' '.', '_Ho', 's', 'pit', 'able', '_staff', '.']
['_The', '_staff', '_is', '_better', '_than', '_other', '_places', ',', '_but', '_the', '_food', '_is', '_okay', '.']
['_People', '_are', '_great', '_here', '.', '_I', '_loved', '_this', '_place', '.']

Lemmatization

Lemmatization is the process of reducing words to their base or dictionary form, known as lemmas. Unlike stemming, which merely chops off prefixes and suffixes, lemmatization considers the context and converts words to their meaningful base forms. For example, the words “running,” “ran,” and “runs” are all lemmatized to the lemma “run.”

Lemmatization

https://www.goml.io/blog/text-preprocessing-techniques-in-nlptokenization-lemmatization-and-stemming

Lemmatization involves several steps:

1. Part-of-Speech (POS) Tagging: Identifying the grammatical category of each word (e.g., noun, verb, adjective).
2. Morphological Analysis: Analyzing the structure of the word to understand its root form.
3. Dictionary Lookup: Using a predefined vocabulary to find the lemma of the word.

For example, the word “better” would be lemmatized to “good” if it is identified as an adjective, whereas “running” would be lemmatized to “run” if identified as a verb.

How Lemmatization works

Techniques in Lemmatization

1. Rule-Based Lemmatization: Uses predefined grammatical rules to transform words. For instance, removing the “-ed” suffix from regular past tense verbs.
2. Dictionary-Based Lemmatization: Looks up words in a dictionary to find their base forms.
3. Machine Learning-Based Lemmatization: Employs machine learning models trained on annotated corpora to predict the lemma of a word.

How Lemmatization Works?