From Slang to Standard: Text Normalization Techniques with Python

What is Text Normalization? Text normalization is the process of converting text into a consistent and standard format. It’s like cleaning up and organizing your text so that it’s easier to work with, especially in tasks like text analysis or processing.

Why is it Important?

Imagine we’re organizing a huge pile of documents. Some documents use short forms or slang, and others use full words. If we’re trying to find specific information, it would be much harder because everything isn’t in a uniform format. Text normalization helps by:

• Making Text Consistent: It ensures that different variations of the same word or phrase are treated the same way.
• Improving Accuracy: For applications like search engines or chatbots, it improves accuracy because the system can understand variations of words or phrases as the same thing.
• Facilitating Analysis: It makes it easier to analyze text by reducing complexity and inconsistencies.

Example

Imagine we’re analyzing customer feedback from emails. Here’s how text normalization helps:

Handling Contractions

In emails, people often use contractions like “don’t” or “isn’t.” For instance, we might get feedback saying:

“I don’t think it’s a good idea.”

If we don’t normalize this text, we might miss that “don’t” and “it’s” should be interpreted as “do not” and “it is,” respectively, making it harder to analyze the sentiment.

By expanding contractions, we convert:

“I don’t think it’s a good idea.” to “I do not think it is a good idea.”

Now, it’s easier to analyze because every contraction is expanded to its full form, making the text uniform and easier to interpret.

Expanding Abbreviations

Consider an email that uses abbreviations like “Dr.” and “Co.” We might get feedback like:

“Dr. Smith and Mr. Johnson went to the Co. headquarters.”

If we don’t normalize this text, we might not realize that “Dr.” stands for “Doctor” and “Co.” stands for “Company.” Expanding these abbreviations makes it:

“Doctor Smith and Mister Johnson went to the Company headquarters.”

This makes the text clearer and more straightforward, especially if we’re aggregating information or trying to understand the feedback fully.

import re

# contraction mapping
contraction_mapping = {
    "don't": "do not",
    "can't": "cannot",
    "won't": "will not",
    "isn't": "is not",
    "aren't": "are not",
    "wasn't": "was not",
    "weren't": "were not",
    "it's": "it is",
    "i'm": "i am",
    "you're": "you are",
}

# abbreviation mapping
abbreviation_mapping = {
    "Dr.": "Doctor",
    "Mr.": "Mister",
    "Mrs.": "Mistress",
    "St.": "Saint",
    "Co.": "Company",
}

def expand_contractions(text, contraction_mapping):

    contractions_pattern = re.compile('({})'.format('|'.join(contraction_mapping.keys())), re.IGNORECASE|re.UNICODE)
    def replace(match):
        return contraction_mapping[match.group(0).lower()]
    return contractions_pattern.sub(replace, text)

def expand_abbreviations(text, abbreviation_mapping):

    abbreviations_pattern = re.compile('({})'.format('|'.join(abbreviation_mapping.keys())), re.IGNORECASE|re.UNICODE)
    def replace(match):
        return abbreviation_mapping[match.group(0)]
    return abbreviations_pattern.sub(replace, text)

text = "I can't believe it's already September! Dr. Smith and Mr. Johnson went to the Co. headquarters."
text = expand_contractions(text, contraction_mapping)
text = expand_abbreviations(text, abbreviation_mapping)
print(text)

I cannot believe it is already September! Doctor Smith and Mister Johnson went to the Company headquarters.