Understanding NLP: From Tokenization to Named Entity Recognition

5 STEP of text processing in summary

Tokenization: Splits the text into a list of individual words.

Stemming: Reduces words to their root form (e.g., converting “running” into “run”).

Lemmatization: Reduces words to their base form with options for different parts of speech (e.g., noun, verb, adverb).

Stopwords: Words that are not that usefull and repeteting (the,a,in)

Speech Tagging: Assigns grammatical categories (e.g., noun, verb) to word

Named Entity Recognition: Recognizes names, dates, amounts in the text.

1. Tokenization

Explanation: Tokenization is the process of splitting text into smaller units, such as words or phrases, which are called tokens.

Code Example:

from nltk.tokenize import word_tokenize
text = "Hello, how are you?"
tokens = word_tokenize(text)
print(tokens)

output ['Hello', ',', 'how', 'are', 'you', '?']

2. Stemming

Explanation: Stemming involves reducing words to their root form by stripping suffixes or prefixes, usually using a predefined set of rules.

Code Example:

from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
stemmed_word = stemmer.stem("running")

output:run

3. Lemmatization

Explanation: Lemmatization is the process of reducing a word to its base or dictionary form, considering the context of the word, like verb tense or word type.

Code Example:

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
lemmatized_word = lemmatizer.lemmatize("better", pos='a')

output: good
#if you will pass sentence pass it as a splitted words

4. Stopwords

Explanation: Stopwords are common words such as “the,” “is,” and “in,” that are usually filtered out during text processing because they carry little meaningful information.

Code Example

from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
filtered_words = [word for word in word_tokenize("This is a simple sentence.") if word.lower() not in stop_words] 

ouput: ['simple', 'sentence', '.']

5. Part-of-Speech Tagging (POS Tagging)

Explanation: POS tagging involves assigning a part of speech to each word in a sentence (e.g., noun, verb, adjective) to understand the grammatical structure.

Code Example:

from nltk import pos_tag
words = word_tokenize("The quick brown fox jumps over the lazy dog")
tagged_words = pos_tag(words)

output:[('The', 'DT'), ('quick', 'JJ'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'VBZ'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')]
'DT' - Determiner
'JJ' - Adjective

6. Named Entity Recognition (NER)

Explanation: Named Entity Recognition (NER) identifies and categorizes entities in the text into predefined categories like names of persons, organizations, locations, etc.

Code Example:

from nltk import ne_chunk
from nltk import pos_tag
words = word_tokenize("Apple is looking to buy a startup in the UK")
tagged = pos_tag(words)
named_entities = ne_chunk(tagged)