What is NLP and Why is it Needed?

Photo by zhendong wang on Unsplash

NLP stands for Natural Language Processing, and it is an overarching field providing a framework to enable computers to understand and process human language. Human language is very complex, and it has a lot of context dependence on surrounding words, the situation in which it is spoken, and it depends on the speaker’s background. Also, the same words can have different meanings in different contexts, and this leads to ambiguity, which creates challenges for computers to exactly understand what humans meant to say. Also, human language is rich and expansive in terms of communication nuances such as emotions, accent, and tone, cultural and regional differences like the different dialects, grammatical rules and errors, misspellings, and pronunciations, etc. All this can make it very difficult for computers to successfully understand, interpret, and further process the human language for various applications. Computers try to emulate the human language learning process through NLP.

Conceptual Model of NLP

Figure 1. Conceptual model of NLP

Humans have an innate ability to learn their native language, understand the language through interplay of processing of audio and visual sensory inputs, learning word patterns through social and environmental interactions, mimicking sounds, and iteratively building an understanding of the combination of words that are often used together, the grammatical rules, and the intent and context in which words are used in the sentences. In human’s this process of learning a language starts even before or soon after their birth, where their knowledge and understanding of the language keeps expanding gradually with more and more interactions taking place with their family and the world.

The human brain is constantly working on building new relationships between the existing knowledge and the new knowledge, where the initial understanding and learning can be overgeneralized, and it gets refined over time through continuous feedback and updates, helping continuously improve the ability to understand and communicate in a complex language.

As babies start by recognizing single word patterns, and as they grow to be toddlers, they expand their language understanding to a combination of two to three words, and slowly and through continuous learning, correction, and adaptation, the young children develop an understanding of a language vocabulary, and the context in which the words are used. As they become adults, they leverage their knowledge of expansive language vocabulary, understand the meaning of similar words used in different contexts, the semantic and syntactic information, and they can communicate fluently. Humans utilize their working memory for processing the information received through their sensory inputs and the information recalled from their long-term memory, whereas once the information is processed and learnt, they use the long-term memory to store the new information while also making connections with the old information already stored in the memory.

Similar to human language learning, computer language models also followed an evolutionary cycle, where early ML and Deep learning models were designed to predict the next character, then further enhanced to predict a sequence of characters i.e a word, and then further developed to understand the context and generate combinations of two to three words to for a word sequence, and finally they could work simple to complex sentences.

What are the main functions of NLP?

There are three main functions of NLP:

1. Natural Language Understanding
2. Natural Language Generation
3. Natural Language Manipulation

Natural language understanding is where NLP focuses on understanding the context of words in sentences. It helps in understanding the grammatical structure of the sentences, and the contextual relationships between the words(tokens) and phrases (semantics). It mainly involves tasks such as understanding the underlying intention or objective of the sentence is intent recognition, and it also identifies and classifies predefined categories such as names, locations, dates, organizations, numerical, or monetary values in the text, and this is known as named entity recognition(NER).

In the example sentence “I want to book a flight from Boston to Houston”. The NLU model would identify the intent as “book a flight”, and it would identify the entities (predefined categories) as “Boston” and “Houston”. The NLU model utilizes the list or database of entities that we have provided to it to identify the words in the sentences as entities. NLU models enable machines to understand, interpret, and respond to human language. They help analyze the interrelationship of words in the input text and grasp the meaning of words and the overall sentence, and also help determine the sentiments of emotions, tones, and attitudes expressed in the sentence. NLU is powered by machine learning (classifiers such as logistic regression, naive bayes, support vector machines(SVM), decision trees and random forests trees) and deep learning algorithms/models (recurrent neural network(RNN), long short-term memory(LSTM), convolutional neural network(learn and predict n-grams), the latest state-of-the-art transformer network(uses attention mechanism to capture long-term dependencies in the text and are more effective than the RNN in maintaining the contextual information in long sentences), and pre-trained transformer models(WordNet, BERT, GPT, RoBERTa, distilBERT )that are trained on massive data sets of text-based and spoken language.

Natural Language Generation (NLG) is where NLP focuses on producing human-readable text from the data. It comprises of capability to transform structured and non-structured data into coherent, grammatically correct, and contextually relevant text. NLG algorithms utilize a combination of pre-defined categories or templates, leverage the linguistic knowledge, grammatical rules, and heuristics to implement rule-based extraction, rewards, and selection(POS tagging, Parsing, Categorizing based on predefined rules), and mathematical and statistical probabilistic models for generating text. Machine and deep learning models are used with NLG to help improve the quality and sophistication of text generation.

Natural Language Manipulation(NLM) is where NLP focuses on pre-processing data (loading data and getting necessary libraries, exploratory data analysis, addressing class imbalance if it exists(SMOTE, Class Balance Weights, Focal Loss, Text Augmentation, Ensembling Model Techniques), creating word tokens, encoding labels for converting text in numerical format, defining model architecture, creating word embeddings, attention masks in case of transformer models, data loaders for efficient batch processing, model training, saving, and evaluation, etc.) and transforming the information learnt and generated into another format. It leverages NLP capabilities such as NLU and NLG models to accomplish tasks such as language translation (for example: Russian to English, French to English, or vice versa), text summarization, spelling and grammar correction, intent and entity identification and extraction, enabling conversations using text and speech language modalities used in personal assistants and chatbots, etc.