Text data processing: the most frequent activities

In many data science applications, text fields are used to process natural language (NLP: natural language processing) or to represent documents.

In these cases we cannot work directly with the representation of the raw text: we must first convert the text into a suitable representation based on numbers or number vectors.

We already saw the most common representations: Bag of Words (BoW) , Word Count, N-Gram , TF-IDF in another article https://medium.com/@m.capurso/representations-and-metrics-for-texts-and-documents-f039b52d5b43

We will see now the most common activities.

The most frequent activities

The most common activities on document-type properties are the removal of unnecessary words (stop word removal) and the reduction of words at the roots (stemming and lemming).

The term stop word refers to “very common” words in a language, which do not contain important information, are necessary to complete sentences but which can be safely ignored without losing the meaning of the document.

Some examples of English stop words are “in”, “the”, “is”, “an”. Although stop words refer to “common words”, there is no universal set but there are several reference dictionaries. These dictionaries are used for filtering stop words, for example for applications when training deep learning models for classification. However, it should be noted that the removal of stop words is not recommended in automatic translation and document summaries activities because they help to identify the role of concepts in sentences.

In the same situations, the reduction of words at their roots is used.

As it is known in natural language, words are formed starting from a root with the addition of a prefix, a suffix and an ending. In the words waterfront, waterproof, watercraft, watercolor, waterworks, watertight, watercress, waterborne there is an identical part, the water group. This group is the root, that is, the essential part of the words quoted, the immutable part, which expresses the fundamental meaning. All words that have the same root form a word family.

Let us consider the word act-or. In it, the root act is joined to the suffix or, which is not immutable, but changes when we pass from the singular to the plural ( actor-s ) and to the feminine (act-ress).

This changeable part of the word that indicates change of number (singular or plural), of gender (masculine or feminine) and, in verbs, also of mode, time, person, is called ending or desinence.

While the ending joins directly to the root, in other words like

water-front, water-proof, water-craft, water-color, water-works, water-tight

between the root and the ending you insert a letter or different groups of letters which give each word a different meaning. And this is the same for many languages such as Italian, with the radix acqu and the words

acqu — at — a, acqu — azzon — e, acqu — erell — o, acqu — atic — o, acqu — e — o, acqu — os — o

This letter (or groups of letters) which are found between the root and the ending and which gives each word of the same family a different meaning, is called a suffix. The part of a word formed by the root + suffix is called the theme.

In the following words the root acqu is still found: annacquato, risciacquato, sciacquato.

In them, in addition to the root and the suffix (at) and the ending (o) there are, in front of the root, other groups of letters: ann-, risci-, sci-. These groups of letters that are placed in front of the root of a word to vary its meaning are called prefixes.

Processes known as stemming and lemming replace a word with its root. Stemming makes a rough cut of the words. A Stemmer follows a set of predefined rules for removing prefixes, suffixes and endings. For example: connects, connected, connection can be converted to connect.

Lemmatization is similar to stemming; however, a Lemmatizer always returns a valid word. Stemming uses rules to cut the word, while a Lemmatizer searches for the root word, also called Lemma, for example from WordNet [WN-1].

Furthermore, lemmatization is concerned with converting a word into its basic form; ie English words like “am”, “is”, “are” will be converted to “be”.

The most common applications

What are the most common applications of Data Science applied to texts?

Speech recognition is a technology that allows the computer to convert speech input data into a machine-readable format. There are many fields where speech recognition is used such as virtual assistants, adding text-to-speech to traditional applications, speech translation, voice messaging, search engines.

We are familiar with voice assistants like Alexa, Siri, and Google Assistant, and chatbots built into many websites to help and guide new users. A voice assistant is a piece of software that uses NLP and speech recognition to understand a user’s voice commands and act accordingly. Likewise, chatbots are programs designed to assist users 24/7 and respond appropriately to any questions the user may have.

Most chatbots and virtual assistants have preprogrammed answering systems that follow specific rules and patterns when replying. With the use of artificial intelligence, some voice assistants interact with the user and respond appropriately. With more use, they even improve themselves. Assistants like Siri and Alexa can even converse with you like a normal human being!

There are many software available nowadays that check the grammar and the spelling of the text we type and save us from spelling errors and grammatical errors in our emails, texts or other documents. NLP plays an important role in self-correction and self-preparation.

Gmail categorizes emails into main, social, promotional and spam sections. This is done with the help of text classification, which is an NLP technique.

Through sentiment analysis it is possible to analyze customer reactions, manage social media disputes by eliminating negative comments and obtaining suggestions from a company’s customer base. Expressions such as sarcasm, threat, exclamation are often very difficult to recognize using a traditional program. But with the help of Natural Language Understanding (NLU), which is a branch of NLP, it is possible to recognize different feelings expressed in a user’s comment.

When we search for a product or item on any shopping site, we often see ads for related products on other sites. This type of targeted online advertising is produced with the help of NLP and is known as targeted advertising. The keywords the user searches for are matched to the keywords in the product ad. If they are similar, the user receives an advertisement. This process is called keyword matching.

Automatic translation of posts or websites is also an application of NLP. Platforms like Facebook and Instagram have their own translation software built into the main application.

Social Media Analytics is also an NLP application, which allows companies to estimate their market position and obtain customer reviews by automatically analyzing millions of comments and posts.

Companies often receive thousands of resumes from job seekers. It is difficult for the HR team to go through all resumes and select the best candidate for a single position. NLP makes the job easier by filtering all resumes and selecting candidates with different techniques. Attributes such as place of work, skills, training are considered, and candidates who meet the company’s requirements are selected. It is one of the most practical and impartial ways to filter curricula without interference or human intervention. And it can also recommend other suitable positions for the candidate if there are any vacancies.

Text Summary is an NLP application that helps simplify huge amounts of data in articles, news, research papers available on the Internet. This application is used in Investigative Discovery to identify patterns in report writing, in Social Media Analytics to identify influencers and to classify content into significant or unimportant topics.

Computational linguistics techniques

As we said previously, not all previous representations are able to capture the meaning of a document or support its translation. For more elaborate activities it is necessary to resort to computational linguistics techniques.

These techniques are currently used to translate programming languages (formal languages) and also applied to natural languages. Generally speaking, we talk about translator programs. A translator translates a text written in one language (source code) into another language (object code). A translator has the following architecture:

Architecture of a translator

Grammatical or syntax analysis takes the text and analyzes it from a grammatical point of view. If the text passes the grammar analysis, it goes to the analysis of the meaning otherwise it stops by issuing error messages. The analysis of meaning or semantics takes the text and analyzes it from the point of view of meaning. If the meaning is correct, it will proceed, otherwise it will stop by issuing error messages. In the final step, the object code is generated.

The language is described in its grammatical aspects through rules. Here is an example for the English language:

<English sentence> :: = <subject> <verb> <object complement>

<subject> :: = I

<subject> ::= you

<subject> :: = he

<verbo> ::= have

<verb> ::= has

<object complement> :: = a book

<object complement> :: = a pen

A grammar is a succession of rules. The rules consist of two parts separated by the symbol “:: =“ (“is defined by”). On the left we have the grammatical concept / category we are defining. On the right we have the “how” we are defining it. A grammar allows you to recognize a sentence (if correct) and to build an internal representation in the form of a tree.

Syntax tree of an english sentence

The grammar analyzer constructs the tree and passes it to the semantic analyzer which enriches it with meaning by associating to each node its properties such as the root word, the gender (masculine, feminine or neuter), the number (singular or plural), the person of the verb (first / second / third singular / plural), the mode (indicative, subjunctive …) and the tense (present, past, future …) [SEM -1].

The semantic analyzer enriches each node with meaning by identifying the entities (or objects), verbs, classes or concepts to which the entities belong, the relationships between entities and also carries out a process of disambiguation by operating on the homonyms (same term, meanings completely different), polysemies (same term, similar but different meanings), synonyms (different terms, same meaning), on contraries (terms that express opposite meanings) and on rhetorical figures of a semantic type (such as, for example, the antithesis, the hyperbole , the metaphor) [SEM-2] by searching for terms for example on WordNet [WN-1].

In the end, the tree somehow represents the meaning of the input text. But there is not only the tree representation of semantics. There are others in the literature:

• First order predicates logic
• Semantic networks
• Frames
• Conceptual Dependency (CD)
• Rules-based Architectures
• Case-based Grammars
• Conceptual Graphs

Among these types of representation, only Conceptual Dependence has the ability to represent the meaning of sentences without resorting to conventions external to representation, and this is because it was born explicitly for NLP applications.

Conceptual dependency theory (CD) was invented by Roger Schank [CD-1] at Yale University in the 1970s as a cognitive and psychological approach to AI. The purpose of the CD is to define a specific language to represent the meaning of a sentence without ambiguity.

The CD is based on events and actions. Each event (if applicable) has an ACTOR, an ACTION played by the actor, an OBJECT on which it performs the action, a DIRECTION in which that action is oriented.

Schank and Abelson define 11 primitive actions that they originally hoped would be sufficient to represent arbitrary sentences:

• ATRANS: Abstract transfer, e.g. give
• PTRANS: Physical transfer of position, eg. to go
• PROPEL: Application of physical force to an object, eg. push
• MOVE: Movement of a part of the body by its owner, eg. to hit.
• GRASP: Grasping an object by an actor
• INGEST: Ingestion of an object by an animal, eg. eat or drink
• EXPEL: Expulsion of something by an animal, eg. spit or cry or bleed
• MTRANS: Transfer of mental information, eg. tell
• MBUILD: Creation of new information, eg. mental construction or decision
• SPEAK: Production of sounds, eg. to say. Instrumental act.
• ATTEND: Focus the sense organ towards the stimulus, eg. listen or watch.
• DO: Any action, used for unknown actions

An event graphically appears as:

Event according to CD

For example, the phrase “Mary fell” becomes:

Representation of “Mary fell” with CD

Experimenting with documents

We can now experiment with Python what we have just seen, by selecting and installing the library called NLTK, dedicated to the representation and processing of document corpora.

A document corpus is a set of text documents associated with properties. With NLTK you can read and create corpuses or access a set of available corpuses, listed at https://www.nltk.org/nltk_data/ .

But it is also possible to build corpora from a series of online services such as Wikipedia, Twitter, The Guardian, the New York Times, Pubmed using their respective APIs.

After running the pip utility and installing the nltk module:

pip3 install nltk

you can download a corpus from those available, with the Python instructions:

import nltk

nltk.download(“nomecorpus”)

from nltk.corpus import nomecorpus

The NLTK corpora

NLTK includes a small selection of texts from the Project Gutenberg electronic text archive, which originally contains about 25,000 free e-books, hosted at http://www.gutenberg.org/. For example, the following instructions import the selection from Project Gutenberg and list the corpus files:

import nltk

nltk.download(“gutenberg”)

from nltk.corpus import gutenberg

print(gutenberg.fileids())

The files from corpus gutemberg

We can now visualize the number of characters and words of the novel Emma by Jane Austen, which is part of the corpus:

num_chars = len(gutenberg.raw(‘austen-emma.txt’))

num_words = len(gutenberg.words(‘austen-emma.txt’))

print(num_words, num_chars)

Note how gutenberg.raw(‘austen-emma.txt’) produces the characters of the raw document, gutenberg.words(‘austen-emma.txt’) produces the words of the document, and gutenberg.sents(‘austen-emma.txt’) produces the sentences of the document.

Each corpus is different from the others not only in terms of documents of course but also in terms of associated variables. For example, the Reuters Corpus contains 10,788 news documents for a total of 1.3 million words. The documents have been classified into 90 topics and grouped into two sets, called “training” and “test”. This division is for the training and testing algorithms that automatically detect the topic of a document. So the topic is a variable.

There are thus different types of corpus: the simplest type of corpus is a collection of isolated texts without any particular organization; some corpora are structured into categories such as genre (Brown Corpus); in some others the categorizations overlap, such as thematic categories (Reuters Corpus); other corpora represent the use of language over time (Inaugural Address Corpus).

Corpus types

The basic operations are:

Viewing a Word Cloud

You can view the relevance of words in the form of Word Cloud using NLTK and the wordcloud library (which must be installed) with the program:

The program takes the bare text of Jane Austen’s novel Emma, divides it into words, removes punctuation marks and very common words (stop words), sorts the words by frequency, and displays the word cloud.

In more detail, the program downloads (if not already downloaded) the gutenberg, stopwords and punkt_tab corpuses and imports them; then it takes the bare text of the novel from the gutenberg corpus and tokenizes it into a sequence of words.

It then scans the words that are alphabetic and converts them to lowercase. Then it keeps the ones that are not stopwords.

Then it calculates the Term Frequencies of the words, draws the Word Cloud according to the Term Frequencies with a white background and displays the image with matplotlib, without axes and bilinear magnification.

Word Cloud

Please remember to install beforehand with

pip install wordcloud

the wordcloud library module. In some cases reported by users, wordcloud when displaying the result highlights an error message that indicates the impossibility of using the font of the characters. This message seems to occur in different operating systems and versions of the library. It has been suggested to update the Pillow program with the following commands:

pip install — upgrade pip

pip install — upgrade Pillow

It happened to us on Linux, and the solution worked.

Note how the code

freqdist = nltk.FreqDist(words)

calculates the word count of words. You can then view the frequency graph of the first thirty words

freqdist.plot(30, cumulative=True)

plt.show()

or print the frequencies of the three most frequent words with the instruction

print(freqdist.most_common(3))

Using a user text corpus

It is possible with NLTK to create a corpus with your own documents present in a folder containing documents in .txt format and have a corpus ready for further processing. The following instructions:

from nltk.corpus import PlaintextCorpusReader

corpus_root = ‘/usr/share/dict’

wordlists = PlaintextCorpusReader(corpus_root, ‘.*txt’)

take files in usr/share/dict that end with .txt and consider them as a corpus. Be careful with the second parameter, which contains a regular expression and MUST NOT start with either * or + , otherwise an error will occur.

We suggest that in the previous Word Cloud viewer program you can transform the word/wordcount table into a pandas dataframe and save it in csv format with the code:

import pandas as pd

df=pd.DataFrame(freqdist.items(), columns=[‘word’, ‘wordcount’])

df.to_csv(“wordcount.csv”)

So let’s try to put the last suggestions all together.

The following Python program, which we will call mycorpus.py, assumes that there is a user corpus made of text documents with names ending in .txt , inside the folder /home/capurso/corpus_test .

The program, after the necessary library imports, downloads the corpus with the stopwords, opens the user corpus and writes the names of the files that constitute it and then calculates and writes the number of characters and words that constitute the file articolo_privacy_ita.txt.

Then the text is tokenized into words and becomes a list of words. These words are then converted to lowercase, the Italian stopwords are removed, the frequencies (the wordcount) of the words are calculated and the cumulative graph of the most frequent words is displayed.

Finally the word list with the wordcount is converted into a pandas dataframe, a column with the term frequency is added and it is saved as a csv file.

Cumulative plot of most frequent words

Stemming, lemming e POS tagging with NLTK

NLTK also has stemming capabilities. The following tokenizes and stems the text of the novel Emma from the Gutenberg corpus.

from nltk.stem import PorterStemmer

from nltk.tokenize import word_tokenize

text = gutenberg.raw(‘austen-emma.txt’)

words = nltk.word_tokenize(text)

votes = PorterVotes()

stemmed_words = [stemmer.stem(word) for word in words]

First 30 words before and after stemming

POS (Part of Speech) is a term that refers to the roles that words play when used together in sentences, such as a verb or a noun. Part of speech tagging, or POS tagging, is the task of labeling words in text based on their part of speech. The following code downloads a corpus used for POS tagging in English and performs POS tagging:

nltk.download(‘averaged_perceptron_tagger_eng’)

pos_words=nltk.pos_tag(words)

POS tagging for first 30 words in Emma

As you can see, each word is labeled with its role in the speech using tags. For example, JJ means adjective, NN means noun, DT means definite or indefinite article, VBZ verb. To get the meaning of the tags, just write the following code in the Python interpreter, which downloads a help corpus and invokes an online help:

nltk.download(‘tagsets_json’)

nltk.help.upenn_tagset()

NLTK is also capable of lemmatization. The following downloads the wordnet corpus and lemmatizes a text:

from nltk.stem import WordNetLemmatizer

nltk.download(‘wordnet’)

lemmatizer = WordNetLemmatizer()

lemma_words = [lemmatizer.lemmatize(word) for word in words]

In summary, the following program nltk_stemmer.py after importing the appropriate libraries, tokenizes, stems, lems and POS tags the work Emma by Jane Austin from the Gutenberg corpus. It then writes the first thirty original, stemmed and lemmed words and their POS tag.

…

To learn more, see https://mybook.to/V0QAMPh

and https://mybook.to/dKB8

DS-5: Mario A. B. Capurso, Data Science and Engineering — A learning path — Volume 1 — Methodological Aspects, Data Acquisition, Management and Cleaning, Analysis and Visualization , 2022

DS-6: Mario A. B. Capurso, Data Science and Engineering — A learning path — Volume 2 Exploratory Data Analysis, Metrics, Models , 2022

DS-7: Mario A. B. Capurso, Data Science and Engineering — Volume 3 Advanced Learning, Deployment and Deep Learning, 2023