TEXT TALES: rule-based aspect-based sentiment analysis and evaluation on travel literature

Text tales blog image

👤 **WHO: **this blog is part of a series on TECH TALES — an initiative by the Ghent Center for Digital Humanities (GhentCDH) and funded by the Computational Literary Studies (CLS) project.

🤖 WHAT (TL;DR): This blog post will show you one of the many ways to build an aspect-based sentiment analysis system using spaCy, NLTK’s synsets, SenticNet in Python. This is a rule-based approach combined with off-the-shelf tools. We will apply this to an example set of English travelogues from the 19th century. The task is cut up in three subtasks: 1) aspect extraction, 2) opinion word extraction and 3) sentiment scoring. Each task is evaluated by transforming the annotations to BIO-labels and calculating evaluation metrics with the Nervaluate package.

🤔 **WHY: **we want to build bridges between the (digital) humanities and NLP by experimenting with NLP-approaches for literary-historical data sources, sharing experiences, insights, resources and useful code snippets.

🌍 WHERE: the Jupyter Notebooks connected to these blog posts can be found on GhentCDH’s GitHub Page.

🧠 HOW: to follow along with this blog post, you need to have at minimum an intuitive understanding of the following topics and software:

• spaCy
• NLTK
• SenticNet
• POS-tagging
• Named Entity Recognition (NER)
• Sentiment analysis
• NER evaluation practices
• BIO-labelling
• Lemmatization
• Tokenization

1. Load in our example data

# Load in our example texts
!git clone https://github.com/TessDejaeghere/example_data_CLS.git

This is some example data you can download from our GitHub to play around with. The dataset copies into a folder in your Jupyter Notebook, and includes Dutch, French, English and German travel texts from the 19th century, as well as gold standard annotations we will use for evaluation in the next steps (gs_aspects.csv, annos_en_sentwords.csv).

2. Extract aspects and evaluate approach

In this section, we will apply our aspect extraction approach to the chunks in our gold standard annotations, and compare the output of our system to the manual annotations.

1. We load in a partition of our gold standard dataset (gs_aspects.csv).
2. Extract noun chunks as aspects. These can contain nouns and proper nouns. We exclude common English stop words as defined by spaCy.
3. We transform the extracted aspects to IOB-labels, and evaluate the outcome using the Nervaluate package.

🔍 Tool note: Nervaluate is a useful package to get a more detailed look at the quantitative evaluation of a Named Entity Recognition model. Not only do you create a system tailored to your goals, the way you evaluate it numerically can paint a completely different picture of your model. Check out the different metrics calculated by Nervaluate if you’re interested!

gold = pd.read_csv("example_data_CLS/gs_aspects.csv")

The spaCy package includes a Tokenizer class. This class is used to split a text into smaller linguistic units called Tokens. In our case, the gold standard data is Tokenized in a specific way. In the end, we need the lenghts of our IOB-labels for both the gold standard annotations and the model annotations to be of the same length in order to evaluate it. This is why the tokenization approach we apply needs to match that of the gold standard approach.

Example of the gold standard data and model output, which need to be of the same length

nlp.tokenizer = Tokenizer(nlp.vocab, token_match=re.compile(r'\S+').match)

Now our Tokenizer is ready, we’ll write a couple of functions to extract the aspects and transform them into BIO-labels!

def get_aspect_labels(txt):

  doc = nlp(txt)
  chunks = [chunk for chunk in doc.noun_chunks for tok in chunk if tok.pos_ in ["NOUN", "PROPN"] and tok.lemma_ not in nlp.Defaults.stop_words] #return chunk if it contains a noun/proper noun and is not made up of stop words

  tokens = ["O" for tok in doc]

  for chunk in chunks:
    len_chunk = len([tok for tok in chunk])

    if len_chunk > 1: #if the chunk has more than 1 token
      indices = [tok.i for tok in chunk] #indices of the chunk tokens
      tokens[indices[0]] = "B-aspect" #the first element of the chunk = B-aspect

      for ind in indices[1::]:
        tokens[ind] = "I-aspect" #the other elements of the chunk = I-aspect

    else: #if the chunk just has one element, = B-aspect
      indices = [tok.i for tok in chunk]
      tokens[indices[0]] = "B-aspect"

  return tokens

def get_tokens(txt):
  return [tok for tok in doc]

def get_chunks(txt):
  doc = nlp(txt)
  chunks = [chunk for chunk in doc.noun_chunks for tok in chunk if tok.pos_ in ["NOUN", "PROPN"] and tok.lemma_ not in nlp.Defaults.stop_words] #return chunk if it contains a noun/proper noun and is not made up of stop words

  return chunks

# Apply our functions to the sentences!
gold["chunks"] = gold["sentence"].apply(get_chunks)
gold["predicted_label"] = gold["sentence"].apply(get_aspect_labels)

# Here we make sure that the "labels" result is interpreted as a list.
gold["labels_y"] = gold.labels_y.apply(lambda x: ast.literal_eval(x))

The result of our operation should look like the image below! We have now created a column with the words, the sentence, the gold standard labels lists in the labels_y, the chunks, and our predicted labels in predicted_label.

Example output of our BIO-labelling code applied to our gold standard text chunks

In a next step, we’ll take the output we produced in the labels_y and predicted_label columns, transform them to list elements and apply Nervaluate’s Evaluator class to it.

true = gold["labels_y"].to_list()
predicted = gold["predicted_label"].to_list()

evaluator = Evaluator(true, predicted, tags=['aspect'], loader="list")
results, results_by_tag = evaluator.evaluate()

Output of our quantitative evaluation

You’ve now quantitatively evaluated your very own rule-based aspect extraction model! How you read these results and whether they are good enough depends entirely on your purpose.

3. Opinion word extraction

In this step, we’re extracting the opinion words using spaCy’s POS-tagging module. We assume that opinion words will be expressed mainly as adjectives, coordinating conjunctions (CCONJ) and subordinating conjunctions, which we extract from the noun phrases.

# Load in our gold standard sentiment annotations
gold_opinion = pd.read_csv("example_data_CLS/annos_en_sentwords.csv")

matcher = Matcher(nlp.vocab)

However, these wouldn’t capture everything! We’re initializing a Matcher class from spaCy, and adding some syntactic patterns to it which we assume will be interesting (such as auxiliary phrases!). We’re also interested in logging negations, so we can take these into account when calculating a sentiment score for the negated words. 🧐 If words are negated, we retrieve its antonym through WordNet’s SynSets and calculate the sentiment score based on this antonym. For example, in an auxiliary phrase like “The house is not beautiful”, “not beautiful” would be extracted — and “beautiful” will be replaced by its most likely antonym.

pattern = [ [[{"POS": "AUX"}, {"DEP": "neg", "OP": "*"}, {"POS": "ADV", "OP": "*"}, {"POS": "ADJ"}]] ] #is + (not) + (very) + nice
pattern_neg_conj =  [ [[{"POS": "AUX", "OP": "{0}"}, {"POS": "ADV", "OP": "{0}"}, {"POS": "ADJ", "OP": "{0}"}, {"POS": "CCONJ"},  {"DEP": "neg"}, {"POS": "ADJ"}]]  ] # [is + (very) + nice]DONOTMATCH + but + not + warm

matcher.add("aux_adv_adj", pattern[0])
matcher.add("negations_cconj", pattern_neg_conj[0])

def match_auxiliary_phrases(doc):
  spans = []

  matches = matcher(doc)

  for match_id, start, end in matches:
    span = [x for x in range(start,end)]
    spans.append(span[1::]) #auxiliary verb 'is' or "cconj" "but"/... = not important

  return spans

The opinion_extractor takes in a text, extracts the relevant constructions we have defined, and transforms the result into BIO-labels. This is the same procedure we’ve applied for the aspect extraction step!

# Fetch adjectives in noun chunks

def opinion_extractor(txt):
  doc = nlp(txt)
  tokens = ["O" for tok in doc] #initialize token list, length of list = all "O"s

  ### AUXILIARY PHRASES ###

  #fetch auxiliary constructions (the house *is very nice*)
  auxiliary_spans = match_auxiliary_phrases(doc) #get indices of spans auxiliary sentences
  for span in auxiliary_spans:
    tokens[span[0]] = "B-opinion"
    for span_ind in span[1::]:
      tokens[span_ind] = "I-opinion"

  ### NOUN CHUNKS ###

  #return chunk if it contains a noun/proper noun and is not made up of stop words
  all_modifier_indices = []
  for chunk in doc.noun_chunks:
    #print(chunk)

    for tok in chunk:
      if tok.dep_ == "compound":
        continue
      elif tok.pos_ in ["ADJ", "CCONJ", "SCONJ"] and tok.lemma_ not in nlp.Defaults.stop_words:
        modifier_indices = []
        modifier_index = tok.i
        modifier_indices.append(modifier_index) #save index of adjectives looped over

  # Fetch intensifiers by navigating children (adapted from https://towardsdatascience.com/aspect-based-sentiment-analysis-using-spacy-textblob-4c8de3e0d2b9)
        for child in tok.children:
          if child.pos_ == "ADV":
            intensifier_index = child.i
            modifier_indices.append(intensifier_index)
          elif child.dep_ == "neg": #fetch negations to account for negations
            intensifier_index = child.i
            modifier_indices.append(intensifier_index)

        all_modifier_indices.append(sorted(modifier_indices))

  for mod_pair in all_modifier_indices:
    if len(mod_pair) > 1:
      tokens[mod_pair[0]] = "B-opinion"
      for opinion_id in mod_pair[1::]:
        tokens[opinion_id] = "I-opinion"

    else:
      tokens[mod_pair[0]] = "B-opinion"

  return tokens

Let’s apply this code to our column _sentence_text, which contains the entire chunk.

gold_opinion["opnion_labels"] = gold_opinion["_sentence_text"].apply(opinion_extractor)

Result of the opinion word label extraction

As you can see, an opinion word has now been added to our sentences.

4. Sentiment scoring and evaluation

To evaluate our sentiment analysis approach, we’ll apply a model to our gold standard opinion words and compare the scores to the gold standard ones. Let’s see how we can transform these labels into sentiment scores using SenticNet and SynSet.

First, we’ll create a function to fetch the antonym of the negated opinion words. If a negation is found, we’ll return the inverse of the resulting score. Let’s create a couple of functions:

1. fetch_antonym: fetches the antonym of a word in SynSets.
2. sentiment_scorer: calculates a sentiment score for each token, and calls to the antonym fetcher if a negated construction is found. After that, it calculates the mean of the opinion word scores and pushes it through a Sigmoid function.
3. sig pushes the result of the polarity_scores function through a Sigmoid to map it to a 0–1 float range to normalize them. We do this because the SenticNet’s scoring range is -1:1, and we found a 0:1 range easier to work with.
4. polarity_label is a threshold we manually set. Based on the Sigmoid scores, we return a polarity label on a 1–5 scale.
5. add_mean_polarity_score calculates the mean polarity score.

# Apply sentic scorer to all of the words in the gold standard data
# if negation: turn word into antonym using wordnet OR swap the scores
# return mean

def sentiment_scorer(text):
  doc = nlp(text)

  text = [tok.lemma_ for tok in doc if not tok.is_punct]

  ## Convert negations to antonyms ###

  negation_tokens = [tok for tok in doc if tok.dep_ == 'neg']
  negation_head_tokens = [(token.head, token.head.i) for token in negation_tokens] #get dependency head (= negated word)

  inverse_scores = []
  for w in negation_head_tokens:
    word = str(w[0])
    ind = w[1] #index of token

    antonym = fetch_antonym(word)

    if not antonym: #if an antonym cannot be found, find the opposite label

      word_label = polarity_label(sig(return_polarity_scores(word)))
      print(word_label)
      inverse_score = [*map(inverses.get, str(word_label))][0]
      inverse_scores.append(inverse_score)
      text[ind] = "o"

      continue

    else:
      print(text)
      ind =- 1
      text[ind] = antonym

  if len(negation_tokens) > 0:
    text_negless = [str(tok) for tok in text if tok not in [str(t) for t in negation_tokens]] #remove negation tokens from string if they're present
  else:
    text_negless = [str(tok) for tok in text] #if there are no negations present, just return every token from the text

  ### Collect polarity scores for all words in list ###

  pol_scores = [return_polarity_scores(str(word)) for word in text_negless]

  #append inverse score for when antonyms aren't found
  if len(inverse_scores) > 0:
    for score in inverse_scores:
      pol_scores.append(inverse_score)

  # Return the mean score of the collected scores across all the words in the snippet
  score = round(sig(statistics.mean(pol_scores)), 2)
  label = polarity_label(score) #rond score af tot 2 decimalen na de komma

  print(score, label)
  return label

def return_polarity_scores(word):
  try:
    polarity_value = bsn.polarity_value(word) #try to find the word in the multilingual senticnet
  except KeyError:
    try:
      polarity_value = sn.polarity_value(word) #try to find the word in the English senticnet
    except KeyError:
      polarity_value = 0 #if not found, return 0 (neutral)
  return float(polarity_value)

#normalize scores with sigmoid in 0-1 range
def sig(x):
 return 1/(1 + np.exp(-x))

def polarity_label(score): #add polarity label based on Senticnet score

  if score <= 0.20:
    return 1
  elif score > 0.20 and score <= 0.40:
    return 2
  elif score > 0.40 and score <= 0.60:
    return 3
  elif score > 0.60 and score <= 0.80:
    return 4
  elif score > 0.80 and score <= 1:
    return 5

def add_mean_polarity_score(noun_adj_pairs):
  for k, v in noun_adj_pairs.items():
    if v['modifier_polarity']:
      #take the mean polarity scores of the modifiers + push in a 0-1 range w/ sigmoid
      #because the range of Sentic is -1 : 1
      mean_pol = sig(statistics.mean([float(x) for x in v["modifier_polarity"]]))
      label = polarity_label(mean_pol)

      noun_adj_pairs_en[k]["mean_polarity"] = mean_pol
      noun_adj_pairs_en[k]["polarity_label"] = label #add a polarity label according to the gold standard annotations

  return noun_adj_pairs

The next step is to apply this to the opinion words which were annotated in our gold standard, calculate a sentiment score and evaluating it against our manual annotations.

gold_opinion["sentiment_predictions"] = gold_opinion["text"].apply(sentiment_scorer)

Result of the sentiment scoring step

As we can see, we have added a new column to our dataframe called sentiment_predictions. How would our system compare to the gold standard annotations? We can apply sklearn’s classification_report to find out!

gold_opinion["sentiment_predictions"] = pd.to_numeric(gold_opinion["sentiment_predictions"])
gold_opinion["annotation"] = pd.to_numeric(gold_opinion["annotation"])

#list of true instances, list of predicted instances
true = list(gold_opinion["annotation"])
pred = list(gold_opinion["sentiment_predictions"])

print(classification_report(true, pred))

Output of our sentiment scoring step

Oops, there’s some room for improvement there! The model didn’t grasp any opinion words in classes 1 and 5. In a next step, we could try to mitigate this by playing with the threshold values we set.

Hope this helps to inspire!

✅ The advantages of a rule-based and off-the-shelf approach like this are that it’s modular, easy to understand and adaptable. The output can be controlled and explained. Specifically in highly specialized settings where a lot of expertise on the text material is available (which is often the case in the Humanities) — this type of approach can be useful and insightful!

❌The downsides are that it takes a lot of time to build a system like this, and that they do not generalise well over other domains. They also take quite a bit of expertise to build — and off-the-shelf tools are not always adequate for the literary-historical domain.

5. Resources

• Check out the paper we wrote on the evaluation of different ABSA-systems for DH — where we compare a rule-based system to machine learning and generative LLMs.
• If you want to learn more about ABSA in Python.