F1 Score in Named Entity Recognition Tasks — An Intuition

It’s been 10 years now since I’ve stepped into the world of machine learning when I first took such a course at university. One of the first things we got taught was how to measure the success of the models we were training. Specifically, the subset of ML problems aimed at classification or information extraction (be it LLMs, Agents or older systems) has several ways of measuring how good our model is in at guessing whatever in terms of metrics.

The famous ‘four’ of these metrics are: accuracy, precision, recall and F1 score. And to all of us working with them, they’re a routine and the standard way to evaluate most AI systems. But how to explain them to people outside of the ML bubble?

Not until recently I realized how unintuitive they sound to non ML practitioners and we end up to the same question: why do I need these precision and recall when I can simply count hints vs. misses? So it got me to put this short intuition together.

Intuition with binary classification problems

There is a reason why we usually start with binary classification problems, that is, when we try to classify a sample into one of two classes, the evaluation metrics were initially designed to support these sorts of problems and were then expanded to evaluate different ML problems.

In binary classification, for a group of samples, we try to categorize each one into one of two classes. For example, this can be classifying patients into healthy and sick based on their lab test results. In reality, number of sick patients would be much lower than the number of healthy ones.

So when we measure how ‘good’ our system is, we take into account four groups of samples when we compare what we classified with the original truth for the patients:

• True positives (TP): The number of patients we classified as sick and are indeed sick
• True negatives (TN): The number of patients we classified as healthy and are indeed healthy
• False positives (FP): The number of patients we classified as sick but are in fact healthy
• False negatives (FN): The number of patients we classified as healthy but are in fact sick

Then with the evaluation metrics, we measure:

Accuracy: Measure of how often we’ve correctly classified a patient (regardless of whether the patient was sick or healthy). It is measured by the number of correctly classified patients divided by the total number of patients. It is calculated by the formula:

(TP + TN) / (TP + TN + FP + FN)

→ Why it is not the best metric: Let’s say we have one sick patient and 99 healthy ones. Let’s also say we misclassified that one sick patient as healthy, and we got all the 99 healthy patients right. This means we have:

TP = 0, TN = 99, FP = 0 and FN = 1, which translates to an accuracy of 99/100=0.99

We are getting a 0.99 accuracy and the only sick patient we were supposed to diagnose is getting treated as healthy, which makes our experiment a failure.

So accuracy metric is often very misleading which is why in both academia and industry it is used with a grain of salt when measuring machine learning systems. This is where other metrics come along.

Precision: Answers the question, out of all the patients that we’ve classified as sick, how many are actually sick? In other words, precision is a measure of quality. If it is low, it means we have a lot of false positive alarms (healthy patients diagnosed as sick)

We calculate it with the formula: TP / (TP + FP)

Recall: Answers the question, out of all sick patients in our examination set, how many did we correctly diagnose? So this is a measure of quantity. If it is low, means there are a lot of sick patients out there that we’re not finding and will therefore go without treatment.

We calculate it with the formula: TP / (TP + FN)

F1: So what does F1 do. It measures how well we satisfy both quality and quantity criteria at the same time. It is high, it means we found most of the sick patients and we did not alarm any healthy patients into falsely thinking they’re sick.

The fact that F1 manages to encompass most information we care about in one score, is why it is chosen as industry standard when evaluating machine learning systems.

It is calculated as the harmonic mean of precision and recall:

2 * Precision * Recall / (Precision + Recall)

Translating evaluation metrics entity recognition

The calculation of metrics for tagging systems (where we tag entities in some text) follows a bit different intuition, so it is usually easier connecting it with classification.

In such systems, we do not classify per se, but we find things, i.e. tags. For example, the tags are entities, or for example product names.

When we compare, there is:

• Some sort of dictionary list of all the possible products that are eligible to be looked into the text (all products we care about),
• A pool of product names that a reference (human expert) has found in the text,
• A pool of products that the AI system has found in the article.

From here, when we compare the product names the AI system has found in the text, and the number of car brands the human expert has found in the same text, we have the following groups:

• True Positives (TP): The number of products that both the AI system and the human expert have found.
• True Negatives (TN): This is where it becomes tricky, this is the number of products that both the AI system and the human expert didn’t find in the text. This can be viewed as all the eligible product names in the dictionary. So since these are a lot more, they will always mask the real results, so they have to be taken out of the equation.
• False Positives (FP): These are the product namess that the AI system has found but the human expert hasn’t. In other words we can treat these as noise. They are tags that shouldn’t appear on the screen when an text is tagged.
• False Negatives (FN): These are the products that the human expert has found but the AI system hasn’t. These are products that the AI system has missed, or failed to find.

Accuracy:

Now if we want to calculate accuracy as before as the portion of guesses out of all products, (TP + TN) / (TP + TN + FP + FN), we cannot do it reliably since we have to include TN, or all other products in the dictionary. This will hype the score beyond reason since per text sample we measure the number of tags in tens (10-50 tags per text sequence), and the dictionaries may contain hundreds of entries.

So we have to adjust the score — we have to take out true negatives (TN) out of the equation. By this logic we’d calculate accuracy as such:

TP / (TP + FP + FN)

What this equation presents is the portion of matches between the AI system and the human coder out of all tags (or products) that both the AI system and the human expert have found. In other words, it measures how similar two sets of items are (This formula in literature is known as Jaccard but that is irrelevant for the example).

If we look at the example in Table 1, we have a column ‘AI System’ which contains all of the tags that the AI found, and ‘Ground Truth’ column with all in the tags we compare against, we can see in the ‘Match’ column that in 5 out of 10 cases the tags that both parties found match, and in the other 5 cases they don’t.

While this is very intuitive and tells us how often we’re correct, it lacks the information what type of mistake we’ve done. If we take a look at the ‘Precision’ and ‘Recall’ columns, we can see that out of those 5 mismatches, in 2 the AI system has found an additional tag, and in 3 has missed a tag that should’ve been there. These two are mistakes of a different nature, and we do not capture them entirely with this approach.

Table 1: Example calculations with various matches and misses for all relevant metrics

That is why the Precision-Recall-F1 trio is a best option to address the types of mistakes that the AI system does.

Precision:

With precision, we address quality. If it is high, we are sure that we are not finding additional tags (or for intuition car brands). In other words, it tracks the false positive rate (the first mistake type that the AI model can make).

For recap: TP / (TP + FP)

Recall:

With recall, we address quantity. If it is high, means we’ve found all tags (or car brands) that we should in the article. If it’s low, means we’re missing out a lot. In other words, we track the false negative rate (the second type of mistake that the AI model can make).

For recap:TP / (TP + FN)

F1 Score:

The F1 score holds the information of both of precision and recall into one number, or it is a balanced score between the both. While it is not intuitive at first, it captures a lot of crucial information on both the quality and quantity of predictions.

Looking again from the example of Table 1, the F1 score is higher than the Accuracy (column ‘Match’). What does it tell us? That we’ve done a little of the first mistake and a little of the second mistake, which all together is not that bad.

So when will F1 be same as the accuracy? If we only make one type of mistake. If for instance, all the mismatches were false positives, i.e. tags that the AI system found but shouldn’t have, then the F1 score would become the same with the accuracy score.

This is important because it is a lot worse for the AI system to miss out on 5 tags than miss only 3 and add two more additional ones, and vice versa, it would be worse for the system to add 5 additional tags (car brands) that should not be there instead of adding only 2 on account on missing 3.

So what makes F1 different (together with Precision and Recall) is the fact that it evaluates the system for the type of mistake it makes. It penalizes it more if makes more of the same mistake, and it rewards it if it balances them.

Formula for F1 recap: 2 * Precision * Recall / (Precision + Recall)

Why F1 is the most reliable score

• It is the most widely used metric for the evaluation of Machine Learning and AI systems, both in academia and industry,
• It is directly interpretable in terms of business trade-offs. If it’s high it means we’re both good on finding the correct tags (entities, brands), and also on not missing out anything,
• It ‘compresses’ the most amount of information. We track two types of mistakes that an AI system can make with it, so if we make more of one type of mistake, we cannot mask it, so F1 score will drop,
• It makes it easier to backtrack and troubleshoot an issue. In an accuracy score, we treat all mismatches as one and the same, so if our system underperforms we cannot for sure say whether we’re ‘missing stuff’ or ‘adding junk’, whereas with F1 and by extension, precision and recall, we can see which of these two scores is lower so to see where to troubleshoot and fix the system.