Evaluating AI Models: When Metrics Don’t Lie But Aren’t the Ultimate Truth (II)

Photo by Igor Omilaev on Unsplash

In matters of artificial intelligence and machine learning, metrics guide us. They tell us whether a model is working, whether it is improving during training, and whether it is worth deploying in the real world. But relying too heavily on these metrics can be misleading: none of them is perfect. We find ourselves in a scenario where, on the one hand, every metric has some limitation worth understanding, and on the other, those who implement artificial intelligence (everyone, without exception) have responsibilities.

In these lines we will explore the weaknesses of four widely used metrics: precision, recall, specificity, and F1-score. All four are necessary and we use them daily. So, far from demonizing them, the goal will be to show their blind spots, so that readers of model evaluations know when and in what aspect to keep a distance and thus make better decisions.

Precision: The illusion of accuracy

The precision metric measures how many of the model’s positive predictions are actually correct. Something like the model saying: Of everything I said was positive, how many did I get right?

Precision = True Positive ÷ (True Positive + False Positive)

Where:

True Positive = the cases that are actually positive in the dataset

False Positive = the cases the model flagged as positive, but are not in the dataset / reality.

But: It ignores false negatives. The model will have high precision if it avoids predicting positives, yet it can still be missing many real cases.

Precision has a bias toward the majority class: if the positive class is rare, the model will not detect it and can still show high precision. For example, a spam detector that only flags the most obvious emails. It will undoubtedly have high precision because it does not make mistakes, but it will let more subtle spam through. Thus it can create an illusion of accuracy because it may be hiding risky errors.

Recall: Obsession with not letting anything escape

It measures how many of the actual positive cases were detected. The model will say: Of all the positives that existed, how many did I find?

Recall = True Positive ÷ (True Positive + False Negative)

But: Recall ignores false positives. This allows a model to have high recall if it predicts positive almost always, even though it is wrong most of the time.

It has a strong dependence on other measures and can overstate success: a fire detector that triggers alarms at any smoke will have high recall because it is not missing any smoke detections, but that includes smoke from a toaster. In this case, it will issue absurd alarms and thus become useless.

Recall is valuable when no cases should be missed, but without thoroughly reviewing those cases, it becomes a double-edged sword.

Specificity: The guardian of the negatives

It measures how many actual negative cases were correctly identified. Its function is to answer: Of all the negatives that existed, how many did I recognize as negative?

Specificity = True Negative ÷ (True Negative + False Positive)

But: It focuses on the negative class, so it becomes irrelevant in problems where detecting positives is necessary.

It can give a false sense of security with models that always predict negative, even if they never detect a positive: if, in a medical test, the model always answers “not sick” for all examinees when positive cases exist, its usefulness is zero.

Specificity is very useful in contexts where false positives are costly, but it can hide the model’s inability to detect cases that also matter.

F1-score: A balance that does not always balance

This is the harmonic mean between precision and recall. It is a balanced metric, useful in scenarios where we must consider both false positives and false negatives.

F1-score = 2 × (Precision × Recall) ÷ (Precision + Recall)

But:

It does not reflect context. The F1-score treats precision and recall as if they were equally important, when in practice they are not always so. Because of this, it can hide extremes. For example, a model with very low precision and very high recall will average to an acceptable F1-score, but it remains impossible to implement. This is the case of a security model that detects almost all intruders (high recall) but also accuses many innocents (low precision). The F1-score number may look acceptable, but that model is impossible to implement.

Although the F1-score summarizes information in a single value (and that is very useful), it only works if we interpret it in the context of the problem.

What remains unsaid

What these metrics have in common is that all of them can be manipulated if used without context.

• Precision can hide risky predictions.
• Recall can predict positive all the time.
• Specificity can predict negative all the time.
• F1-score can produce an average that hides extremes.

In all cases the same thing happens: a high number does not reflect the model’s reality or its impact in the real world.

¿What to do?

• Look at several metrics at the same time: relying on just one is too risky.
• Use a confusion matrix: to stay aware of the errors explicitly.
• Consider the consequences: in health, false negatives can range from minor consequences to considerable ones, to serious ones, and end in life or death. In security, false positives can trigger legal consequences. In movie recommendations, they never have major importance, except for the platform’s commercial concerns. Essentially, the context of the decisions the model makes is indispensable.
• Test on real data: we must assume that clean datasets will not reflect performance in production.

Model evaluation is not only mathematics: it is also a matter of responsibility. Numbers must be interpreted with care, because behind every error there are human consequences.

It is only a matter of attention

Precision, recall, specificity y F1-score are fundamental metrics in the toolbox of any artificial intelligence team. Each has weaknesses that can lead us to wrong conclusions if we use them hastily. The challenge is not to abandon them, but to learn to read them with attention. To understand what they hide, what they show, and what they leave out.

Because, in the end, evaluating a model is not only calculating percentages: it is asking ourselves what kind of errors we are willing to accept and what impact they will have in real life. In a world increasingly governed by algorithms, that attentive gaze will allow us to distinguish mirages from realities.