It’s Time to Upgrade from AffectNet — Here’s Why AffectNet+ Changes Everything

Seven years after the release of AffectNet, we rebuilt it from the ground up. Soft-labels, richer metadata, and difficulty-tiered subsets make AffectNet+ the dataset that modern FER research actually needs.

In 2017, we introduced AffectNet — at the time the largest in-the-wild facial expression dataset in existence, with over one million images annotated for both categorical emotions and continuous valence/arousal. Thousands of researchers have since relied on it to train and benchmark their facial expression recognition (FER) models.

But the field has grown. And so have the demands placed on benchmark datasets. After years of observing how AffectNet’s limitations constrain model development, we built its successor: AffectNet+.

“Assigning a single label to a facial image might not be the right approach — some faces express compound emotions that no single word can capture.”

What Was Holding AffectNet Back?

AffectNet was a landmark contribution, but it carried structural limitations that the community has been working around ever since:

Single-annotator labels. Each of the 450K annotated images was labeled by exactly one human expert. This single point of judgment introduces significant noise — human annotators disagree on compound or ambiguous expressions more than 32% of the time.

Hard labels only. Every image is assigned exactly one emotion category, even when a face clearly expresses a blend — like a sad smile or fearful surprise. This forces models to learn artificially clean boundaries that don’t match reality.

Noisy metadata. Facial landmark points were extracted with older algorithms, and critical attributes like age, gender, ethnicity, and head pose were simply absent.

No difficulty stratification. All training images were treated identically, giving no signal about which samples are genuinely ambiguous versus straightforward — a key variable for curriculum learning approaches.

AffectNet vs. AffectNet+: A Side-by-Side Look

AffectNet (2017)

• Single hard-label per image
• Basic landmark metadata only
• No difficulty tiers
• No age, gender, or ethnicity info
• No head pose data

AffectNet+ (2025)

• Soft-labels: a probability vector across all 8 emotions
• Re-extracted, high-quality facial landmarks
• Easy / Challenging / Difficult image subsets
• Age, gender, and ethnicity metadata included
• Head pose, valence & arousal retained

The Core Innovation: Soft-Labels

The most significant change in AffectNet+ is the introduction of soft-labels. Instead of a single emotion tag, every image is now annotated with a probability vector across all eight emotion categories — quantifying how much of Happy, Sad, Fear, Disgust, Anger, Surprise, Contempt, and Neutral is present in a given expression.

We compute soft-labels through two complementary methods:

1. Ensemble of binary classifiers. Eight separate classifiers, each trained to predict the probability of one specific emotion versus all others, are combined into a single soft-label vector.

2. Action Unit (AU)-based classifier. Using the Emotional Facial Action Coding System (EMFACS), we leverage the overlap between facial muscle movements associated with different emotions to generate AU-grounded probability scores.

This gives each image a richer, more honest descriptor. A face expressing a sad-neutral blend is no longer forced into one box — its annotation reflects the ambiguity directly.

“Soft-labels enable smoother decision boundaries, native multi-label classification, and more principled handling of imbalanced, noisy training data.”

Difficulty Tiers for Smarter Training

Every image in AffectNet+ is categorized into one of three subsets based on how difficult it is to recognize the expressed emotion:

Easy — High annotator agreement; the model-assigned label matches the hard-label reliably. Ideal for initial training stages.

Challenging — Some ambiguity present; the soft-label shows multiple emotions with moderate confidence. Useful for intermediate training.

Difficult — High uncertainty; compound expressions or extreme intra-class variation. Critical for robustness testing and advanced curriculum learning.

These tiers unlock curriculum learning — where models are exposed to progressively harder examples — and allow fairer benchmarking, since researchers can report performance separately on each difficulty level.

Why This Matters for Your Models

The three perennial headaches for FER researchers training on AffectNet have been label uncertainty, class imbalance, and lack of diversity. Soft-labels directly attack all three:

Label uncertainty is mitigated because the soft distribution is a more faithful representation of the annotation signal, rather than a noisy majority vote collapsed into one class.

Class imbalance becomes easier to handle when images carry probability mass across classes — over- and under-represented categories are no longer artificially hard boundaries.

Bias and fairness research is now possible thanks to the included age, gender, and ethnicity metadata, enabling models to be audited and corrected for demographic disparities.

How to Access AffectNet+

AffectNet+ is publicly available to the research community through the Mahoor Lab at the University of Denver.

📄 **Read the paper here: **https://ieeexplore.ieee.org/abstract/document/11259096

📄 Or here: arxiv.org/abs/2410.22506

💾 Download the dataset: mohammadmahoor.com/databases-codes

If you have been using AffectNet in your research, we encourage you to switch to AffectNet+ for your next project. The richer annotations, cleaner metadata, and difficulty stratification will make a measurable difference in the quality and interpretability of your models.

AffectNet+ was introduced in: Pourramezan Fard, Hosseini, Sweeny & Mahoor (2025). “AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels.” IEEE Transactions on Affective Computing.

AffectNet was introduced in: Mollahosseini, Hasani & Mahoor (2017). “AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild.” IEEE Transactions on Affective Computing.