Handling Imbalanced Datasets

Oversampling, Undersampling, and SMOTE

In the world of machine learning, data is rarely perfect. One of the most common challenges you’ll face is class imbalance.

Imagine you’re trying to train a model to detect fraudulent transactions.

Out of 10,000 transactions, only 100 are fraudulent.

That means fraud accounts for only 1% of the dataset.

If you train a model without addressing this imbalance, it might learn to always predict “non-fraud” and still achieve 99% accuracy, while being completely useless for detecting fraud.

This is why handling imbalanced datasets is crucial.

In this article, we’ll explore three widely used techniques: oversampling, undersampling, and SMOTE.

We’ll use analogies, code examples, and practical insights to give you a deeper understanding of each.

Table of Contents

· Table of Contents · What is an Imbalanced Dataset? · Approaches to Handle Imbalance ∘ 1. Oversampling ∘ Techniques ∘ 2. Undersampling ∘ 3. SMOTE (Synthetic Minority Oversampling Technique) · How it Works · Choosing the Right Method · Beyond Resampling · Final Thoughts

What is an Imbalanced Dataset?

An imbalanced dataset occurs when the distribution of classes is skewed. For example:

• Binary classification: 95% class A, 5% class B
• Multi-class classification: one class has significantly more samples than others

This imbalance causes models to be biased toward the majority class, making them poor at recognizing the minority class, the very one we often care about the most (fraud, disease, rare event detection).

Think of it like a classroom with 95 boys and 5 girls. If you ask the teacher to randomly pick a student, chances are much higher they’ll pick a boy. Without deliberate effort, the minority group gets ignored.

Approaches to Handle Imbalance

There are two main ways to handle imbalance:

1. Data-level methods: Modify the dataset distribution itself.
2. Algorithm-level methods: Adjust how algorithms handle imbalance (e.g., weighted loss functions).

In this post, we’ll focus on data-level methods, oversampling, undersampling, and SMOTE.

1. Oversampling

Oversampling means artificially increasing the number of samples in the minority class.

Think of it like making multiple photocopies of the rare notes in a library so that students have equal access compared to the common books.

Techniques

• Random Oversampling: Simply duplicate existing minority samples until the dataset is balanced.
• Advanced Oversampling: Use algorithms like SMOTE (which we’ll discuss later) to generate new, synthetic samples.

Example in Python

from collections import Counter
from sklearn.datasets import make_classification
from imblearn.over_sampling import RandomOverSampler

# Create an imbalanced dataset
X, y = make_classification(n_classes=2, class_sep=2,
                           weights=[0.9, 0.1], n_informative=3,
                           n_redundant=1, flip_y=0,
                           n_samples=1000, random_state=42)

print("Original dataset distribution:", Counter(y))

# Apply random oversampling
ros = RandomOverSampler(random_state=42)
X_res, y_res = ros.fit_resample(X, y)

print("After oversampling:", Counter(y_res))

Output:

Original dataset distribution: Counter({0: 900, 1: 100})  
After oversampling: Counter({0: 900, 1: 900})

Here, we balanced the dataset by duplicating the minority class.

Pros:

• Simple to implement
• Helps models pay attention to the minority class

Cons:

• Risk of overfitting since the minority samples are just repeated copies

2. Undersampling

Undersampling means reducing the number of samples from the majority class to match the minority.

Imagine you’re hosting a debate and you have 95 boys and 5 girls. To ensure equal participation, you randomly pick only 5 boys and pair them with 5 girls.

Example in Python

from imblearn.under_sampling import RandomUnderSampler

# Apply random undersampling
rus = RandomUnderSampler(random_state=42)
X_res, y_res = rus.fit_resample(X, y)

print("After undersampling:", Counter(y_res))

Output:

After undersampling: Counter({0: 100, 1: 100})

We cut down the majority class to match the minority.

Pros:

• Faster training (smaller dataset)
• Avoids overfitting on duplicated samples

Cons:

• Risk of losing important information from the majority class
• May under-represent the overall data distribution

3. SMOTE (Synthetic Minority Oversampling Technique)

SMOTE is a smarter oversampling method.

Instead of just duplicating minority samples, it creates synthetic samples by interpolating between existing ones.

Think of SMOTE like a music remixer. Instead of just replaying the same track (oversampling), it mixes two tracks together to create something new but still in the same genre.

How it Works

• Pick a minority sample
• Find its nearest minority neighbors
• Create a new sample by interpolating between them

This way, the algorithm generates new, diverse synthetic samples that enrich the minority class.

Example in Python

from imblearn.over_sampling import SMOTE

# Apply SMOTE
smote = SMOTE(random_state=42)
X_res, y_res = smote.fit_resample(X, y)

print("After SMOTE:", Counter(y_res))

Output:

After SMOTE: Counter({0: 900, 1: 900})

SMOTE balances the dataset with synthetic data instead of duplicates.

Pros:

• Reduces overfitting compared to random oversampling
• Creates more diverse samples in the minority class

Cons:

• Can generate noisy samples if the minority class is sparse
• May overlap with majority class samples, leading to confusion

Choosing the Right Method

• Use Oversampling if you want to keep all majority data and don’t mind potential overfitting.
• Use Undersampling if your dataset is huge and you can afford to lose some majority data.
• Use SMOTE when you want a balance between keeping majority samples and generating realistic synthetic minority samples.

Often, practitioners combine methods (e.g., SMOTE + undersampling) for better results.

Beyond Resampling

Other techniques include:

• Cost-sensitive learning: Give higher penalties to misclassifying minority samples.
• Ensemble methods: Techniques like Balanced Random Forest or EasyEnsemble.
• Anomaly detection: Treat the minority class as anomalies and use outlier detection methods.

Final Thoughts

Handling imbalanced datasets is like ensuring every voice in a meeting gets heard.

If you only listen to the loudest (majority class), you’ll miss the important but quieter insights (minority class).

• Oversampling: Copies the quiet voices to make them louder.
• Undersampling: Lowers the loud voices so the quieter ones stand out.
• SMOTE: Creates new, unique voices inspired by the existing ones.

Choosing the right method depends on your dataset size, model type, and tolerance for information loss or overfitting.

When applied thoughtfully, these techniques turn an imbalanced dataset into a fairer playing field, ultimately leading to more accurate and trustworthy models.