Handling Imbalanced Data

When Your AI Is 99% Accurate — and Completely Useless

The quiet crisis inside machine learning that nobody talks about at the dinner table

By Anushree Kale· 7 min read

You’ve been heads-down on a project for two weeks. The model trains cleanly. Loss curves slope downward like they’re supposed to. You run evaluation on the test set and the number appears on your screen:

99.1% accuracy.

You lean back. You feel that quiet satisfaction that only comes from a number like that. You start mentally drafting the Slack message to your team.

Then someone asks: “So how many of the actual fraud cases did it catch?”

You run the numbers.

Eleven out of a hundred.

Not eleven percent. Eleven out of every hundred real fraudulent transactions — the exact things the entire project was supposed to find — slipped through completely undetected. The model had essentially learned one party trick: predict “not fraud” for everything, collect its 99% accuracy, and take a bow.

This is imbalanced data. And what makes it genuinely dangerous is not just that it breaks models — it’s that it breaks them silently, in ways that look like success until the moment they very much don’t.

First, Let’s Talk About Why This Happens at All

Your model is not stupid. It’s actually doing exactly what you asked it to do.

You handed it a dataset with 99,000 legitimate transactions and 1,000 fraudulent ones. You told it to minimise error. So it ran the numbers and figured out: if I call everything legitimate, I’m wrong 1% of the time. That’s pretty good. It learned the path of least resistance, because that’s all gradient descent ever does — find the lowest point in the landscape you gave it.

The problem isn’t the algorithm. The problem is that the landscape you gave it was crooked from the start.

This is what data scientists mean when they talk about the class imbalance problem — datasets where one outcome is so much rarer than the other that standard training procedures effectively forget it exists.

And here’s the uncomfortable truth: this isn’t some edge-case academic scenario. It is the default state of almost every high-stakes prediction problem that actually matters in the real world.

“The model wasn’t broken. It was doing exactly what we rewarded it for. We just rewarded it for the wrong thing.”

The Problems That Are Always Imbalanced (And Why That’s Not a Coincidence)

Think about the kind of predictions that companies spend serious money trying to get right:

Fraud detection — Credit card fraud accounts for roughly 0.1–0.2% of transactions. Out of every 1,000 swipes at a terminal, maybe two are someone stealing from you. The other 998 are just people buying groceries.

Cancer screening — Early-stage lung cancer appears in about 1–2% of high-risk screening populations. The radiologist reading 200 scans a day is looking for 2–4 anomalies in a sea of normal tissue.

Equipment failure prediction — Industrial turbines fail perhaps 0.3% of operating hours. A factory running 24/7 might see one failure event every few months.

Loan defaults — Typically 2–5% of loans go into default. The other 95–98% are repaid on schedule, unremarkably, without incident.

Rare disease diagnosis — Conditions like ALS, Huntington’s, or early-stage pancreatic cancer affect fractions of a percent of the people who get tested for them.

Notice the pattern? In every single domain where a wrong prediction has catastrophic consequences — financial ruin, missed cancer, industrial disaster — the thing you’re trying to predict is also the rarest thing in the dataset.

The universe is not being perverse. This is just how high-stakes problems work. Bad outcomes are rare precisely because systems are designed to prevent them. But when they do happen, they matter enormously. And a model that can’t find them is not a model — it’s a very expensive coin flip tilted toward “everything is fine.”

This is what your training data actually looks like.

The Accuracy Paradox: A Number That Lies Beautifully

Let’s do the maths that nobody does in the excitement of a high accuracy score.

You’re building a sepsis prediction model for a hospital ICU. Sepsis affects about 3% of ICU admissions in your dataset. You have 10,000 patient records: 9,700 without sepsis, 300 with it.

You train a logistic regression model with default settings. It achieves 97% accuracy.

Here’s what that actually means when you look at the confusion matrix:

Predicted: No SepsisPredicted: SepsisActual: No Sepsis9,7000Actual: Sepsis3000

The model predicted “no sepsis” for literally every single patient. It has never once raised an alert. Every one of those 300 sepsis patients was discharged from the model’s attention without a flag.

97% accuracy. 0% usefulness.

This is the accuracy paradox. And the reason it persists is that most people stop at the top-line number and never look underneath it. The confusion matrix is where the truth lives. The accuracy score is just the cover story.

So What Do You Actually Measure?

When your classes are imbalanced, you need metrics that care about the minority class specifically. Here are the four that matter:

Recall (Sensitivity) — Out of all the actual positive cases in the real world, what fraction did your model find? This is your “did I catch the cancer?” number. In high-stakes medical or safety contexts, this is often the only number you should optimise for first.

Precision — Out of all the cases your model flagged as positive, what fraction were actually positive? This is your “how often is my alarm a false alarm?” number. Important when false positives are expensive (unnecessary surgeries, frozen bank accounts).

F1 Score — The harmonic mean of Precision and Recall. It’s the number that forces you to be honest about both. A model can’t game the F1 by sacrificing one to inflate the other.

AUC-PR (Area Under the Precision-Recall Curve) — Forget the ROC curve for severely imbalanced problems. The Precision-Recall curve doesn’t flatter you. It shows exactly how your model performs across all decision thresholds, specifically on the positive class. A random classifier on a 1:99 imbalanced dataset still gets an AUC-ROC of 0.5 — but its AUC-PR will be a brutal 0.01.

The shift from “what is my accuracy” to “what is my recall on the minority class” is not a technical adjustment. It’s a philosophical one. It’s asking: what does it actually mean for this model to work?

“A 97% accurate sepsis model that has never once detected sepsis is not a medical tool. It’s a liability wearing a lab coat.”

Tell the Model What Matters: Class Weights

This is the most underused, cleanest, fastest fix — and it should be your first stop before you touch the data at all.

Every major ML framework has a class_weight parameter. Setting it to 'balanced' automatically computes weights inversely proportional to class frequencies. What this does mathematically: it multiplies the loss function contribution of minority-class mistakes by a larger number, so the model is penalised harder every time it gets one wrong.

You’re not adding data. You’re not removing data. You’re just changing what the model is punished for.

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression

# For scikit-learn models
model = LogisticRegression(class_weight='balanced')

# For XGBoost — set scale_pos_weight to ratio of negatives:positives
# If you have 9700 negatives and 300 positives: 9700/300 ≈ 32
import xgboost as xgb
model = xgb.XGBClassifier(scale_pos_weight=32)

Create More of the Rare: SMOTE Oversampling

When class weights aren’t enough, it’s time to give the model more minority-class examples to learn from. SMOTE (Synthetic Minority Oversampling Technique) is the most widely respected way to do this.

from imblearn.over_sampling import SMOTE
from sklearn.model_selection import train_test_split

# CRITICAL: Split FIRST. Then resample.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

sm = SMOTE(random_state=42, k_neighbors=5)
X_train_resampled, y_train_resampled = sm.fit_resample(X_train, y_train)

# Only X_train_resampled is used for training
# X_test stays untouched — always

The stratify=y in the split is not optional. It ensures both training and test sets preserve the original class ratio. Without it, you might randomly end up with a test set that has zero minority examples — and you'd never know your evaluation was measuring nothing.

One Line You Must Add to Every Imbalanced Project

# Not this:
cv = KFold(n_splits=5)
# This. Always.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

Standard K-Fold can accidentally create folds with almost no minority examples. StratifiedKFold preserves your class ratio in every fold. One parameter. Massive difference.

“99% of the board is shadow. One piece holds the light. Your model learned the shadow. It never found the piece.”

The Thing Under All of This

The imbalance in your data isn’t a bug. It’s the world being honest with you.

Fraud really is rare. Tumours are genuinely uncommon. The dataset looks lopsided because reality is lopsided — because the things worth predicting are exactly the things that don’t happen often enough to generate balanced training data.

The techniques above aren’t hacks. They’re corrections — ways of aligning your model’s incentives with the actual stakes of the problem.

The next time you see 99% accuracy, open the confusion matrix before you celebrate. Find the row for your positive class. Count how many examples landed in the “correctly predicted” column.

If that number is small, everything else is theatre.

Follow for more practical ML writing. Drop a comment with the worst accuracy-vs-reality gap you’ve seen — I’m sure the stories are wild.