Stop Tuning Hyperparameters Until You Run These 7 Checks

Your model probably doesn’t need another learning rate experiment. It needs you to verify the boring stuff first.

A familiar scene in machine learning work goes like this.

The model performs badly.

Someone opens the training config.

learning_rate = 0.001
batch_size = 32
max_depth = 8

Five minutes later

learning_rate = 0.0007
batch_size = 64
max_depth = 10

Then comes Optuna. Grid search. Bayesian optimization. Twenty-seven experiments. Four GPUs quietly turning electricity into heat.

Accuracy improves from 84.1% to 84.3%.

Everyone celebrates.

Meanwhile, the validation split contains duplicate customers from the training set.

Wonderful.

Hyperparameter tuning is useful. But developers reach for it far too early. Before touching max_depth, C, dropout, weight_decay, or whatever knob currently has your attention, run these seven checks.

They often matter more than tuning.

Non members can read full article by clicking here!

Image edited using Canva.

1. Beat a Stupid Baseline First

Before asking

“Which hyperparameters should I optimize?”

Ask

“Is my model actually learning anything useful?”

For classification, compare against something embarrassingly simple

from sklearn.dummy import DummyClassifier

baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
print(baseline.score(X_test, y_test))

For regression, try predicting the mean or median.

Then try a simple linear/logistic model.

If your expensive gradient-boosting pipeline scores 91% and the dumb baseline scores 90%, you don’t have a hyperparameter problem.

You have a value problem.

This becomes especially important with imbalanced datasets.

Suppose 95% of transactions are legitimate.

A classifier predicting

NOT FRAUD

for every transaction gets 95% accuracy while being completely useless.

That’s why baseline models and appropriate evaluation metrics should come before fancy optimization. Scikit-learn even provides dummy estimators specifically for establishing simple baselines.

Do this: Record the performance of a naive baseline and one simple model before running your serious model.

If you can’t comfortably beat them, stop tuning.

2. Check Whether Your Split Represents Reality

This mistake can destroy an entire experiment while producing beautiful charts.

Imagine predicting whether customers will cancel subscriptions.

Your dataset contains several records from each customer.

You randomly split rows

train_test_split(X, y, test_size=0.2)

Now Customer #472 appears in training and validation.

Your model may indirectly learn characteristics of that customer and then conveniently “generalize” to… the same customer.

That’s not generalization.

That’s an open-book exam.

The correct split depends on the real production problem.

If predicting future events, consider a time based split.

If multiple records belong to the same patient, customer, device, company, document, etc., consider group based splitting.

Scikit-learn provides tools such as GroupKFold and StratifiedGroupKFold for exactly these situations.

Do this: Ask one question

“Could information about the same real-world entity appear on both sides of this split?”

If yes, investigate before running another experiment.

3. Hunt for Data Leakage

Data leakage is the silent assassin of ML projects.

The model looks brilliant.

Management gets excited.

Production happens.

Performance falls off a cliff.

Why?

Because training accidentally included information the model wouldn’t actually have when making a prediction.

One classic example

scaler.fit(X)
X_scaled = scaler.transform(X)

X_train, X_test = train_test_split(X_scaled)

You fitted preprocessing using the entire dataset before creating the split.

Information from the test set influenced preprocessing.

Instead, preprocessing should generally be learned from training data and then applied to unseen data.

Using a pipeline helps prevent this class of mistake

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(
    StandardScaler(),
    LogisticRegression()
)

Scikit-learn explicitly warns about both data leakage and inconsistent preprocessing.

Also inspect your actual features.

Predicting loan default?

A feature called

account_closed_due_to_default

might produce fantastic accuracy.

It may also be information available only after the thing you’re trying to predict.

Congratulations. Your model has discovered time travel.

Do this: For every suspiciously powerful feature, ask

“Would I genuinely know this value at prediction time?”

4. Check Your Labels Before Blaming the Model

Developers love blaming algorithms.

Sometimes the algorithm deserves it.

Sometimes y is garbage.

Take 100–200 examples and manually inspect them.

Look for

• incorrect labels
• ambiguous labels
• duplicates
• missing values
• impossible values
• inconsistent definitions
• stale records
• class imbalance
• obvious annotation mistakes

This isn’t glamorous work.

Nobody posts

“Spent Tuesday manually inspecting 150 CSV rows 🔥”

on LinkedIn.

But ten mislabeled edge cases can teach you more about your problem than another 200 training runs.

Suppose you’re building a sentiment classifier and humans themselves disagree heavily about whether sarcastic comments are positive or negative.

Increasing num_layers won't magically repair an ambiguous definition of “positive.”

Reality: Your model cannot consistently learn rules that your dataset doesn’t consistently contain.

5. Make Sure You’re Optimizing the Right Metric

One of the strangest ML habits is obsessively optimizing a metric nobody has connected to the actual problem.

“Our F1 increased!”

Cool.

Does that matter?

Consider fraud detection.

False negative

You miss fraud.

False positive

You temporarily inconvenience a legitimate customer

Those errors have different consequences.

Or medical screening. Or spam detection. Or recommendations. Or search ranking.

The useful metric depends on what mistakes actually cost.

You might care about

• precision
• recall
• F1
• ROC-AUC
• PR-AUC
• MAE
• RMSE
• ranking metrics
• calibration
• business-specific cost functions

Don’t optimize “accuracy” merely because .score() made it convenient.

Do this: Write this sentence before tuning

“We optimize __ because improving it means __ for the actual user/business.”

If the second blank is difficult to fill, your problem isn’t hyperparameters yet.

6. Look at the Errors, Not Just the Average

A single score compresses thousands of predictions into one number.

Useful?

Yes.

Enough?

Absolutely not.

Suppose Model A scores 92%. Model B scores 92.4%. Obviously B wins, right?

Maybe.

Then you inspect the failures.

Model A struggles mainly with low quality images.

Model B performs significantly worse on an important customer segment.

Suddenly that innocent 0.4% improvement isn’t so impressive.

Slice the errors.

Look at performance by

• class
• geography
• device type
• input length
• data source
• time period
• confidence level
• relevant user or domain segment

Then manually inspect false positives and false negatives.

You’ll often discover something tuning cannot repair

“Half our failures come from one broken data source.”

That’s excellent news.

You now have an engineering problem you can actually fix.

7. Run Learning Curves Before Throwing Compute at the Problem

Finally, figure out whether you’re dealing with underfitting, overfitting, or insufficient data.

Compare training and validation performance as the amount of training data changes.

Scikit learn’s learning-curve documentation describes learning curves as a way to examine training and validation scores across different training-set sizes and diagnose whether more data may help.

Conceptually

High training score + much lower validation score

Likely generalization/variance problem.

Think about regularization, more representative data, simplifying the model, augmentation where appropriate, and only then hyperparameters.

Low training score + low validation score

Your model may be underfitting.

Changing the representation, features, model class, or model capacity may matter more.

Training and validation both plateau

Running another 500 combinations of

learning_rate
max_depth
min_samples_leaf

may produce tiny improvements while avoiding the actual bottleneck.

Also check stability across folds or repeated runs. Cross-validation is useful because one lucky train/validation split can give you an overly optimistic impression of model quality.

Image edited using Canva.

Then Tune the Damn Hyperparameters

I’m not anti tuning. I’m anti premature tuning.

Once you’ve verified

1. Your model beats meaningful baselines.
2. Your validation split represents deployment.
3. You don’t have obvious leakage.
4. Your labels and data make sense.
5. Your metric represents the real objective.
6. You’ve inspected where the model actually fails.
7. You understand the training/validation behavior.

Now tune.

Use randomized search, Bayesian optimization, Optuna, grid search when appropriate whatever fits your problem and compute budget.

At this point, changing hyperparameters is optimization.

Before this point, it can easily become expensive procrastination disguised as machine learning.

And there’s one more production reality worth remembering: the data pipeline matters after training too.

Google’s Rules of Machine Learning specifically discusses training serving skew the problems created when the data or processing used during serving differs from training.

The best learning_rate on Earth won't rescue a broken pipeline.

The Checklist I’d Save

Before your next tuning run, ask

[ ] Did we beat a simple baseline?
[ ] Is the train/validation split realistic?
[ ] Did we check for leakage?
[ ] Did we manually inspect the data and labels?
[ ] Are we optimizing the right metric?
[ ] Did we inspect actual failure cases?
[ ] Did we examine learning curves and validation stability?

Seven checkboxes.

Potentially hundreds of wasted experiments avoided.

Your GPU may hate me for saying this.

Your engineering budget probably won’t.

If you’ve seen a team spend days tuning a model only to discover the dataset was broken, I’d love to hear the story. Disagree with one of these checks? Even better debate it in the comments.

And if you know a developer currently running experiment number 437 because 0.0005 might finally be the magical learning rate, send them this article.

Or save it for the next time that developer is you.