Hyper parameter tuining in machine learning

Introduction

Building a Machine Learning model is not just about choosing an algorithm.

Even powerful algorithms can perform poorly if their hyperparameters are not optimized properly.

That’s where Hyperparameter Tuning becomes important.

Hyperparameter tuning helps us find the best settings for a Machine Learning model to improve performance.

Two of the most popular tuning techniques in Scikit-Learn are:

✅ GridSearchCV ✅ RandomizedSearchCV

In this article, we will learn:

✅ What hyperparameters are ✅ Why hyperparameter tuning matters ✅ GridSearchCV explained ✅ RandomizedSearchCV explained ✅ GridSearchCV vs RandomizedSearchCV ✅ Real-world Python examples

What are Hyperparameters?

Hyperparameters are settings that control the learning process of Machine Learning models.

These values are set before training the model.

Examples:

• k in KNN
• max_depth in Decision Trees
• C in SVM
• learning_rate in Gradient Boosting

Hyperparameters are different from model parameters because they are not learned automatically during training.

Why Hyperparameter Tuning is Important

Without tuning:

• models may underfit
• models may overfit
• accuracy may decrease
• training becomes inefficient

Benefits of hyperparameter tuning:

✅ Better accuracy ✅ Improved model performance ✅ Reduced overfitting ✅ Better generalization

Many ML practitioners underestimate how much tuning affects performance. Reddit discussions frequently mention models jumping from poor to strong accuracy after proper tuning.

What is GridSearchCV?

GridSearchCV is a hyperparameter tuning technique that tries all possible combinations of hyperparameters.

It uses:

• cross-validation
• exhaustive search

to find the best parameter combination.

GridSearchCV performs a brute-force search over predefined parameter combinations.

How GridSearchCV Works

Suppose we have:

Press enter or click to view image in full size

GridSearchCV tests all combinations:

• (3, uniform)
• (3, distance)
• (5, uniform)
• (5, distance)
• (7, uniform)
• (7, distance)

Then it selects the best-performing combination.

GridSearchCV Visualization

FIG.3

Python Example using GridSearchCV

from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier

model = KNeighborsClassifier()

param_grid = {
    'n_neighbors': [3,5,7],
    'weights': ['uniform', 'distance']
}

grid = GridSearchCV(
    estimator=model,
    param_grid=param_grid,
    cv=5
)

grid.fit(X_train, y_train)

print(grid.best_params_)
print(grid.best_score_)

Advantages of GridSearchCV

✅ Finds the best combination ✅ Easy to implement ✅ Works well for small search spaces

Disadvantages of GridSearchCV

❌ Computationally expensive ❌ Slow for large datasets ❌ Tests unnecessary combinations

As parameter combinations grow, GridSearchCV becomes very expensive computationally.

What is RandomizedSearchCV?

RandomizedSearchCV randomly selects combinations of hyperparameters instead of testing every possibility.

Instead of exhaustive search, it samples random parameter combinations.

RandomizedSearchCV is more computationally efficient than GridSearchCV for large search spaces.

How RandomizedSearchCV Works

Instead of checking every combination:

Press enter or click to view image in full size

RandomizedSearchCV randomly selects a fixed number of combinations.

Example:

• only 10 random combinations tested

This significantly reduces computation time.

RandomizedSearchCV Visualization

from sklearn.model_selection import RandomizedSearchCV
from sklearn.neighbors import KNeighborsClassifier

model = KNeighborsClassifier()

param_dist = {
    'n_neighbors': range(1,50),
    'weights': ['uniform', 'distance']
}

random = RandomizedSearchCV(
    estimator=model,
    param_distributions=param_dist,
    n_iter=10,
    cv=5,
    random_state=42
)

random.fit(X_train, y_train)

print(random.best_params_)
print(random.best_score_)

Advantages of RandomizedSearchCV

✅ Faster than GridSearchCV ✅ Efficient for large search spaces ✅ Reduces computation cost

Disadvantages of RandomizedSearchCV

❌ May miss the absolute best combination ❌ Results depend on random sampling

GridSearchCV vs RandomizedSearchCV

FIG.4

Press enter or click to view image in full size

Random search often finds strong results faster in high-dimensional parameter spaces.

Important Parameters in SearchCV

1. cv

Defines cross-validation folds.

Example:

cv=5

Means:

• dataset split into 5 parts
• model trained and validated 5 times

2. scoring

Defines evaluation metric.

Example:

scoring='accuracy'

For imbalanced datasets:

• F1-score
• precision
• recall

may be better than accuracy.

3. n_iter

Used only in RandomizedSearchCV.

Controls number of random combinations tested.

Example:

n_iter=20

Common Beginner Mistakes

1. Very Large Parameter Grids

Huge parameter spaces make GridSearchCV extremely slow.

Start with smaller ranges first.

2. Tuning on Test Data

Never use test data during tuning.

Correct workflow:

• Train set → tuning
• Test set → final evaluation

Data leakage during tuning is a common beginner mistake.

3. Using Accuracy for Imbalanced Data

Accuracy can be misleading.

For fraud detection or medical diagnosis:

• F1-score
• Recall
• Precision

are often better metrics.

Real-World Applications

Hyperparameter tuning is widely used in:

• Fraud Detection
• Recommendation Systems
• Healthcare
• NLP
• Computer Vision

Companies use tuning to optimize:

• model accuracy
• training efficiency
• production performance

Conclusion

Hyperparameter tuning is one of the most important steps in Machine Learning.

In this article, we explored:

✅ Hyperparameters ✅ GridSearchCV ✅ RandomizedSearchCV ✅ Cross-validation ✅ Grid Search vs Random Search ✅ Common tuning mistakes

A good Machine Learning model is not only about algorithms — it is also about choosing the right hyperparameters.