Adaboost Algorithm

“One weak model makes mistakes → the next model is forced to focus more on those mistakes.”

This single sentence is the whole intuition.

1. Intro

AdaBoost = Adaptive Boosting

• It is a Boosting algorithm.
• It combines many weak learners to create one strong learner.
• The weak learner is usually a Decision Tree Stump.

What is a Decision Tree Stump?

A stump is a Decision Tree of depth = 1.

It can make only one split.

Example:

If Glucose > 120 → Diabetic
Else → Non-Diabetic

Only one question is asked.

So each stump is very weak, but AdaBoost combines many such stumps.

Random Forest vs AdaBoost

Random Forest

Many trees work independently and then vote.

AdaBoost

Trees work one after another, and each new tree learns from previous mistakes.

2. Maths Intuition

I remember AdaBoost using 5 steps.

• Create a stump.
• Measure its performance.
• Update weights.
• Normalize weights.
• Sample data for the next stump.

Step 1: Creating a Decision Tree Stump

Initially, all data points get equal importance.

Suppose we have 10 samples.

Each sample gets weight:

wi=1/10=0.1

Then we train the first stump.

Step 2: Performance of the Decision Tree Stump

The stump predicts the training data.

Some points are:

• Correctly classified
• Incorrectly classified

Suppose it misclassifies 2 out of 10 points.

Weighted Error

AdaBoost does not count just the number of mistakes.

It uses the sum of weights of misclassified samples.

error=∑wi (for wrong predictions)

If the wrong points have weights 0.1 and 0.1:

error=0.2

Step 3: Calculate Model Importance (Alpha)

This is the magic step.

The stump gets a score called Alpha.

α=1/2ln⁡(1−error/error)

Intuition

• Small error → Large alpha
• Large error → Small alpha

So a better stump gets more voting power.

Example:

error=0.2

α=1/2ln⁡(0.8/0.2)=0.693

This stump is fairly good.

How to Remember Alpha

Think:

Less error → More confidence → Bigger alpha

Step 4: Updating Weights

This is where “Adaptive” comes from.

Correctly Predicted Samples

Their weights are decreased.

Incorrectly Predicted Samples

Their weights are increased.

Why?

Because we want the next stump to pay more attention to the difficult records.

“Assigning Bins”

What actually happens is:

After updating weights, we create a probability distribution from those weights.

Higher weight = higher probability of being selected for the next training set.

So “bins” can be remembered as:

Weight → Probability region → Sampling chance

Step 5: Normalizing Weights

After updating, weights may not sum to 1.

So we divide each weight by the total weight.

wi=wi/∑wi

Now all weights again form a valid probability distribution.

Example:

Before normalization:

[0.05, 0.05, 0.25, 0.25, 0.10]

Sum = 0.70

After normalization:

[0.071, 0.071, 0.357, 0.357, 0.143]

We should notice how the wrong samples now have much larger probabilities.

Selecting New Data Points for the Next Tree

Using the normalized weights, we perform weighted sampling.

Misclassified points are more likely to appear in the next dataset.

Correct points may appear fewer times.

This is the exact reason why AdaBoost gradually focuses on difficult examples.

The Whole Algorithm in One Flow

• Equal weights
• Train stump
• Find weighted error
• Compute alpha
• Increase wrong weights
• Decrease correct weights
• Normalize
• Sample next dataset
• Repeat

3. Final Prediction for AdaBoost

After training many stumps, we do not simply take majority vote.

Each stump has a different alpha, so its vote has a different strength.

Suppose:

StumpPredictionAlphaTree 1+10.8Tree 2+10.6Tree 3–10.2

Final score:

0.8(+1)+0.6(+1)+0.2(−1)=1.20.8(+1)+0.6(+1)+0.2(-1)=1.20.8(+1)+0.6(+1)+0.2(−1)=1.2

Since the result is positive, the final class is +1.

Easy Recall

AdaBoost = Weighted Voting

• Good stump → Bigger vote.
• Bad stump → Smaller vote.

4. AdaBoost Model Training (Classification)

Workflow

Dataset

↓

Train-Test Split

↓

Initialize AdaBoostClassifier

↓

Fit

↓

Predict

↓

Evaluate

Practical Code

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier

base = DecisionTreeClassifier(max_depth=1)

model = AdaBoostClassifier(
    estimator=base,
    n_estimators=50,
    learning_rate=1.0,
    random_state=42
)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)

Important Parameters

estimator

The weak learner (usually a stump).

n_estimators

Number of weak learners.

More estimators → more boosting rounds.

learning_rate

Controls how much each learner contributes.

• Small → slower learning.
• Large → aggressive learning.

AdaBoost for Regression

Yes, AdaBoost can also perform Regression.

The idea is similar:

• Train weak regressors sequentially.
• Focus more on samples with large prediction errors.
• Combine predictions.

Code

from sklearn.ensemble import AdaBoostRegressor
from sklearn.tree import DecisionTreeRegressor

base = DecisionTreeRegressor(max_depth=3)

model = AdaBoostRegressor(
    estimator=base,
    n_estimators=100,
    learning_rate=0.1,
    random_state=42
)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)

Evaluation metrics:

• MAE
• MSE
• RMSE
• R²

The “Movie Director” Analogy

Imagine a movie director.

Actor 1

Forgets 20% of scenes.

Director says:

“These scenes are important. Practice them again.”

Actor 2

Focuses more on the forgotten scenes.

Actor 3

Focuses even more on the remaining mistakes.

Finally, the movie becomes good because each actor concentrated on the previous mistakes.

That is AdaBoost.

AdaBoost vs Random Forest

Random Forest{Bagging,Parallel trees, Reduces variance, Trees are independent, Majority vote, More robust to noise}

AdaBoost{Boosting, Sequential trees, Reduces bias, Each tree depends on previous one, Weighted vote (alpha), More sensitive to outliers/noisy labels}

Why is AdaBoost Sensitive to Outliers?

Suppose one data point is mislabeled.

AdaBoost keeps increasing its weight because it is repeatedly misclassified.

Eventually, the algorithm may spend too much effort trying to fit that noisy point.

When Should We Use AdaBoost?

Use it when:

• Dataset is relatively clean.
• You need better accuracy than a single tree.
• Relationships are not extremely complex.
• You want a strong baseline boosting model.

Avoid it when:

• There are many outliers.
• Labels contain significant noise.
• Dataset is extremely large (XGBoost/LightGBM may be better).

AdaBoost Inshort

• Weak learner = Decision Tree Stump (depth = 1).
• Start with equal weights.
• Train stump.
• Compute weighted error.
• Calculate alpha.
• Increase weights of wrong samples.
• Decrease weights of correct samples.
• Normalize weights.
• Train next stump on weighted data.
• Final prediction = weighted vote using alpha.

AdaBoost = “Increase weight of mistakes, train next stump, combine all stumps using weighted voting.”