From Decision Stumps to Strong Classifiers: AdaBoost Explained

Welcome to another post in my ongoing machine learning adventure. This blog is part of a series where I’m diving into the world of ML — learning, experimenting, and (hopefully) not getting completely lost in the algorithms! 😅 If you’re interested in learning along with me, check out the whole series **HERE 📚**. Or, if you’re just here for this particular topic, no worries — you’re in the right spot too..!

Now we will be learning AdaBoost. AdaBoost, short for Adaptive Boosting, is a powerful ensemble learning algorithm designed to convert several weak learners into a single strong classifier. The key idea behind AdaBoost is simple yet elegant: instead of training all models independently, it trains them sequentially, allowing each new model to focus more on the mistakes made by the previous ones. A weak learner, in this context, is a model that performs only slightly better than random guessing. While such models may seem useless on their own, AdaBoost demonstrates that their careful combination can lead to surprisingly strong performance.

The most common weak learner in AdaBoost is a decision stump. A decision stump is a decision tree with depth = 1. Because of this limitation, a stump can only create a very simple decision boundary — essentially a straight line in the feature space. On complex datasets, such a model will inevitably make many mistakes, but this weakness is exactly what AdaBoost exploits.

How AdaBoost Works :

Step 1: Initial Weights Assignment AdaBoost begins by assigning an equal weight to every training example in the dataset. If the dataset contains n rows, each row is given an initial weight of 1\n​. For example, if there are 5 data points, each point starts with a weight of 0.2. At this stage, all samples are treated as equally important.

Step 2: Training the First Weak Learner A weak learner, typically a decision stump (a decision tree with depth 1), is trained using the current sample weights. The algorithm chooses the feature and split that minimizes weighted error, often using entropy or information gain as the splitting criterion.

Step 3: Calculating the Model Weight (α) After training the weak learner, its performance is evaluated. The error rate e is computed as the sum of the weights of all misclassified data points. Based on this error, the model’s influence (α) is calculated using the formula:

If the error is very small (e→0), the value of α becomes large, giving the model high influence. If the error equals 0.5 (random guessing), α becomes zero, meaning the model contributes nothing. If the error is greater than 0.5, α becomes negative, and the model’s predictions can be flipped to make them useful.

Step 4: Updating Data Weights The weights of the training samples are updated to emphasize the difficult examples. Misclassified samples have their weights increased using:

Correctly classified samples have their weights decreased using:

This ensures that incorrectly classified points become more important in the next iteration.

Step 5: Normalization of Weights After updating, all sample weights are normalized by dividing each weight by the total sum of weights. This step ensures that the weights again sum to 1, maintaining numerical stability.

Step 6: Up-sampling for the Next Stage To prepare data for the next weak learner, AdaBoost performs up-sampling based on the updated weights. A cumulative weight range is created, and random numbers between 0 and 1 are generated. Data points with higher weights occupy larger ranges and are therefore more likely to be selected multiple times in the new dataset. This process ensures that harder-to-classify points dominate the training of the next model.

Step 7: Iteration Steps 2 through 6 are repeated for a fixed number of estimators. Each iteration adds a new weak learner that focuses increasingly on previously misclassified samples.

Step 8: Final Prediction Once all weak learners are trained, predictions are combined using a weighted vote:

The final class prediction is determined by the sign of the weighted sum of all weak learners’ outputs. If the result is positive, the model predicts the positive class; if negative, it predicts the negative class.

Hyperparameters in AdaBoost

**base_estimator This defines the weak learner used by AdaBoost. By default, it is a decision stump** (a decision tree with max_depth = 1). Decision stumps work best with AdaBoost because the algorithm is designed to combine many very simple models. Although other models can be used, decision stumps almost always give the best results.

**n_estimators This specifies the number of weak learners trained sequentially. A very small value leads to underfitting and an overly simple decision boundary, while a very large value can cause overfitting** by creating a highly complex and jagged boundary.

**learning_rate** The learning rate scales the influence of each weak learner. A smaller learning rate reduces the impact of each step, slowing down learning.

*α_new = learning rate α_old**

This technique, known as shrinkage, helps control overfitting, especially when using a large number of estimators.

**algorithm AdaBoost supports SAMME and SAMME.R**. SAMME.R (the default) uses probability estimates instead of class labels, allowing faster convergence and better performance with fewer estimators.

Bagging vs Boosting: Key Differences

1. Type of Base Model Used

The main goal of ensemble learning is to achieve low bias (good training accuracy) and low variance (stable performance on unseen data).

Bagging uses base models that have low bias but high variance. A classic example is a fully grown decision tree. Such models fit the training data very well but are highly sensitive to small changes in the dataset. Bagging reduces this variance by training multiple models on different random subsets of the data.

Boosting, on the other hand, uses base models that have high bias but low variance. These models are intentionally simple and weak, such as decision stumps (trees with depth 1). Boosting works by sequentially reducing this high bias, gradually improving the accuracy of the combined model.

2. Type of Learning (Parallel vs Sequential)

Bagging follows a parallel learning approach. Multiple base models are trained independently and simultaneously on different bootstrap samples of the dataset. Since models do not depend on each other, the order of training does not matter.

Boosting follows a sequential learning approach. Models are trained one after another, where each new model focuses on correcting the mistakes made by the previous ones. Because of this dependency, the order of models is crucial.

3. Weightage of Base Models

In Bagging, all base models contribute equally to the final prediction. Each model has the same voting power, similar to a democratic system where every vote counts equally.

In Boosting, base models are assigned different weights based on their performance. Models with fewer errors receive higher weights and have more influence on the final prediction, while weaker models contribute less.

So, this was all about AdaBoosting from my side. If you wanted to learn more you can just do google search or directly ask our new best friends (Chatgpt or what-ever you use). See you next time.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

You can buy me coffee at :- https://www.buymeacoffee.com/aryanbisht

Thanks for being a part of our community! Before you go:

• 👏 If you find this helpful Clap for the story and follow the author
• 🔔 Follow us: LinkedIn, *Twitter*