AdaBoost Explained Simply (Part 2.1: Building)

[AdaBoost — Building 1]

In this article, we describe how to create a forest of stumps using AdaBoost from the following example dataset to predict if patients have heart disease.

Example Data:

Step 1:

Give each sample a weight that indicates how important it is to be correctly classified. At the start, all samples get the same weight 1/8 which is 1/number of samples.

These weights the samples all equally important. However, after we make the first stump, these weights will change in order to guide how the next stump is created.

Step 2:

Now we need to make the first stump in the forest. This is done finding the variable, Chest Pain, Blocked Arteries or Patient Weight, that does the best job classifying the samples.

NOTE: Because all of the weights are the same, we can ignore them right now.

Now we calculate the GINI Index for the stumps.

Since the Gini Index for Patient Weight is the lowest, this stump will be the first stump in the forest.

Step 3:

Now we need to determine how much say this stump will have in the final classification. (Remember: In the forest of stumps, some stumps get more say in the final classification than others.)

We determine how much say a stump has in the final classification based on how well it classified the samples. This stump made 1 error for the patient who weighs less than 176, has heart disease, but the stump says they do not. (as shown in the right leaf node).

The Total Error for a stump is the sum of the sample weights associated with the incorrectly classified samples. In this case, the Total Error is 1/8.

NOTE: Because all of the Sample Weights add up to 1, Total Error will always be between 0, for a perfect stump, and 1, for a horrible stump.

We use the total error to determine the Amount of Say this stump has in the final classification with the following formula.

We can draw a graph of the Amount of Say (y-axis) (x-axis is the Total Error):

The figure tells us the Amount of Say for Total Error values between 0 and 1.

We can observe that:

NOTE: If the Total Error is 1 or 0, then this equation will freak out. In practice, a small error term is added to prevent this from happening.

With patient Weight > 176, the Total Error is 1/8, so we just use the formula to get the Amount of Say = 1/2log(7)=0.97. Therefore, the Amount of Say that this stump has on the final classification is 0.97.

Step 3 (Extra Example):

Here is one more example to demonstrate how the Amount of Say is computed for the stump which uses Chest Pain. NOTE: We will not add this stump in our forest. We just take the computation of Amount of Say for this stump as an example and thus you will have a better understanding.

We can see the stump made 3 errors on the three patients:

Ending:

Now we know how the sample weights for the incorrectly classified samples are used to determine the amount of say each stump gets.

Next: We need to learn how to modify the weights so that the next stump will take the errors that the correct stump made into account.

This note summarises the Youtube Videos published by Josh Starmer (Youtube Account: StatQuest with Josh Starmer). I would like to say a big thank you to him and his super useful videos!