Ensemble Learning: Bagging & Random Forest — A Complete Guide

From zero to hero with simple analogies, code, and visuals

Read here for free

1. Ensemble Learning

Imagine you ask one person to guess the temperature outside. They might be wrong.

Now imagine you ask 100 people and take the average of all their guesses. Much more accurate, right?

That’s Ensemble Learning — combining multiple weak models to build one strong model.

💡 Key Insight: Each individual model makes some mistakes. But when combined, errors cancel out and correct predictions reinforce each other.

The 3 Main Types

2. Bootstrapping — The Foundation

Before understanding Bagging, you need to understand Bootstrapping.

The Ball Bag Analogy

Imagine a bag of 20 numbered balls. You want to estimate the average number:

1. Pick a ball randomly
2. Note it down
3. Put it back ← this is the key! (sampling WITH replacement)
4. Repeat a few times → one bootstrap sample

Do this many times and average the results → good estimate of the true mean!

Why “with replacement”?

Without replacement, after drawing all balls once, you just get the original dataset back. With replacement, every sample is unique and slightly different — giving diversity to your models.

import numpy as np
np.random.seed(1)

dataset = np.array([5, 8, 9, 5, 0, 0, 1, 7, 6, 9,
                    2, 4, 5, 2, 4, 2, 4, 7, 7, 9])
def mean(numbers):
    return sum(numbers) / float(len(numbers))
def subsample(dataset, ratio=1.0):
    sample = []
    n_sample = round(len(dataset) * ratio)  # how many to draw
    while len(sample) < n_sample:
        index = np.random.randint(0, len(dataset))  # random index
        sample.append(dataset[index])               # pick with replacement
    return sample
print('True Mean: %.3f' % mean(dataset))  # True Mean: 4.800
ratio = 0.10
for n_bootstrap in [1, 10, 100]:
    sample_means = []
    for i in range(n_bootstrap):
        sample = subsample(dataset, ratio)
        sample_means.append(mean(sample))
    print('Samples=%d, Estimated Mean: %.3f' % (n_bootstrap, mean(sample_means)))

Output:

True Mean: 4.800
Samples=1,   Estimated Mean: 2.000   ← far off
Samples=10,  Estimated Mean: 4.450   ← getting closer
Samples=100, Estimated Mean: 4.880   ← very close! 

💡 Law of Large Numbers: More bootstrap samples = closer to the true mean. The randomness averages out!

3. Bagging — Bootstrap AGGregating

Bagging = Bootstrap Aggregating

How it Works (Step by Step)

Original Dataset (100 rows)
         │
    ┌────┴─────┐──────────┐
  Sample1    Sample2    Sample3    ← random subsets (with replacement)
(~63 rows) (~63 rows) (~63 rows)
    │          │          │
  Tree1      Tree2      Tree3     ← train one model on each
    │          │          │
   [0]        [1]        [0]      ← each predicts for a new data point
              │
        Majority Vote → 0 ✅      ← final prediction

The 3 Steps of Bagging

Step 1 — Build a tree on a random sample:

from sklearn.tree import DecisionTreeClassifier
def build_tree(sample, max_depth):
    sample = np.asarray(sample)
    tree = DecisionTreeClassifier(max_depth=max_depth)
    tree.fit(sample[:, :-1], sample[:, -1])  # X = all cols except last, y = last col
    return tree

Step 2 — Make all trees vote (majority wins):

def bagging_predict(trees, row):
    # Ask every tree for its prediction
    predictions = [int(tree.predict(row.reshape(1, -1))[0]) for tree in trees]
    # Return the most common prediction
    return max(set(predictions), key=predictions.count)

Step 3 — Tie it all together:

def bagging(train, test, max_depth, n_trees, ratio):
    trees = []
    for i in range(n_trees):
        sample = subsample(train, ratio)        # random subset
        tree = build_tree(sample, max_depth)    # train one tree
        trees.append(tree)
# Get predictions for every test row
    predictions = [bagging_predict(trees, row) for row in test]
    return predictions

Key Properties of Bagging

Models are independent of each other (parallel training)

**Reduces variance — unstable models become stable when averaged**

Works best with high-variance, low-bias models (like deep decision trees)

Does NOT significantly reduce bias

4. Random Forest — Bagging on Steroids

Random Forest = Bagging + one extra twist

In regular Bagging, all trees can use ALL features. In Random Forest, each tree only sees a random subset of features too. This makes trees even more diverse!

Regular Bagging:           Random Forest:
Each tree sees:            Each tree sees:
- Random rows ✅           - Random rows ✅
- ALL features             - Random features ✅ (extra!)

💡 Why random features? If one feature is very strong (like income for loan prediction), ALL trees in regular bagging will use it, making them correlated. Random features break this correlation → more diversity → better ensemble!

Using Random Forest for Regression

import numpy as np
from sklearn.ensemble import RandomForestRegressor

np.random.seed(0)
# Sample data
X_train = [[-2.08], [-0.09], [0.57], [-0.77], [-2.67],
           [-1.76], [2.50], [-0.87], [-0.18], [-0.66]]
y_train = [-76.7, -3.2, 21.1, -28.1, -98.3,
           -65.0, 91.8, -32.3, -6.9, -24.1]
X_test  = [[0.75], [0.62], [-0.12], [0.40], [-2.08]]
# Step 1: Create model with 5 trees
regressor = RandomForestRegressor(n_estimators=5)
# Step 2: Train
regressor.fit(X_train, y_train)
# Step 3: Predict and round to 2 decimal places
pred = np.round(regressor.predict(X_test), 2)
print(pred)
# Output: [ 0.91  0.91 -8.13  0.91 -72.0]

Key Parameters to Know

5. Base Learners — What Kind of Trees Work Best?

In Random Forest, the individual trees (base learners) are:

• Deep — grown fully without pruning
• High variance — they overfit their individual sample
• Low bias — they capture complex patterns

This sounds bad for a single tree, but it’s perfect for bagging because:

Many overfit trees + Averaging = Stable, accurate model

⚠️ Common misconception: Adding more trees to a Random Forest does NOT increase variance. It decreases it (or stabilizes it). More trees = more reliable!

6. Quick Revision Cheatsheet

BOOTSTRAPPING
├── Random sampling WITH replacement
├── Each sample is unique and slightly different
└── More samples → closer to true statistic (Law of Large Numbers)
BAGGING
├── Build N trees on N different bootstrap samples
├── Trees are INDEPENDENT (parallel)
├── Final prediction = majority vote (classification) or average (regression)
├── REDUCES VARIANCE
└── Best for high-variance models (deep decision trees)
RANDOM FOREST
├── Bagging + random feature selection at each split
├── Extra diversity between trees
├── n_estimators = number of trees
├── More trees = lower/stable variance (never increases!)
└── sklearn: RandomForestRegressor / RandomForestClassifier

7. When to Use What?

Key Takeaways

1. Ensemble = combine weak models → strong model
2. Bootstrapping = random sampling WITH replacement
3. Bagging = train multiple models on bootstrap samples, aggregate predictions
4. Bagging reduces variance, not bias
5. Random Forest = Bagging + random feature selection
6. Individual trees in Random Forest are deep and overfit — that’s intentional!
7. More trees = more stable predictions (variance never increases)