Manual Derivation Series: Bias-Variance Decomposition, Adaboost, Boosting Tree and GDBT

Bias-Variance Decomposition: The generalization error of a model f can be decomposed into three components: bias, variance, and noise.

Bias Variance Decomposition

Boosting: Sequentially trains multiple weak learners, where each learner’s training is based on the results of the previous step. The results are aggregated (e.g., via weighted averaging or summation) to reduce bias (and also reduce variance).

Adaboost can be understood as a binary classification learning method where the model is an additive model, the loss function is the exponential loss, and the learning algorithm is the forward stagewise algorithm.

Adaboost with Exercise

Boosting Trees (specifically Gradient Boosting for Squared Error)

Boosting Tree with Exercise1

Boosting Tree with Exercise2

GBDT: It treats the prediction like a coordinate in space. By chasing the negative gradient, it’s basically doing Gradient Descent in the space of functions rather than just updating weights. This is what lets you swap out the loss function for Absolute Error (MAE), Log Loss, or even custom business metrics without changing the underlying tree-building logic.

GDBT with Exexcise