How Gradient Boosting Uses Gradients Internally — Regression

Hi everyone,

It’s been a while since I last shared anything, and over the past few weeks, I’ve been exploring topics I felt were worth diving deeper into.

This time, I’ve decided to focus on ensemble methods — particularly Gradient Boosting and XGBoost. Most of you are probably familiar with terms like the output score, similarity score, and initial prediction used in boosting algorithms. But in this post, let’s go one step further — and actually derive these formulas to understand how and why they’re constructed the way they are.

We’ll explore how the algorithms iteratively improve by minimizing a loss function using gradient-based updates, much like how gradient descent works in regression.

⚠️ Note: To get the most out of this post, it helps to already be familiar with the workflow of gradient boosting — this write-up focuses on the math behind each step rather than the basic flow.

Let’s get started.

We can break down the computation logic into four main steps:

1. Initial prediction
2. Computing residuals for each data point
3. Creating terminal regions
4. Computing the output value for each terminal region
5. Initial Prediction

To begin, we need a dataset (x,y) and a differentiable loss function. We’ll use the Mean Squared Error (MSE) as the cost function for regression tasks and Log Loss for classification, as these are commonly used and well-understood.

Loss function: L(y, γ)

The above equation indicates that we need to choose the value of γ that minimizes the overall loss function. This aligns with the fundamental goal behind many optimization and convergence-based algorithms — to reduce the loss as much as possible at each step.

To find the minimizing γ, we take the derivative of the loss with respect to γ and set it to zero:

This is precisely why we compute the average of the target values as the initial prediction in gradient boosting models

2. Computing Residuals

For each tree t=1 to T, compute the residuals for each data point i=1 to n as:

This means we compute the negative gradient of the loss function evaluated at the current prediction Ft−1(x), which becomes the “residual” for the next tree to fit.

3. Create Terminal Regions

4. Compute output score

For each leaf j=1,2,…,Jm​ of the mth tree:

We want to find the best constant γjm​ to assign to all points in that leaf, by minimizing the loss when added to the current model prediction Fm−1(x).

Take derivative w.r.t. γ:

This is the average residual in that leaf.

Model Prediction Update in Gradient Boosting

We will be repeating this process for mmm trees, and with each tree added, there will be a reduction in residuals, which gradually improves the model’s predictions.

To simplify the derivative, we often use a 1/2 ​ factor in the MSE loss:

This leads to a cleaner derivative:

The negative of this gradient, is called the pseudo-residual, which guides how the next tree should fit the remaining error.

I believe this explanation will be especially helpful for those who are already familiar with how gradient boosting works at a high level, and now want to understand the underlying math behind each step.

In my next blog, I’ll cover the classification version of this algorithm. After that, I’ll also extend this series to include how XGBoost works internally.

Follow for more updates on related topics!

Have a nice day, and thanks for reading.