GRANDE: Teaching Decision Trees to Learn Like Neural Networks

How gradient descent can train entire decision tree ensembles end-to-end — and beat XGBoost.

Paper: Marton, S., Lüdtke, S., Bartelt, C., & Stuckenschmidt, H. (2024). “GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data.” ICLR 2024. Code: github.com/s-marton/GRANDE This blog discusses the above paper for educational purposes. All ideas belong to the original authors.

The Problem

Tabular data — spreadsheets, CSVs, databases — is the most common data type in industry. And despite all the deep learning hype, tree-based models like XGBoost still beat neural networks on tabular data in most benchmarks.

But tree methods have a flaw: they’re greedy. XGBoost builds trees one at a time, and once a tree is locked in, it can’t be revised. What if we could train all trees simultaneously, optimizing every split at once using gradient descent? That’s GRANDE.

How GRANDE Works

1. Trees as Math. Traditional decision trees use if-else rules that can’t be differentiated. GRANDE reformulates them as arithmetic functions (addition and multiplication), expanding feature indices into matrices for GPU-parallel computation. This makes trees compatible with gradient descent.

2. Softsign Splits. Since hard 0/1 splits aren’t differentiable, GRANDE uses a scaled softsign function as a smooth approximation during the backward pass, while keeping hard splits in the forward pass via a straight-through operator. Softsign maintains non-zero gradients even for distant values — unlike sigmoid (which vanishes) or entmoid (which drops to zero).

S(z) = ½ × ( z / (1 + |z|) + 1 ) // scaled to (0, 1)

3. Instance-Wise Weighting. This is the key innovation. Instead of giving each tree one global weight, GRANDE assigns one weight per leaf node. For each data point, the leaf it lands in determines how much that tree matters — so easy samples can rely on a single simple tree while hard samples leverage the full ensemble.

💡 Why this matters: On the PhishingWebsites dataset, GRANDE learns that for obvious phishing pages, one tree of depth 2 captures 94% of the prediction. The model is simultaneously simple where it can be and complex where it needs to be.

Results

Evaluated on 19 binary classification datasets from a predefined, unbiased OpenML benchmark:

GRANDE outperformed XGBoost (MRR 0.417), CatBoost (0.570), and NODE (0.395). It was especially strong on smaller datasets and — importantly — also won with default hyperparameters, meaning no expensive tuning required.

Limitations

The paper only evaluates binary classification — no multi-class or regression. Runtime is 250× slower than XGBoost (47s vs 0.175s), a GPU is required, and LightGBM wasn’t included as a baseline. Scalability to million-row datasets is untested.

Key Learnings

1. Greedy sequential training is a real bottleneck — jointly optimizing all trees produces better ensembles.

2. Hard, axis-aligned splits are the right inductive bias for tabular data. Deep learning’s smooth representations are fundamentally mismatched.

3. Instance-wise weighting improves both accuracy and interpretability — models can be simple for easy patterns and complex for hard ones.

4. No model wins everywhere. GRANDE is best seen as a powerful addition to the ML toolkit, not a universal replacement.

Future Directions

The most promising extensions include multi-class and regression support, stacking multiple tree layers for deeper architectures, learning categorical embeddings within the ensemble, and plugging GRANDE into multimodal pipelines alongside image or text models — something only possible because it’s gradient-based.