Are Deep Neural Networks Better than Gradient Boosted Trees for Credit Underwriting?

New research shows LSTMs have advantages over GBTs

In credit risk modeling, gradient boosted tree (GBT) algorithms like LightGBM have become standard practice over the last decade. Despite improvements in Deep Neural Network (DNN) algorithms, they’ve yet to show obvious, meaningful improvements on general data sets over GBTs. However, in the credit risk industry, small differences in model performance can make a big difference in a lender’s bottom line or in the usefulness and pricing of a product for a customer. The architecture of a Long Short Term Memory (LSTM) network is a good conceptual fit for the nature of evaluating a credit card customer’s financial health over time to see if they would be successful with, and use, a credit line increase.

Credit utilization is the ratio of a customer’s credit card balance to their credit limit. When GBTs are applied to credit utilization prediction, these models typically consider monthly historical data as isolated points in time and may miss important temporal patterns. For instance, they may overlook recurring large purchases or recovery trends after maxing out a card. Time series approaches, like LSTM, may be able to incorporate temporal patterns more accurately. To test this hypothesis, we operationalize a credit utilization prediction as a time series forecasting problem with a hybrid 1-Dimensional Convolutional Neural Network (Conv1D) + LSTM model.

We constructed a hybrid deep learning architecture as follows:

• Conv1D layers: To extract short-term local patterns such as monthly fluctuations
• LSTM layers: To capture long-term dependencies and trends in usage behavior
• Stacked & Bi-directional variants: Enhance the model’s ability to learn complex temporal relationships by processing data in multiple layers and both forward and backward directions.

This combination is intended to allow the model to learn both short-term behavior and long-term trends, providing a more comprehensive view than architectures that use either component in isolation.

Architecture of the Conv1D-LSTM model used to predict credit utilization over a 12-month horizon. The model takes 18 months of credit-related statement information as input and generates a 12-month forecast vector.

Our dataset combined both internal and external data sources to provide a holistic view of each customer’s credit behavior.

The same set of input features was used consistently across 18 historical statements, allowing the model to learn how these factors evolve over time for each customer. Unlike the GBT baseline, which relied on a broad set of engineered features using static aggregates of historical trends, the LSTM utilized a leaner input of raw temporal sequences. This approach allowed the model to learn these dynamics natively. By doing so, we could test whether a deep learning architecture could extract its own internal representations of risk directly from raw history to compensate for a lack of manually crafted statistics.

For each customer, the prediction target was defined as a 12-dimensional vector representing the next 12 months of utilization on a statement-by-statement basis. This sequential formulation allowed us to shift away from traditional aggregate predictions and toward a fine-grained, temporal understanding of customer behavior. By training the model to predict the entire sequence at once, we empowered it to learn trends, cycles, and evolving risk patterns over time. These are dependencies that are inherently difficult to capture with the aggregate predictions typically used in decision tree models.

We experimented with various LSTM architectures, including stacked and bidirectional variants, combined with Conv1D layers. All models included a final linear layer and were trained using a Mean Squared Error (MSE) loss function to minimize prediction error across the 12-month output vector.

To evaluate model performance, we used Kendall’s Tau as our primary evaluation metric, because it aligns closely with the goals of credit decisioning, where relative risk ranking often matters more than exact predictions. For instance, when deciding which customers may be eligible for a credit limit increase, it’s more critical to correctly rank them by expected utilization than to know the exact value. To fairly assess their performance, we evaluate both models at two levels: Aggregated performance over the last 3 statements and detailed performance across all 12 statements.

Over the final 3 statements, both models achieved a similar Kendall’s Tau of ~0.45, indicating that they are equally effective at maintaining the correct order of customers based on their utilization. In terms of RMSE, the benchmark model (0.2866) slightly outperformed hybrid model (0.2893), reflecting a marginally better fit to the actual utilization values. The preservation of rank ordering is often more critical in credit contexts, where relative customer positioning drives decision-making.

Comparison of model performance using Kendall’s Tau on the mean of the last three statements. The Conv1D + LSTM hybrid model achieves a performance score on par with the decision tree baseline. Error bars represent the 95% confidence interval.

Performance across 12 statements: To compare the hybrid model’s performance with the decision tree-based benchmark model on a statement-by-statement basis, we took the following approach: We used the benchmark’s average prediction across the last 3 statements as the predicted value for each individual statement. Kendall’s Tau was then calculated for each statement by comparing these constant predictions to the actual utilization values. While this approach isn’t perfect, it provides a practical way to compare the two models across all statements.

The hybrid model maintained strong ranking performance in the earlier months, but Kendall’s Tau values declined steadily as the forecast horizon extended. This degradation highlights a limitation in long-range predictions, likely due to the absence of intermediate feedback signals during training. While overall performance remained higher or comparable to the benchmark model, this downward trend underscores the challenge of maintaining ranking accuracy deeper into the prediction window, which is a key consideration for long-term credit planning.

Kendall’s Tau performance of the Conv1D + LSTM model compared to the decision tree baseline across the 12-statement forecast horizon. The plot shows strong ranking accuracy in the early statements, with a gradual decline over time for both models. This highlights the challenge of long-term prediction while demonstrating that the sequential model maintains comparable ranking quality to the baseline throughout the horizon.

Feature Efficiency: An interesting observation from our evaluation is that the hybrid model achieved strong performance using only 60% of the original feature set. This suggests that the model may be effectively distilling information from temporal patterns, potentially reducing the heavy reliance on the vast array of static features typically required by decision tree models. While further research is needed to determine the exact performance floors of GBTs under similar constraints, these initial results are promising. They indicate that capturing dynamic signals through a time-series lens could lead to more scalable and robust production models.

Model comparison showing the Conv1D + LSTM achieves similar RMSE and Kendall’s Tau as the Decision Tree while using only 60% of the features.

Lessons Learned

• The Conv1D + LSTM hybrid model achieves predictive accuracy comparable to traditional decision tree models, based on RMSE.
• It maintains similar Kendall’s Tau scores, showing a strong ability to preserve relative customer rankings, which is vital for credit decisioning.
• Interestingly, the LSTM model reached this level of performance while utilizing 40% fewer features. This suggests that the architecture is more efficient at distilling information from raw temporal sequences natively, potentially reducing the heavy reliance on the broad sets of manually engineered static aggregates typically required by decision trees.
• Performance decreases over longer forecast horizons, with Kendall’s Tau declining across the 12-statement prediction window, highlighting challenges in long-range prediction that we aim to address in future work.
• A key challenge remains interpretability: unlike decision trees, deep learning models like LSTMs are less transparent, making it harder to explain decisions to stakeholders or regulators.

Although in our experiments the LSTM architectures and features investigated did not outperform a GBT-based approach in terms of accuracy, the advantages of the LSTM may make it, or another deep learning architecture, an attractive alternative to GBTs in the near future.

Acknowledgements:

I would like to thank Lee Murray, Seetha Pothapragada, Carlo di Francescantonio, and Juan Rios for their insightful discussions and thoughtful review of this work. I am also grateful to Jason Capehart and Mike Kuhlen for reviewing and editing this article. Special thanks to Jason for his support and for enabling me to work on this project.