Can we predict which clients default on their credit card payment?

Introduction

Have you ever wondered how the financial responsibility of credit card holders can be assessed? One measure that banks are interested in is whether or not a client will default on their credit card payment. Banks want to know if a client can be trusted to make payments and pay back their debt! Conveniently, Kaggle provides access to 30 000 credit card clients’ data we can analyze to address this concern. With supervised machine learning, we can use this data to predict whether or not a client will default on their credit card payments.

The dataset

Kaggle’s Default of Credit Card Clients Dataset contains 30 000 examples which correspond to 30 000 unique credit card clients in Taiwan. Of the 24 dataset features, it includes information on each client’s demographic (age, sex, income, etc.) and their monthly billing, payment, and credit history over a 6-month period from April 2005 to September 2005. Most importantly, each client is labelled with whether or not they defaulted payment the month after this period.

The problem

We can approach this as a binary classification problem and train a model to make a prediction on whether a credit card client defaults (positive class) or not (negative class). Intuitively, we see how a classifier model can use the relevant client demographic and credit history data to make a good prediction.

Prepare the training data: EDA and feature engineering

We notice that all columns are numerical and there are no missing values. We will need to transform PAY_X(repayment status), SEX , EDUCATION, and MARRIAGE into one-hot encoded categorical variables.

Notice we have important columns with X=1, 2, 3, 4, 5, 6 corresponding respectively to the 6 months between September 2005 to April 2005:

• PAY_X is repayment status in month X (-1=pay duly, 1=payment delay for one month, 2=payment delay for two months, … 8=payment delay for eight months, 9=payment delay for nine months and above)
• BILL_AMTX is bill statement amount (NT dollar) in month X
• PAY_AMTX is previous payment amount (NT dollar) in month X

What differences can we see between non-default and default payment clients?

We observe that the average monthly outstanding (BILL_AMTX- PAY_AMTX) amount over the last 6 months is slightly higher for default payment clients, meaning they owe slightly more to the bank:

• non-default payment clients = $39606.97
• default payment clients = $40086.01

We also observe that the average balance limit (LIMIT_BAL ) is much lower for default payment clients which means their spending is more restricted:

• non-default payment clients = $178297.94
• default payment clients = $131633.46

Visualization Balance limit vs Age of client scatterplot

• From the scatterplot below, we see that most default payment clients are lower in age and have lower balance limits.

Do we have class imbalance that may influence our model?

The target feature we are predicting, default.payment.next.month shows there are 4688/21000 default payment clients, making a 22% positive class rate. When a model is trained on data with a low rate of positive class examples, the model may be biased to predicting the false class. There are ways we can ensure our model doesn’t fall into this bias. Since the risk associated with predicting a False Positive (incorrectly predicting a client defaults) or False Negative (failing to predict a client that defaults) is relatively equal for this problem, we won’t need to change the way we train the model to account for the class imbalance. However, we can choose prediction accuracy and F1 score as evaluation metrics to evaluate if the model is overly biased to make negative predictions on the data. We will explore what these evaluation metrics mean later.

Feature engineering

• We obtain a more meaningful variable, OUTSTAND_AMTX (BILL_AMTX — PAY_AMTX) that tells us the difference between a client’s last payment and their monthly bill, demonstrating the monthly debt they may collect.
• We identify the following numerical (numeric_feats) and categorical (categorical_feats) features, and the features we exclude from the model (drop_feats):

numeric_feats = ["LIMIT_BAL", "AGE", "OUTSTAND_AMT1", "OUTSTAND_AMT2", "OUTSTAND_AMT3", "OUTSTAND_AMT4", "OUTSTAND_AMT5", "OUTSTAND_AMT6"]  # apply scaling
categorical_feats = ["EDUCATION", "MARRIAGE", "PAY_0", "PAY_2", "PAY_3","PAY_4", "PAY_5", "PAY_6"]  # apply one-hot encoding
passthrough_feats = ["ID", "SEX"]  # do not apply any transformation
drop_feats = ["BILL_AMT1", "BILL_AMT2", "BILL_AMT3", "BILL_AMT4", "BILL_AMT5", "BILL_AMT6", "PAY_AMT1", "PAY_AMT2", "PAY_AMT3", "PAY_AMT4", "PAY_AMT5", "PAY_AMT6"]  # do not include these features in modeling

• It is necessary to apply scaling on all numeric features and one-hot encoding on the categorical features so the classification model can learn the significance of the feature values.
• PAY_X has now been encoded into columns PAY_X_-1 , PAY_X_1 , …, PAY_X_9 corresponding to each repayment status

Now that we have meaningful features, let’s dive into the classification model results.

Which classifier model should be used?

Classifier model training results

We trained and explored a variety of different classifier models to see what would obtain the best results on training data. To choose a model, we are looking at mean cross-validation (CV) accuracy rather than training accuracy. Evaluating based on training accuracy alone will lead to us selecting a model that is overfitted to the training data.

Listed below are the models from best-performing to worst-performing. These models were evaluated after hyperparameter tuning each model to improve their accuracy performance on training data.

Best performing: The Gradient Boosting classifier model had the highest mean cross-validation accuracy of ~0.82. This means that on varying validation splits of the training data, it on average predicted default or non-default status correctly on ~82% of the examples.

This model performed equally as well as the linear Logistic Regression model, which means tuned non-linear models can capture the same relationships in our data with strong generalization.

1. Gradient Boosting classifier

• ‘learning_rate’= 0.1
• ‘max_depth’= 3
• Mean cross-validation (CV) accuracy is ~0.820

2. Logistic Regression with C=10.0

• We used GridSearchCV for hyperparameter tuning to find C=10.0
• Mean cross-validation (CV) accuracy is ~0.820

3. Random Forest

• ‘max_depth’: 10
• ‘max_features’: ‘sqrt’
• ‘n_estimators’: 100
• Mean cross-validation (CV) accuracy is ~0.818

4. KNN classifier

• n_neighbors=9
• Mean cross-validation (CV) accuracy is ~0.762

Gradient Boosting results on test data

Using the Gradient Boosting classifier, which was our best model obtained from its mean CV training scores, we find that it has a ~82% prediction accuracy on the test data! This means it performs equally as well as it did in the training phase. In addition to the prediction accuracy, let’s take a look at its F1 score to ensure there is no bias from the data’s class imbalance.

Precision = 0.655 and Recall = 0.359

• the model only correctly identified 35.9% of all actual positive classes (default clients).
• However, the model is moderately confident when it makes a positive prediction since 65.5% of its positive predictions were correct.

F1 Score = 0.464 (harmonic mean of Precision and Recall)

• a F1 score < 0.5 reflects an imbalance between Precision (proportion of correct positive predictions) and Recall (true positive rate).

The model is not doing a great job at capturing the minority negative class, even though it’s somewhat precise when it does.

What features are influencing the predictions?

You may be curious how the Gradient Boosting model is making its predictions at such a good accuracy. We can look at the model feature importance to see which features are “important” and influence the predictions the most.

SHAP Summary Plot

Below is a SHAP Summary Plot (on training data) that gives us a summary across all examples of what features and values have high SHAP scores (high influence on a positive prediction). Negative SHAP scores influence a negative prediction.

SHAP Summary Plot — Gradient Boosting

Observations from the SHAP Summary Plot

• Having a 2 month payment delay repayment status in September, 2005 (PAY_0_2 = 1) has the highest SHAP scores, and thus highest impact on positive model predictions! Absence of this class (PAY_0_2 = 0) has distinctly negative SHAP values, meaning the model is more likely to have a negative prediction for a different status in September, 2005.
• Features representing 2–3 month payment delays in the earlier months of this period (PAY_2_2, PAY_2_3, PAY_3_2, PAY_4_2, PAY_5_2, PAY_6_2), also have moderately high impact on a default client prediction.
• Low balance limits (LIMIT_BAL) have high impact on positive default client predictions, while high balance limits influence negative non-default predictions.
• Low outstanding amounts from July to September 2005 (OUTSTAND_AMT1, OUTSTAND_AMT2, OUTSTAND_AMT3) influence negative predictions that the client doesn’t default.

SHAP Force Plot

We can also analyze the feature importance of an individual prediction. Below, is the SHAP Force Plot for a random prediction example from the test data set.

SHAP Force Plot

Observations

• f(x) = -1.75 is the model’s SHAP value output for this particular sample (in log-odds). Since this is less than 0, the model predicts the negative class for this example.
• PAY_0_0 = 0.0 and LIMIT_BAL = -0.98 are red, meaning they increase the log-odds of the positive class (likelihood for default client prediction). These features make the model more likely to predict positive, but not enough to overcome the blue (negative) features.
• OUTSTAND_AMT4 = -0.95and multiple PAY_x_x values are blue and negative/close to 0, meaning they decrease the probability of predictive the postive class (default client).
• Together, these features “push” the output to -1.63, so the model predicts the negative class (non-default client).

Conclusion and Caveats

Out of the models we trained, we found that the Gradient Boosting classifier model provided the strongest predictive performance on both validation and test sets with ~82% accuracy!

However, while accuracy and precision were high, the recall and F1 score was relatively low — suggesting the model tends to predict the majority class (non-default client) from our unbalanced dataset.

So, our model is not is not doing a great job at predicting when a client defaults, even though it’s somewhat precise when it does.

Caveats

1. The low recall and F1 score suggests possible optimization bias, caused by tuning the hyperparameters for the Gradient Boosting classifier too closely to the validation set during training. Hyperparameter tuning provides this risk and could be explored further to prevent this.
2. The model’s tendency to predict non-default clients could also be improved by using class weights or threshold tuning with a Precision-Recall curve, oversampling, or trying the XGBoost model to balance the precision and recall metrics.
3. Despite using mean CV accuracy to choose the best model, we are not eliminating all risk of over-fitting. We chose Gradient Boosting because it had the minimal overfitting compared to Random Forest (slight overfitting), and KNN (low overfitting). With hyperparemeter tuning, we were able to manage this, but there is still a risk.