Building a Credit Risk Classifier: What Random Forest and XGBoost Actually Taught Me

I built a machine learning model for loan approval prediction as my final year project at university. The goal was to address a real problem in Sri Lanka’s banking sector, credit officers making manual, subjective lending decisions with no automated decision-making support.

The first version used Random Forest and hit 80.66% accuracy on paper. Then it fell apart in testing. The second version was a ground-up rebuild with XGBoost that actually worked. Now, a few years later, I’m the CTO of a fintech startup building AI-native wealth management software. Looking back at that project, I can see both what it got right and everything it missed.

This isn’t a tutorial. It’s an honest look at a real project, the failures, the rebuild, and what I’d do completely differently if I built it today.

The Problem

Sri Lanka’s lending system has a specific gap. The Credit Information Bureau (CRIB) introduced a credit score in 2020, ranging from 250 to 900. But as the then-CRIB General Manager Nandi Anthony pointed out, CRIB only holds data on about 8 million people and organizations who have received loans, and no decision-making factor is included. The score is a snapshot of credit history. The actual lending decision is left entirely to credit officers doing manual assessments.

That manual process introduces exactly the problems you’d expect subjectivity, inconsistency, and vulnerability to bias. My project aimed to build a machine learning model that could automate the loan approval decision for mortgage loans specifically, taking the subjective judgement out of the equation.

My handwritten sketch showing the initial concept

The Dataset

The dataset came from a practicing credit officer and contained over 600 rows of mortgage loan application data. Each row captured the features that matter for a lending decision: applicant income, co-applicant income, loan amount, loan term, credit history (good or bad), marital status, education level, self-employment status, property area type (urban, semi-urban, rural), and number of dependents.

One deliberate decision from the start is that I removed the gender column from training entirely. The project’s goal was to reduce bias in lending decisions, and gender should never be a factor in whether someone qualifies for a mortgage. That column was dropped before any model saw it.

Class distribution was imbalanced, leaning toward approvals. Missing values showed up across several columns and were handled with median imputation for numerical features and mode imputation for categorical ones.

Version 1: The Failure

Version 1 was the textbook approach. I tested five algorithms side by side: Logistic Regression, Support Vector Classifier, Decision Tree Classifier, Random Forest Classifier, and Gradient Boosting Classifier.

After K-Fold Cross Validation, the results were:

• Logistic Regression: 80.48%
• SVC: 79.39%
• Decision Tree Classifier: 70.89%
• Random Forest Classifier: 77.76%
• Gradient Boosting Classifier: 77.4%

The top three went through hyperparameter tuning using Randomized Search (Grid Search was taking hours with the parameter ranges I initially set, lesson learned about computational cost). After tuning:

• Logistic Regression: 80.48%
• SVC: 80.66%
• Random Forest Classifier: 80.66%

Random Forest won. 80.66% accuracy. On paper, the project was working.

Then I actually tested it with real inputs.

Two critical problems surfaced. First, the model was over-relying on the Credit History column. Change that one input and the prediction flipped regardless of everything else. Income, loan amount, employment status none of it mattered as much as that single binary field. Second, the predicted values were inverted against the dataset. The model was saying “approved” when the training data said “denied” and vice versa, likely due to errors in how I separated features and targets and handled feature scaling.

Version 1 was a failure. Not a partial success, an actual failure that produced wrong results. I kept the notebook as-is in the repository so the progression from broken to working is visible.

Version 2: The Rebuild

I started over. New file, new approach, simpler pipeline.

The key changes:

Instead of replacing values in existing columns during preprocessing, I created new columns for the mapped values. This sounds minor but it eliminated a category of bugs where transformations were interfering with each other. Clean data in, clean features out.

I switched to XGBoost. Random Forest had the higher accuracy number, but XGBoost generalized better to unseen data. When I tested both versions with manually entered inputs, XGBoost’s predictions made sense against the training data. Random Forest’s didn’t consistently.

The feature and target separation was rebuilt from scratch with explicit variable assignment. No ambiguity about what was being predicted and what was being used to predict it.

I also built a Tkinter GUI on top of the model so predictions could be tested interactively. Enter an applicant’s details into the form fields, hit “Check Loan Status,” and get an immediate result whether the loan approved or not approved, with a credit score.

Tkinter GUI made for the application

Entering sample data for testing purposes

Result of the loan status and credit score

A note on the credit score: the score displayed in the GUI is not a real credit score. It’s the raw XGBoost prediction value multiplied by 1000 to put it in a displayable range. I was transparent about this in the report, a real credit score would need CRIB data and a completely different model. The score was there to demonstrate how the output could be extended, not to claim production readiness.

Testing with Real Scenarios

I ran 37 structured test cases against the final model, varying inputs across all feature combinations, different income levels, loan amounts, credit histories, property areas, employment statuses.

The model achieved 75% accuracy across those 37 cases. That’s lower than the training accuracy, which is expected. What mattered more was whether the failures made sense. Looking at the mismatches, many were edge cases where the inputs were borderline, moderate income with high loan amounts in specific property areas. The model was being conservative rather than making obviously wrong calls.

Test Results

What I’d Do Differently Today

Working in fintech now, building AI systems where outputs directly affect financial decisions under FCA regulation, I see the gaps clearly.

The model treated accuracy as the primary metric. In credit scoring, that’s naive. Approving someone who defaults costs the lender money. Denying someone creditworthy costs them an opportunity. These errors have different costs. I’d use cost-sensitive learning today, weighting false positives and false negatives differently based on the business impact.

The dataset was a snapshot where each applicant as a single row of current attributes. Real credit models use temporal data. Payment patterns over months. Balance trajectories. A borrower whose debt is trending down is fundamentally different from one whose debt is trending up, even if today’s number is identical.

No explainability. The model said “denied” with no explanation of why. In financial services, that’s not acceptable. Regulations in many jurisdictions require lenders to explain rejections. SHAP values or similar interpretability methods should have been built in from the start.

No fairness audit. I removed gender from the training data, which was the right instinct. But I didn’t check whether other features were acting as proxies for demographic characteristics. Property area, for example, can correlate with demographics. Removing one column doesn’t guarantee unbiased outcomes.

No monitoring. The model was static, train once, deploy, done. In production, economic conditions change, lending patterns shift, the model drifts. I had no concept of retraining triggers or performance monitoring.

And the deployment itself, a Tkinter desktop GUI is fine for a demo. A production credit model needs to be an API endpoint with request validation, logging, versioning, and audit trails. The model is one component in a system, not a standalone application.

The Gap Between Academic ML and Production ML

University taught me to build a pipeline. Data in, features engineered, model trained, accuracy reported. That’s valuable. You need to know the tools before you can use them responsibly.

But production ML in financial services is a different discipline. The model is maybe 20% of the work. The other 80% is what surrounds it, data quality, monitoring, explainability, fairness, regulatory compliance, graceful degradation when inputs are missing, and the organizational processes for deciding when to retrain.

At Avagance, where we’re building AI tools for UK wealth management, I think about this gap constantly. We have calculation engines that return explicit “unavailable” states rather than guessing when data is missing. We have audit trails for every AI-generated output. We have compliance gates that check AI work before it reaches a client.

None of those concepts appeared in my university project. They didn’t need to. But the distance between “it works on a dataset” and “it works in the real world, with real consequences, under regulatory scrutiny” is where most of the actual engineering lives.

That credit scoring project was where I started learning to build ML systems. The work I’m doing now is where that thinking had to go.

Dinuwan Fernando is Co-Founder and CTO of Avagance, an AI-native wealth management platform being built for UK financial advisors. He writes about building AI systems for regulated financial services.

The full project code is available at: https://github.com/dinuwanfernando/credit-score-prediction-ml