Build a Fraud Detection System at Production Scale

A complete ML engineering walkthrough of data pipelines, boosting models, online feature store, monitoring, and automated retraining.

Introduction

I wanted to explore the classical ML field quite a lot, so I thought building a fraud detection system with boosting models would give proper exposure to that domain. Until now, we have mainly focused on agentic and other GenAI projects. So let’s dive into this core ML part step by step. Similar to previous blogs, I won’t share all the code snippets because this project is quite large and it would be a waste of time and energy. So I’m sharing the GitHub repository URL so you can access all the code in a single place. In the blog, I will share vital points, architectural decisions, and the reasons behind them, and other details. So this blog is going to show my way of thinking and can create a proper discussion between us. You can share your thoughts and improvements on this. Let’s start.

repo: https://github.com/priyanthan07/production-fraud-detection-system

Discussion

Let’s dive into this smoothly.

Data Source

I downloaded this raw data from Kaggle for the training, validation, and other processes.

source: https://www.kaggle.com/competitions/ieee-fraud-detection/data

I downloaded this and saved it in the data/raw folder for processing. Since these files consume large amounts of storage, we cannot push them to GitHub, so anyone else who uses this needs to download them again.

Usually, in an organization, this kind of large data is collected through heavy, time-consuming work. Whenever new data comes in, it needs to be stored under a new version. Also, they won’t upload it publicly. Because of that, we need a system to manage those datasets. DVC (Data Version Control) can handle that efficiently.

It takes the large files from local storage to a common place like the cloud and maintains each version of the datasets. In the project directory, a .dvc file is created and saved for each data file. This will contain all required metadata, including the path to the source file. So whenever needed, that data can be pulled from the source in whichever version is required for the training and other processes.

Data Ingestion Process

The raw dataset contains two types of data:

Transactions

Identities

So when loading the raw data, those two datasets are merged into a single dataframe and returned for the validation process. In the validation layer, the raw data goes through multi-step checks to make sure that the data is valid for training. If the checks pass, that dataframe is forwarded for further processing. There are 11 main validation layers, which are:

1. Check the required columns exist

2. Check that the dataset is not empty

3. Check that there are no duplicate TransactionIDs

4. Check the target column isFraud only contains 0 and 1

5. Check the fraud rate is within the expected range (1% to 10%)

6. Check the TransactionAmt is non-negative

7. Check the TransactionAmt null rate is below 5%

8. Check the card1 null rate is below 5%

9. Check the ProductCD cardinality is within the expected range

10. Check that the TransactionDT is non-negative and increasing

11. Check the Identity join rate is within the expected range. If less than 1% of transactions have identity data, something is wrong

If any test fails, the process will halt until the issue is fixed before proceeding. This process runs during training.

Feature Engineering

This is one of the critical parts of this project. I say this because preparing the data for training is vital for getting a high-performing model. Without proper features, the model won’t learn the correct patterns for fraud detection. Fraudulent transactions are mostly identified based on the amount, time, transaction velocity, the user’s past activity, and other details.

Time Features

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/features/time_features.py

This is the first feature set we need to compute. The order is important because other features also depend on the time order. This creates features based on what hour of the day and what day of the week the transaction happened, among others, as shown below.

1. hour_of_day

2. day_of_week

3. time_since_last_txn_card1

4. days_since_card_first_seen

5. is_night_transaction

6. is_weekend

From these timeline-based details, the model can identify and detect unusual transactions for a user.

Example: A user usually makes transactions in the daytime and only on weekdays. If any transaction happens in the early morning or on weekends, that could be suspicious activity. So the model needs to identify those things properly.

Velocity Features

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/features/velocity_features.py

Let’s say a fraudster gets hold of cards; they try to get the money out as soon as possible before the card is blocked. So the model needs to be able to see how frequently a transaction happens on a card or from the purchaser’s email (P_emaildomain). Here, we compute the velocity for card1 and P_emaildomain columns in different time windows.

For each transaction, this computes the rolling count and sum of TransactionAmt within 1hr, 24hr, and 7-day windows grouped by group_key.

If a user usually makes one or two transactions per day and all of a sudden more than 20 transactions happen per day, then that should be flagged as fraud by the model. Now the model needs to know the velocity of each card and purchaser.

User Aggregations Feature

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/features/user_aggregations.py

Apart from the above features, the model needs to know the user’s historical aggregation and deviation features. Here, the mean, std, and z-score are computed for TransactionAmt for each combination of card1 and P_emaildomain. The deviation is computed using the current amount and each card1’s and P_emaildomain’s mean.

If any large-amount transaction happens above the mean, the deviation will be higher. So the z-score will also be high. This helps to identify outliers and flag them as fraud.

Categorical Encoding

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/features/categorical_encoder.py

The dataset contains categorical data. ML models won’t understand strings, so we need to assign numerical values to them. We cannot randomly assign 0, 1, 2, 3, … because the model might interpret higher numbers as more dominant in the prediction, which is mostly not the case.

So we are going to use target encoding. In this approach, categorical values get a numerical value based on how much that category correlates with fraud = True. So the encoding will be meaningful. But when computing this value, there is a risk of data leakage. So here I used the cross-validation method to create different folds. In each fold, the encoding is computed through the training folds and assigned to validation folds. In that way, all the rows will get the mean fraud rate for the category value without leaking. In the end, the updated dataframe and encoding file should be used during inference to encode the incoming live data.

Training Process

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/training/train.py

This is the main process to get a proper model from the preprocessed datasets. For observability, MLflow is used in most of the steps. MLflow stores the logs, analysis, and artifacts. This method runs the data loading and preprocessing steps and then splits the training and validation data based on time.

Models

code: https://github.com/priyanthan07/production-fraud-detection-system/tree/main/src/training/models.

Here we are going to use XGBoost, CatBoost, and LightGBM models, each implemented with default parameters.

Next, they are trained on the same data to compare and select the best model. Before executing the training, there is an optional tuning step using Optuna. If automatic tuning is not necessary, we can add the required parameter values for each model manually.

The entire training, evaluation, logging, and model/artifact-saving process runs inside an MLflow run. So every step is observable in the MLflow UI. For training, the optimal threshold is selected through either the f1 or recall_constrained methods.

Xgboost training overview

The trained model is evaluated through many metrics, such as:

1. AUC-ROC: measures overall discriminative ability

2. AUC-PR: more informative than AUC-ROC for imbalanced datasets, because it focuses on the positive (fraud) class

3. Precision: of all flagged fraud how many are actually fraud

4. Recall: of all actual fraud, how many did we catch

5. F1: harmonic mean of precision and recall

SHAP Analysis

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/explainability/shap_analysis.py

For each trained model, SHAP analysis is conducted to understand how the features contributed to the predictions using the validation data. This helps to find the reason behind each prediction decision.

This entire analysis is based on game theory and mathematics. All the diagrams, waterfall, beeswarm, and bar charts will be accessible through the MLflow UI.

Waterfall Diagrams

This is one of the waterfall diagrams showing a subset of the features. You can see that the features in pink are contributing to fraud detection positively; if they increase, the fraud probability will increase. Likewise, the features in blue contribute negatively.

Bar Diagram

This bar diagram shows the top n features that contribute to the model’s prediction. For a fraudulent transaction, the model currently sees these features as the most important; if they increase abnormally, the transaction will be flagged as fraud.

Beeswarm Diagram

This is another diagram I used to find the impact of the top 20 features. Based on the pink and blue color spread, we can assess each feature’s impact. I used these three for the analysis. If you want, you can also use other methods.

So, during inference, if we want to see why the model predicted a transaction as fraud, we can run the analysis based on the diagrams and come to a conclusion. Because if a card owner or business person asks for the reason behind the prediction, we cannot simply say it’s the model’s decision and that we don’t know what happened inside. To avoid those kinds of mistrust situations, using SHAP or other methods will help build trust in the ML workflow.

Model Manager

This is a standalone feature that needs to be run in the CLI manually after training is complete. The main purpose of this feature is to promote the suitable model to Staging and Production. This entire process is maintained through MLflow and its built-in functionalities.

promoted model versions in mlflow registry

Using this, we can auto-upgrade to the latest model or any specific version. This is the flow it follows:

1. Identify which model version to promote (latest if auto_select)

2. Promote it to Staging

3. Fetch its training metrics from the MLflow run

4. Run quality gates

5a. If gates pass → promote to Production

5b. If gates fail → archive with rejection reason

This is the quality gate I used:

QUALITY_GATES = {
     "auc_roc"   : 0.85,
     "auc_pr"    : 0.40,
     "recall"    : 0.60,
     "precision" : 0.30,
}

You can change these values based on your requirements.

Inference

Now, let’s see how this promoted model is used for inference.

code: https://github.com/priyanthan07/production-fraud-detection-system/tree/main/src/inference

The inference workflows are exposed through FastAPI endpoints, such as:

/health: Check if the inference server and model are ready

/metrics: Expose all Prometheus metrics in text format

/predict: Score a single transaction and return a fraud probability.

/predict/batch: Score a batch of transactions in a single request.

Here, there are some important things we need to discuss about handling incoming inference features, because there are two main categories of features: stateful and stateless.

In the training data, we created some features, such as velocity and user aggregation features. Those are based on previous transaction history as well. During EDA, those feature values are computed for each card_1 and p_domain in defined windows. There we have all the past data, so selecting the dataset within a defined window is easy and can be computed for each without any leakage.

During inference, we also want to compute velocity and user aggregation features because those are vital features for the model’s predictions. For that, we need to use an online feature store using Redis. So whenever a new transaction comes into the inference server, stateless feature creation happens through the pre-built feature pipeline. The stateful features are generated with the help of the online feature store.

online feature store

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/feature_store/online_store.py

This handles the real-time data before and after scoring during inference. It uses Redis to keep all the required data. When new data comes in, it first computes the velocity and user aggregation features using the _fetch_entity_features() method.

After making a prediction for the current transaction, that data is written to Redis through the update() method. It uses a Redis pipeline and all commands are batched and sent in a single network round trip instead of one command at a time.

1. Add to timestamp sorted set
2. Store amount by txn ID
3. Increment running sum
4. Increment sum of squares
5. Increment count
6. Set only if not exists
7. Prune timestamps older than 7 days
8. 90-day TTL

Next, the features used for prediction are appended to the data/production/scored_features.parquet file through the log_scored_features() method. This data is used for retraining the model if any data drift is detected. We will implement that below.

The log_raw_transaction() method appends the incoming requests to the data/production/raw_transactions.parquet file. This won’t be used for training; it is purely used for audit logs. This helps to replay or inspect incoming requests. In a production system, this data should be saved into a database table, but for simplicity, I saved it to a file. I highly recommend using a database for this kind of task.

Next, for drift detection, we need to store the last 30 days of data. Those are stored in the data/production/recent_predictions.parquet file. This data will be compared against the training data in order to detect drift. Also, when the inference server first starts, the online feature store won’t contain any past transaction details, so we need to pull some from the training data and update Redis through the bootstrap_from_parquet() method.

Predict

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/inference/predictor.py

Single Predict

This is the main function for making a prediction for a single transaction. It follows these steps:

01. Convert input transaction data into a dataframe for the transformations.

02. Fetch velocity + aggregation features from custom Redis.

03. Inject velocity/aggregation features into the dataframe.

04. Select model features in correct order: All the features used in training are saved in the feature_columns.txt file. The loaded data from that file determines the features during inference.

05. Predict the fraud_probability of that transaction.

06. Apply threshold to get a binary decision(fraud or not)

07. Update custom Redis AFTER scoring: This follows the above-mentioned steps to append new data and prune old data.

08. Log scored features to disk for future retraining

09. Log raw input for audit

10. Log data for drift detection

Finally, return PredictionOutput to the endpoint to show it to the caller.

Batch Predict

This receives a batch of transactions and makes predictions one by one. When data arrives, it is sorted chronologically to compute the correct velocity within the batch. Next, it uses the predict_single() method to make a prediction using a for-loop. So all the above-mentioned processes are handled for every transaction in the batch.

Drift Detection

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/monitoring/drift_detector.py

In this fraud detection system, the data we used to train the model is the main backbone of the model’s performance. If the inference data diverges from what we used for training, the model won’t be able to predict accurately whether a transaction is fraud or not. So we need a system to monitor data drift in the real world. The main reasons for data drift may be that fraudsters adapt their behaviour, new payment patterns emerge, new email domains appear, etc.

That’s what this drift_detector code does. It uses the PSI (Population Stability Index) method to detect drift, giving a numeric value based on which we can decide whether the data has drifted or not.

PSI < 0.1    -> OK, no significant shift
PSI 0.1–0.25  -> WARNING, monitor closely 
PSI > 0.25    -> CRITICAL, distribution has shifted significantly

• For this, baseline data is loaded from the file data/processed/train_features.parquet
• After that, the production data saved during the inference workflow is loaded from data/production/recent_predictions.parquet.
• Here, drift is detected only for some selected features, which are:

 # Statistical features:
 - C13
  - C14
  - card6_encoded
  - V70
  - C1
  - TransactionAmt
  - D1
  - D15
  - C2
  - C11

  # Key business features
  - card1_count_1hr
  - card1_count_24hr
  - card1_amt_mean
  - card1_amt_zscore
  - hour_of_day
  - is_night_transaction
  - is_weekend
  - ProductCD_encoded
  - P_emaildomain_encoded

• For each feature, the PSI value is computed one by one, and the status of that feature is selected based on the defined thresholds.
• Finally, the feature drift detection results are returned as the output.

Next, the target drift is computed using the compute_target_drift() method. In the above process, we computed drift for the selected features between the training and production data. Over time, the global fraud rate in a dataset might also change. Here we have selected 3.5% as the baseline, but if it increases or decreases, that might change the pattern. So drift detection of the target column is also very important.

This will run along with the Airflow DAG. If drift is detected, it will trigger the retraining process. The entire process will be automated using Airflow, as we will see below.

Re-Training

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/retraining/trigger.py

If the model’s performance is decreasing due to data drift or any other reason, we need to retrain to get the best model. By default, this first checks for data drift, if drift is True, it will trigger the retraining pipeline; if drift is False, it won’t trigger retraining.

If we want to force retraining, the pipeline will be executed without data drift detection, so a new model is produced regardless. At the end of that process, if the new model passes all the tests, it will be promoted to production through run_promotion_workflow(), as we did manually above.

Monitoring

prometheus

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/src/monitoring/metrics_exporter.py

Here we are using Prometheus to collect all the data from the system in order to send it through the metrics endpoint to the Grafana dashboard. So during every inference, we can see the model performance and other metrics very clearly. Prometheus collects these values:

1. Request counter

2. Request latency histogram

3. Fraud score distribution histogram

4. Fraud detection counter

5. Batch size histogram

6. Model version gauge

7. Error counter

8. Active requests gauge

Prometheus config: https://github.com/priyanthan07/production-fraud-detection-system/tree/main/monitoring/prometheus

This scrapes the metrics endpoint every 15 seconds and extracts all the required data.

Grafana

code: https://github.com/priyanthan07/production-fraud-detection-system/tree/main/monitoring/grafana

The configuration for the entire dashboard is defined in these files. Please go through them and make changes as needed. Grafana gets the data from Prometheus through the /metrics endpoint, so every time the dashboard pulls data through that endpoint and displays it in the UI.

Airflow Workflows

Up to now, we have implemented most of the core functions to maintain the workflow. The training and model promotion should be conducted manually on a local machine or an allocated server. The drift detection and retraining pipelines, on the other hand, should be automated through Airflow. So we are going to create some DAGs for both workflows with their triggers.

Drift Detection DAG

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/airflow/dags/drift_detection_dag.py

This DAG runs the above-defined drift detector by importing from the /src folder. After completing the process, the evaluate_drift() method evaluates the drift results and decides whether to retrain. This runs repeatedly at a defined interval *(Ex, “0 6 ”, Everyday morning at 6 AM).**

Retraining DAG

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/airflow/dags/retraining_dag.py

If drift is detected, it will trigger the retraining pipeline.

1. Load the training preprocessed data and validate it first.

2. After that, load the production features from scored_features.parquet and the accurate labels created for that data from human help are merged based on TransactionID. This merged data will be concatenated with train_features.parquet data.

3. The combined dataset should contain the data within a 6-month period; the data older than that should be pruned.

4. This newly created training dataset is saved as train_features.parquet file.

5. Using this new data, the training process will be executed as a subprocess to run it in parallel with other services.

6. After completing the training, the best model is promoted through the run_promotion_workflow() that we built above. If the new model passes the testing criteria, it will be promoted to production in MLflow.

7. Although the new model is promoted to production in MLflow, the inference server won’t load that new model unless we restart it manually after every promotion, because the inference server loads the latest production model only at startup.

8. After finishing the entire process, the new training data will be saved as the new baseline data.

Docker

We need to containerize the entire system for deployment or to host it in a different environment.

Mlflow

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/docker/mlflow/Dockerfile

1. Install all the dependencies

2. Create artifact storage directory in /mlflow/artifacts

3. Expose the MLflow UI on port 5000, so you can see all the logs and artifacts through localhost:5000

Inference

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/docker/inference/Dockerfile

1. Install all the dependencies in the pyproject.toml file.

2. Copy /src, /configs, and /monitoring directories to the container.

3. Expose the inference server on port 8000.

4. Run the health check when the container is ready to receive traffic.

5. You can access the inference endpoints through http://localhost:8000.

Airflow

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/docker/airflow/Dockerfile

1. The system dependencies are installed through the root Docker user.

2. After installation, the user is switched to airflow, to give limited permissions to the Airflow DAGs, this is done for security reasons.

3. Install project Python dependencies into Airflow’s environment using pyproject.toml.

4. Copy project source code and DAGs to the airflow directory.

5. Expose the Airflow webserver on port 8080.

docker-compose

Since we have many services that need to run in a Docker environment in an integrated manner, the docker-compose file is used to run them all with a single command and manage everything in one place. These are the services that run inside the Docker environment:

01. Postgres

02. Redis

03. Mlflow

04. Inference: Depends on the Redis, Postgres, and MLflow services. So it will only start once they are healthy.

05. Prometheus: Depends on Inference, because Prometheus can only collect metrics once the inference server is running.

06. Grafana: Depends on Prometheus, because the dashboard can only show graphs once metrics are being collected.

07. airflow-init

08. airflow-webserver

09. airflow-scheduler

All the Airflow services depend on Postgres, because the core DAG data is saved in Postgres.

To persist the Docker services’ data across restarts, it is saved in volumes. Those are:

postgres_data:
mlflow_artifacts:
prometheus_data:
grafana_data:
redis_data:

Make File

code: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/Makefile

In this system, running any particular workflow through the CLI requires commands that expose the internal file structure. For example, to trigger training:

python -m src.training.train

To avoid this kind of exposure and to simplify the commands, the Makefile provides shorthand commands. For example, to trigger training:

make train

Train with auto-tuning:

make train-tuned

Train with already-tuned parameter values:

make train-with-params

Training Process

Until now, we have only looked at the explanation of each section. Now, let’s see how the full EDA and training pipelines work along with the other features. I have already tuned parameter values in this file:

params: https://github.com/priyanthan07/production-fraud-detection-system/blob/main/data/processed/tuned_params.json

So let’s use that for this training, use this command to fetch the parameter file from the artifact store, and use it for training:

make train-with-params

Otherwise, if you need to run the training with the default parameters specified in the code, you can use make train without the params file. Since we already have the tuned parameters, let’s start training with them. Before that, we need to start all the required services in Docker as specified in the Docker Compose file.

(production-fraud-detection-system) F:\projects\production-fraud-detection-system [main ≡ +1 ~6 -0 !]> make docker-up
docker compose up -d
[+] Running 10/10
 ✔ Network production-fraud-detection-system_default                Created                                                                                       0.1s 
 ✔ Container production-fraud-detection-system-redis-1              Healthy                                                                                      13.3s 
 ✔ Container production-fraud-detection-system-postgres-1           Healthy                                                                                      13.3s 
 ✔ Container production-fraud-detection-system-mlflow-1             Started                                                                                      12.6s 
 ✔ Container production-fraud-detection-system-airflow-init-1       Exited                                                                                       37.0s 
 ✔ Container production-fraud-detection-system-inference-1          Started                                                                                      13.2s 
 ✔ Container production-fraud-detection-system-airflow-scheduler-1  Started                                                                                      37.2s 
 ✔ Container production-fraud-detection-system-airflow-webserver-1  Started                                                                                      37.2s 
 ✔ Container production-fraud-detection-system-prometheus-1         Started                                                                                      13.3s 
 ✔ Container production-fraud-detection-system-grafana-1            Started                                                                                      13.4s 
""
"Services starting..."
"  MLflow UI:    http://localhost:5000"
"  Inference:    http://localhost:8000/docs"
"  Prometheus:   http://localhost:9090"
"  Grafana:      http://localhost:3000"
"  Airflow:      http://localhost:8080"
"  Redis:        localhost:6379"

All 7 services are hosted in Docker: MLflow to track the entire process and maintain artifacts, Redis for the online feature store during inference, Prometheus to scrape the logs and send metrics to Grafana for visualization, Airflow to maintain scheduled tasks, and finally the inference server to handle inference inputs. The URLs for each UI are given in the logs above.

mlflow tracking

Logs

(production-fraud-detection-system) F:\projects\production-fraud-detection-system [main ≡ +1 ~6 -0 !]> make train-with-params
python -m src.training.train --params-file data/processed/tuned_params.json
INFO:__main__:MLflow tracking URI: http://localhost:5000
INFO:__main__:Loading cached processed features from disk...
INFO:__main__:Loaded 590540 rows, 476 columns
WARNING:__main__:Dropping object column not in drop list: id_23
WARNING:__main__:Dropping object column not in drop list: id_27
INFO:__main__:Using 432 feature columns
INFO:__main__:Train size: 472432 rows
INFO:__main__:Validation size: 118108 rows
INFO:__main__:Train fraud rate: 0.0351
INFO:__main__:Validation fraud rate: 0.0344
INFO:__main__:X_train shape: (472432, 432)
INFO:__main__:X_val shape: (118108, 432)
INFO:__main__:Loaded pre-tuned params from data/processed/tuned_params.json
INFO:__main__:  xgboost: {'n_estimators': 894, 'max_depth': 9, 'learning_rate': 0.016254839409804413, 'subsample': 0.9088214541160392, 'colsample_bytree': 0.9321256021761833, 'min_child_weight': 6, 'gamma': 3.395511357150693, 'reg_alpha': 2.6396770031101946, 'reg_lambda': 0.5521441976006747}
INFO:__main__:  lightgbm: {'n_estimators': 860, 'max_depth': 6, 'learning_rate': 0.22356289476876465, 'num_leaves': 40, 'subsample': 0.9456751725258562, 'colsample_bytree': 0.946184415287388, 'min_child_samples': 51, 'reg_alpha': 4.238103839785229, 'reg_lambda': 1.9330370045658396}
INFO:__main__:  catboost: {'iterations': 777, 'depth': 8, 'learning_rate': 0.03059462098277472, 'subsample': 0.7356739330515341, 'colsample_bylevel': 0.7808126501197397, 'min_data_in_leaf': 39, 'l2_leaf_reg': 6.65221849837647}
INFO:__main__:Skipping hyperparameter tuning. Using default parameters.
INFO:__main__:Run with --tune to enable tuning.

INFO:__main__:==================================================
INFO:__main__:Training xgboost
INFO:__main__:==================================================

INFO:__main__:Training xgboost...
INFO:src.training.models.xgboost_model:Class distribution: 455833 legitimate, 16599 fraud
INFO:src.training.models.xgboost_model:scale_pos_weight: 27.46
INFO:src.training.models.xgboost_model:Training XGBoost model...
INFO:src.training.models.xgboost_model:Parameters: {'n_estimators': 894, 'max_depth': 9, 'learning_rate': 0.016254839409804413, 'subsample': 0.9088214541160392, 'colsample_bytree': 0.9321256021761833, 'min_child_weight': 6, 'scale_pos_weight': np.float64(27.46147358274595), 'tree_method': 'hist', 'random_state': 42, 'n_jobs': -1, 'eval_metric': 'aucpr', 'early_stopping_rounds': 50, 'gamma': 3.395511357150693, 'reg_alpha': 2.6396770031101946, 'reg_lambda': 0.5521441976006747}
[0]     validation_0-aucpr:0.37666
[100]   validation_0-aucpr:0.45756
[200]   validation_0-aucpr:0.49045
[300]   validation_0-aucpr:0.50301
[400]   validation_0-aucpr:0.51296
[500]   validation_0-aucpr:0.51959
[600]   validation_0-aucpr:0.52538
[700]   validation_0-aucpr:0.53123
[800]   validation_0-aucpr:0.53510
[893]   validation_0-aucpr:0.53896

INFO:src.training.models.xgboost_model:XGBoost best iteration: 892
INFO:src.training.threshold_optimizer: Optimal threshold (recall_constrained, min_recall=0.6): 0.5693
INFO:src.training.threshold_optimizer: F1: 0.4686
INFO:src.training.threshold_optimizer: Precision: 0.3844
INFO:src.training.threshold_optimizer: Recall: 0.6001
INFO:src.training.evaluator:Evaluation metrics:
INFO:src.training.evaluator:  auc_roc: 0.9116
INFO:src.training.evaluator:  auc_pr: 0.539
INFO:src.training.evaluator:  precision: 0.3844
INFO:src.training.evaluator:  recall: 0.6001
INFO:src.training.evaluator:  f1: 0.4686
INFO:src.training.evaluator:  threshold: 0.5693
INFO:src.training.evaluator:  predicted_fraud_rate: 0.0537
INFO:src.training.evaluator:  actual_fraud_rate: 0.0344

INFO:__main__:Running SHAP analysis for xgboost...
INFO:src.explainability.shap_analysis:Running SHAP analysis for xgboost...
INFO:src.explainability.shap_analysis:Validation set has 118108 rows. Sampling 2000 rows for SHAP to control runtime.
INFO:src.explainability.shap_analysis:Computing SHAP values for xgboost on 2000 rows...
INFO:src.explainability.shap_analysis:SHAP values shape: (2000, 432)
INFO:src.explainability.shap_analysis:Saved beeswarm plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_shap_beeswarm.png
INFO:src.explainability.shap_analysis:Saved bar plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_shap_bar.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_waterfall_fraud_0.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_waterfall_fraud_1.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_waterfall_fraud_2.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_waterfall_fraud_3.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_waterfall_fraud_4.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_waterfall_legit_0.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_waterfall_legit_1.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_waterfall_legit_2.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_waterfall_legit_3.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpdmyiqxrv\xgboost_waterfall_legit_4.png
INFO:src.explainability.shap_analysis:Top 5 features by SHAP for xgboost:
      feature  mean_abs_shap  rank
          C13       0.201027     1
          V70       0.193080     2
          C14       0.179935     3
         V258       0.154837     4
card6_encoded       0.137755     5
INFO:src.explainability.shap_analysis:SHAP artifacts logged to MLflow under 'shap/xgboost'
INFO:__main__:SHAP analysis complete for xgboost.

INFO:__main__:xgboost results:
INFO:__main__:  auc_roc: 0.9116
INFO:__main__:  auc_pr: 0.539
INFO:__main__:  precision: 0.3844
INFO:__main__:  recall: 0.6001
INFO:__main__:  f1: 0.4686
INFO:__main__:  threshold: 0.5693
INFO:__main__:  predicted_fraud_rate: 0.0537
INFO:__main__:  actual_fraud_rate: 0.0344
🏃 View run xgboost at: http://localhost:5000/#/experiments/1/runs/3963676c9e7846f58485b3d6ef4ddedd
🧪 View experiment at: http://localhost:5000/#/experiments/1

INFO:__main__:==================================================
INFO:__main__:Training lightgbm
INFO:__main__:==================================================

INFO:__main__:Training lightgbm...
INFO:src.training.models.lightgbm_model:Class distribution: 455833 legitimate, 16599 fraud
INFO:src.training.models.lightgbm_model:Building LightGBM model...
INFO:src.training.models.lightgbm_model:Parameters: {'n_estimators': 860, 'max_depth': 6, 'learning_rate': 0.22356289476876465, 'num_leaves': 40, 'subsample': 0.9456751725258562, 'colsample_bytree': 0.946184415287388, 'min_child_samples': 51, 'is_unbalance': True, 'random_state': 42, 'n_jobs': -1, 'verbose': -1, 'reg_alpha': 4.238103839785229, 'reg_lambda': 1.9330370045658396}
Training until validation scores don't improve for 100 rounds
[100]   valid_0's binary_logloss: 0.26868
[200]   valid_0's binary_logloss: 0.214284
[300]   valid_0's binary_logloss: 0.17612
[400]   valid_0's binary_logloss: 0.15141
[500]   valid_0's binary_logloss: 0.130869
[600]   valid_0's binary_logloss: 0.117856
[700]   valid_0's binary_logloss: 0.114493
[800]   valid_0's binary_logloss: 0.110473
Did not meet early stopping. Best iteration is:
[858]   valid_0's binary_logloss: 0.108733

INFO:src.training.models.lightgbm_model:LightGBM training complete.
INFO:src.training.threshold_optimizer: Optimal threshold (recall_constrained, min_recall=0.6): 0.2360
INFO:src.training.threshold_optimizer: F1: 0.4623
INFO:src.training.threshold_optimizer: Precision: 0.3760
INFO:src.training.threshold_optimizer: Recall: 0.6001
INFO:src.training.evaluator:Evaluation metrics:
INFO:src.training.evaluator:  auc_roc: 0.8896
INFO:src.training.evaluator:  auc_pr: 0.5313
INFO:src.training.evaluator:  precision: 0.376
INFO:src.training.evaluator:  recall: 0.6001
INFO:src.training.evaluator:  f1: 0.4623
INFO:src.training.evaluator:  threshold: 0.236
INFO:src.training.evaluator:  predicted_fraud_rate: 0.0549
INFO:src.training.evaluator:  actual_fraud_rate: 0.0344

INFO:__main__:Running SHAP analysis for lightgbm...
INFO:src.explainability.shap_analysis:Running SHAP analysis for lightgbm...
INFO:src.explainability.shap_analysis:Validation set has 118108 rows. Sampling 2000 rows for SHAP to control runtime.
INFO:src.explainability.shap_analysis:Computing SHAP values for lightgbm on 2000 rows...
F:\projects\production-fraud-detection-system\.venv\Lib\site-packages\shap\explainers\_tree.py:620: UserWarning: LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray
  warnings.warn(
INFO:src.explainability.shap_analysis:SHAP values shape: (2000, 432)
INFO:src.explainability.shap_analysis:Saved beeswarm plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_shap_beeswarm.png
INFO:src.explainability.shap_analysis:Saved bar plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_shap_bar.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_waterfall_fraud_0.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_waterfall_fraud_1.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_waterfall_fraud_2.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_waterfall_fraud_3.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_waterfall_fraud_4.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_waterfall_legit_0.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_waterfall_legit_1.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_waterfall_legit_2.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_waterfall_legit_3.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmpxt2ns_zg\lightgbm_waterfall_legit_4.png
INFO:src.explainability.shap_analysis:Top 5 features by SHAP for lightgbm:
         feature  mean_abs_shap  rank
             C13       0.552761     1
days_since_start       0.507685     2
             V70       0.423845     3
            V294       0.323869     4
             C14       0.271505     5
INFO:src.explainability.shap_analysis:SHAP artifacts logged to MLflow under 'shap/lightgbm'
INFO:__main__:SHAP analysis complete for lightgbm.

INFO:__main__:lightgbm results:
INFO:__main__:  auc_roc: 0.8896
INFO:__main__:  auc_pr: 0.5313
INFO:__main__:  precision: 0.376
INFO:__main__:  recall: 0.6001
INFO:__main__:  f1: 0.4623
INFO:__main__:  threshold: 0.236
INFO:__main__:  predicted_fraud_rate: 0.0549
INFO:__main__:  actual_fraud_rate: 0.0344
🏃 View run lightgbm at: http://localhost:5000/#/experiments/1/runs/3325e78891b54a23977e81cc47210e13
🧪 View experiment at: http://localhost:5000/#/experiments/1

INFO:__main__:==================================================
INFO:__main__:Training catboost
INFO:__main__:==================================================

INFO:__main__:Training catboost...
INFO:src.training.models.catboost_model:Class distribution: 455833 legitimate, 16599 fraud
INFO:src.training.models.catboost_model:Building CatBoost model...
INFO:src.training.models.catboost_model:Parameters: {'iterations': 777, 'depth': 8, 'learning_rate': 0.03059462098277472, 'subsample': 0.7356739330515341, 'colsample_bylevel': 0.7808126501197397, 'min_data_in_leaf': 39, 'auto_class_weights': 'Balanced', 'random_seed': 42, 'eval_metric': 'AUC', 'early_stopping_rounds': 50, 'verbose': 100, 'l2_leaf_reg': 6.65221849837647}
0:      test: 0.8238911 best: 0.8238911 (0)     total: 591ms    remaining: 7m 38s
100:    test: 0.8751580 best: 0.8751580 (100)   total: 38.6s    remaining: 4m 18s
200:    test: 0.8869627 best: 0.8869627 (200)   total: 1m 16s   remaining: 3m 38s
300:    test: 0.8938790 best: 0.8938790 (300)   total: 2m 14s   remaining: 3m 32s
400:    test: 0.8986791 best: 0.8986791 (400)   total: 4m 31s   remaining: 4m 14s
500:    test: 0.9037131 best: 0.9037131 (500)   total: 6m 52s   remaining: 3m 47s
600:    test: 0.9072694 best: 0.9072694 (600)   total: 8m 53s   remaining: 2m 36s
700:    test: 0.9089778 best: 0.9090433 (698)   total: 11m 2s   remaining: 1m 11s
776:    test: 0.9097012 best: 0.9098507 (738)   total: 12m 35s  remaining: 0us

bestTest = 0.9098507005
bestIteration = 738

Shrink model to first 739 iterations.
INFO:src.training.models.catboost_model:CatBoost training complete.
INFO:src.training.threshold_optimizer: Optimal threshold (recall_constrained, min_recall=0.6): 0.6639
INFO:src.training.threshold_optimizer: F1: 0.4465
INFO:src.training.threshold_optimizer: Precision: 0.3554
INFO:src.training.threshold_optimizer: Recall: 0.6004
INFO:src.training.evaluator:Evaluation metrics:
INFO:src.training.evaluator:  auc_roc: 0.9099
INFO:src.training.evaluator:  auc_pr: 0.508
INFO:src.training.evaluator:  precision: 0.3554
INFO:src.training.evaluator:  recall: 0.6004
INFO:src.training.evaluator:  f1: 0.4465
INFO:src.training.evaluator:  threshold: 0.6639
INFO:src.training.evaluator:  predicted_fraud_rate: 0.0581
INFO:src.training.evaluator:  actual_fraud_rate: 0.0344

INFO:__main__:Running SHAP analysis for catboost...
INFO:src.explainability.shap_analysis:Running SHAP analysis for catboost...
INFO:src.explainability.shap_analysis:Validation set has 118108 rows. Sampling 2000 rows for SHAP to control runtime.
INFO:src.explainability.shap_analysis:Computing SHAP values for catboost on 2000 rows...
INFO:src.explainability.shap_analysis:SHAP values shape: (2000, 432)
INFO:src.explainability.shap_analysis:Saved beeswarm plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_shap_beeswarm.png
INFO:src.explainability.shap_analysis:Saved bar plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_shap_bar.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_waterfall_fraud_0.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_waterfall_fraud_1.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_waterfall_fraud_2.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_waterfall_fraud_3.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_waterfall_fraud_4.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_waterfall_legit_0.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_waterfall_legit_1.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_waterfall_legit_2.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_waterfall_legit_3.png
INFO:src.explainability.shap_analysis:Saved waterfall plot: C:\Users\Lenovo\AppData\Local\Temp\tmplkbh1oqa\catboost_waterfall_legit_4.png
INFO:src.explainability.shap_analysis:Top 5 features by SHAP for catboost:
      feature  mean_abs_shap  rank
           C1       0.227276     1
card6_encoded       0.204099     2
          C13       0.191799     3
           D2       0.146888     4
          C11       0.146275     5
INFO:src.explainability.shap_analysis:SHAP artifacts logged to MLflow under 'shap/catboost'
INFO:__main__:SHAP analysis complete for catboost.

INFO:__main__:catboost results:
INFO:__main__:  auc_roc: 0.9099
INFO:__main__:  auc_pr: 0.508
INFO:__main__:  precision: 0.3554
INFO:__main__:  recall: 0.6004
INFO:__main__:  f1: 0.4465
INFO:__main__:  threshold: 0.6639
INFO:__main__:  predicted_fraud_rate: 0.0581
INFO:__main__:  actual_fraud_rate: 0.0344
🏃 View run catboost at: http://localhost:5000/#/experiments/1/runs/df98701c024f40308315064b84d6a665
🧪 View experiment at: http://localhost:5000/#/experiments/1

Registered model 'fraud_detection_model' already exists. Creating a new version of this model...
2026/05/29 06:20:31 INFO mlflow.store.model_registry.abstract_store: Waiting up to 300 seconds for model version to finish creation. Model name: fraud_detection_model, version 3
Created version '3' of model 'fraud_detection_model'.
INFO:__main__:Best model registered as 'fraud_detection_model'.
INFO:__main__:Run 'make promote' or 'python -m src.registry.model_manager' to promote to Production.
INFO:__main__:Top 10 features by SHAP (xgboost):
INFO:__main__:      feature  mean_abs_shap  rank
          C13       0.201027     1
          V70       0.193080     2
          C14       0.179935     3
         V258       0.154837     4
card6_encoded       0.137755     5
           C1       0.136075     6
           C5       0.134086     7
          V91       0.133976     8
          C11       0.126891     9
         V294       0.125149    10

INFO:__main__:==================================================
INFO:__main__:Best model: xgboost
INFO:__main__:AUC-PR: 0.539
INFO:__main__:AUC-ROC: 0.9116
INFO:__main__:F1: 0.4686
INFO:__main__:Precision: 0.3844
INFO:__main__:Recall: 0.6001
INFO:__main__:==================================================

model promotion

The best model is promoted using this command:

make promote

The promotion logs are:

(production-fraud-detection-system) F:\projects\production-fraud-detection-system [main ≡ +1 ~6 -0 !]> make promote
python -m src.registry.model_manager

INFO:__main__:Auto-selected latest version: fraud_detection_model v3
INFO:__main__:Evaluating fraud_detection_model version 3 (run_id: 3963676c...)
INFO:__main__:Promoting fraud_detection_model version 3 to Staging...
INFO:__main__:fraud_detection_model version 3 is now in Staging.
INFO:__main__:Metrics from training run:
INFO:__main__:  auc_roc: 0.9116 (gate requires >= 0.85)
INFO:__main__:  auc_pr: 0.5390 (gate requires >= 0.4)
INFO:__main__:  precision: 0.3844 (gate requires >= 0.3)
INFO:__main__:  recall: 0.6001 (gate requires >= 0.6)
INFO:__main__:Quality gate results:
INFO:__main__:  [PASS] auc_roc: 0.9116 >= 0.85 required
INFO:__main__:  [PASS] auc_pr: 0.539 >= 0.4 required
INFO:__main__:  [PASS] recall: 0.6001 >= 0.6 required
INFO:__main__:  [PASS] precision: 0.3844 >= 0.3 required
INFO:__main__:Promoting fraud_detection_model version 3 to Production...
INFO:__main__:fraud_detection_model version 3 is now in Production. Previous Production version has been archived.
INFO:__main__:Promotion successful. fraud_detection_model version 3 is now serving traffic.

Promotion result:
  Version:  3
  Promoted: True
  Threshold: 0.5693098306655884

Now we have promoted the version 3 XGBoost model to production for inference. The local manual running process ends here.

Testing

After this, we need to use the Docker inference server to test the new model’s performance. Since MLflow is also hosted in the same container network, the promoted new model can be used by the inference server. However, the inference server only loads the latest model from MLflow at startup. So after every new model promotion, we need to manually restart the inference server to pick up the latest model from the registry.

make restart-inference

logs:
[+] Restarting 1/1
 ✔ Container production-fraud-detection-system-inference-1  Started 

Now let’s test the model performance using the batch data stored in this file:

[embed]production-fraud-detection-system/tests/data/attack_batch.json at main ·… Contribute to priyanthan07/production-fraud-detection-system development by creating an account on GitHub.github.com

Now let’s hit the /predict/batch endpoint with that transaction data.

Results:

{
  "predictions": [
    {
      "TransactionID": 9000001,
      "fraud_probability": 0.053872,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000002,
      "fraud_probability": 0.049734,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000003,
      "fraud_probability": 0.072634,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000004,
      "fraud_probability": 0.057026,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000005,
      "fraud_probability": 0.057403,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000006,
      "fraud_probability": 0.070014,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000007,
      "fraud_probability": 0.057148,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000008,
      "fraud_probability": 0.05546,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000009,
      "fraud_probability": 0.065781,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000010,
      "fraud_probability": 0.068727,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000011,
      "fraud_probability": 0.058042,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000012,
      "fraud_probability": 0.063301,
      "is_fraud": false,
      "risk_level": "LOW",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000013,
      "fraud_probability": 0.763081,
      "is_fraud": true,
      "risk_level": "HIGH",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000014,
      "fraud_probability": 0.693906,
      "is_fraud": true,
      "risk_level": "HIGH",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000015,
      "fraud_probability": 0.745029,
      "is_fraud": true,
      "risk_level": "HIGH",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000016,
      "fraud_probability": 0.784371,
      "is_fraud": true,
      "risk_level": "HIGH",
      "threshold_used": 0.5693,
      "model_version": "3"
    },
    {
      "TransactionID": 9000017,
      "fraud_probability": 0.817486,
      "is_fraud": true,
      "risk_level": "CRITICAL",
      "threshold_used": 0.5693,
      "model_version": "3"
    }
  ],
  "total_transactions": 17,
  "flagged_as_fraud": 5,
  "fraud_rate_in_batch": 0.2941,
  "model_version": "3"
}

If you look at the input data carefully, from the 13th transaction onward, the amount, C columns, and other feature values change. The model catches that change and flags those transactions as fraud. That is the correct behaviour.

Conclusion

Okay, we have come to the end of the blog. I hope you understood the explanations at each stage. Since this is a large codebase, I only included the URLs here. Please go through them for a clearer understanding. I know there are many improvements that can be made to this project. Since this is meant to provide an understanding of a basic fraud detection system, and due to time constraints, I didn’t focus on more aspects. I highly encourage you to explore more in this field for a deeper understanding and experience.

Thank You