Detect, Retrain, Repeat: Building a Model Drift Monitor with Alibi-Detect

Introduction

The Problem: ML Models Decay in Production

Machine learning models, once trained and deployed, aren’t immune to the passage of time. As environments change, data distributions shift — and suddenly, that once-high-performing model starts making questionable predictions.

This phenomenon is known as model drift, and if left unchecked, it can silently degrade model accuracy and business value.

What We Built

To combat this, I built a real-time model drift detection and retraining pipeline using:

• Alibi-Detect for statistical drift detection
• Simulated sensor data with injected concept drift
• Apache Airflow for automating the detection + retrain pipeline
• scikit-learn for lightweight retraining
• FastAPI for a prediction-serving interface

We call this strategy: Detect → Retrain → Repeat.

Simulating Sensor Drift

I created two datasets:

• sensor_baseline.csv – stable sensor readings with normal behavior
• sensor_drifted.csv – same sensors, but from timestamp 50 onward, the mean values shift.

# Sensor Drift Injection (from t=50)
sensor_1[50:] += 2
sensor_2[50:] += 10

Visualizing the Drift

Detecting data drift isn’t just about statistical thresholds — it’s also about understanding how feature distributions evolve over time. In our setup, we intentionally introduced drift in two synthetic sensor variables. Here’s how it shows up across different visualization techniques:

KDE Plot (Sensor 1)

The KDE plot clearly reveals the distribution shift of sensor_1 after timestamp 50. The new distribution not only shifts to the right but also becomes slightly wider, indicating both location and scale drift.

Box Plot (Sensor 1)

The box plot makes it easy to see how the median of sensor_1 increases in the drifted dataset. There's also a visible change in the interquartile range and the appearance of new outliers — strong signs of a shift in the data-generating process.

Violin Plot (Sensor 1)

A violin plot blends a box plot with a kernel density plot. Here, we see that the distribution becomes bimodal post-drift, a signal that something non-trivial is occurring — potentially due to a new process affecting the sensor.

CDF Plot (Sensor 1)

The cumulative distribution function reveals a large divergence between baseline and drifted data. The same percentile maps to drastically different values, confirming a shift that would impact any ML model trained on the original data.

KDE Plot (Sensor 2)

Just like sensor_1, the KDE plot for sensor_2 shows a rightward shift and an increase in spread. The density peak has flattened, hinting at noisier data or broader value ranges.

Box Plot (Sensor 2)

The increase in both the median and range for sensor_2 suggests the sensor is recording fundamentally different behavior — potentially due to wear, calibration drift, or environmental changes.

Violin Plot (Sensor 2)

The violin plot for sensor_2 reinforces the change in distribution shape and central tendency — making it an excellent tool to visualize subtle and complex changes in the data.

CDF Plot (Sensor 2)

Even small shifts in a CDF can create large discrepancies in model predictions, especially for models relying on percentile-based thresholds. Here, the drifted distribution is clearly more spread out and slower to accumulate probability mass.

Drift Detection Results

Once the drifted data was in place, we used Alibi-Detect’s KSDrift detector to quantify the shift. This method is based on the Kolmogorov–Smirnov test, a statistical test that compares distributions of each feature in the baseline and test datasets.

We defined a significance threshold of 0.05, meaning that any feature with a p-value lower than this is considered statistically different — i.e., drifted.

Detection Summary

• Detector Used: alibi_detect.cd.KSDrift
• Significance Level: p < 0.05
• Input Shape: (100, 2) with features sensor_1, sensor_2

Drift Status

This plot shows a binary decision (1 = drift, 0 = no drift) for each feature. Both sensor_1 and sensor_2 clearly cross the drift threshold.

p-values from KS Test

A low p-value (< 0.05) indicates strong evidence of drift. Both sensors register significant drops in p-value, confirming what we observed in the KDE and violin plots earlier.

Drift Score Magnitudes

While not used directly to flag drift, these scores (1 — p-value) give a confidence-like view of how far the test distribution deviates from the baseline. Higher scores imply stronger drift signals.

Key Insight

The combination of statistical and visual evidence makes it clear that both features underwent meaningful distributional changes — the exact condition that would warrant model retraining in a production environment.

Retraining the Model

Once drift is detected, the next logical step is to retrain the model using the latest (drifted) data to adapt to the new conditions. For simplicity, we built a minimal retraining module using:

• scikit-learn's RandomForestClassifier
• sensor_1 as a proxy label generator
• Simulated sensor data with injected drift

Retraining Logic

df['label'] = (df['sensor_1'] > 11).astype(int)
X = df[['sensor_1', 'sensor_2']]
y = df['label']

This creates a synthetic label assuming that values above a threshold imply failure/risk, mimicking real-world predictive maintenance or anomaly scenarios.

We split the drifted dataset and retrained a simple classifier:

model = RandomForestClassifier()
model.fit(X_train, y_train)
joblib.dump(model, "model.pkl")

Post-Retraining Evaluation

To evaluate model performance before and after drift, we compared predictions on the drifted dataset. Here’s a sample evaluation report:

Classification Report:
       precision    recall   f1-score   support 
           0         0.89      0.93      0.91        44
           1         0.92      0.88      0.90        46

    accuracy                             0.91        90

Even a basic classifier trained on drifted data shows recovered accuracy and balanced precision/recall, proving the importance of timely retraining.

Final Thoughts & Takeaways

Drift in production ML models is inevitable — but silent failure doesn’t have to be.

By combining:

• Alibi-Detect for robust drift detection,
• Data visualization to interpret the drift,
• Retraining logic to adapt,
• And Airflow for automation,

you can build a fully responsive ML lifecycle that adapts over time.

GitHub Repo

You can find the complete project with data, detector, retrainer, and visualizations here: 👉 **github.com/rajkumar160798/model-drift-monitor**

What’s Next?

• Integrate the FastAPI endpoint to serve retrained predictions
• Add an Airflow DAG for true automation
• Monitor for model performance drift, not just data drift