Concept Drift in Production: Detecting and Responding to Shifting Data Distributions

A model that performs well at deployment may silently degrade months later because the world it was trained on no longer matches the world it is operating in. This is concept drift — the phenomenon where the statistical properties of the input data or the relationship between inputs and outputs changes over time. It is one of the most common causes of "mysterious" performance degradation in production ML systems.

Types of Drift

Data drift (covariate shift): the distribution of input features changes, but the relationship between features and labels remains the same. A fraud detection model trained on 2022 transaction patterns may see different transaction amounts, merchant categories, and device types in 2024 — the features have drifted even if the definition of fraud has not.

Concept drift (label drift): the relationship between features and the label changes. A content moderation model trained before a slang term was coined to mean something harmful will misclassify posts using that term — the concept of what constitutes harmful content has drifted.

Label drift: the distribution of outcomes changes. If a previously rare event becomes more common (or vice versa), a model calibrated for the old class distribution will be miscalibrated for the new one.

Detecting Drift

Statistical Tests on Feature Distributions

For each input feature, compare the distribution of recent production data to the training distribution using statistical tests:

• Kolmogorov-Smirnov test: for continuous features; tests whether two samples come from the same distribution
• Chi-squared test: for categorical features; tests independence of distributions
• Population Stability Index (PSI): industry standard from credit risk; PSI < 0.1 indicates no drift, 0.1-0.2 indicates moderate drift, > 0.2 indicates significant drift

Run these tests on a rolling window of recent data (e.g., the past 7 days) vs. a reference window (the training data distribution or a stable historical baseline).

Performance-Based Detection

The most direct approach: track model performance metrics on recent labelled data. If ground truth labels are available with a short lag (e.g., you can observe outcomes within 7 days of prediction), monitor AUC-ROC, precision, recall, or RMSE on rolling windows and alert on significant drops.

The challenge: in many applications, ground truth labels arrive with significant delay (loan default takes months to observe; long-term churn takes years). In these cases, performance-based detection is not fast enough for operational purposes.

Prediction Distribution Monitoring

Monitor the distribution of the model's predictions over time. If the model was trained on data where 5% of examples were labelled positive, sudden shifts to 15% positive predictions signal that something has changed. This does not require labels and can detect drift immediately.

Responding to Drift

Do nothing (for now): if drift is detected but performance has not yet degraded below an acceptable threshold, document it and monitor more closely. Not all drift requires immediate retraining.

Manual retraining: retrain the model on a more recent training window. This is appropriate for gradual drift and for models that are retrained on a scheduled basis (monthly, quarterly).

Online learning: update the model incrementally as new labeled examples arrive. Appropriate for applications with fast, continuous label feedback.

Automated retraining triggers: implement a drift metric threshold that automatically triggers a retraining pipeline. MLflow, Kubeflow Pipelines, and similar MLOps tools support triggered retraining workflows.

Model retirement: if the relationship between features and labels has changed fundamentally (concept drift in the true sense), retraining on recent data may not be enough. You may need to redesign the feature set or the modelling approach.

MLOps for Drift Management

Operationalising drift detection requires:

• A feature store or data pipeline that can replay historical feature distributions
• A model registry that tracks which training data version corresponds to which model version
• A monitoring dashboard (Evidently, WhyLabs, Arize, or custom-built) that shows feature and prediction distribution statistics over time
• Alert rules that notify on-call teams when drift metrics exceed thresholds

Conclusion

Concept drift is not an edge case — it is the normal behaviour of production ML systems over time. The question is not whether drift will occur but whether you will detect it before it causes a business problem. Statistical monitoring, prediction distribution tracking, and automated retraining pipelines are the operational foundations of ML systems that remain reliable after deployment.

Keywords: concept drift, data drift, production ML, model monitoring, covariate shift, distribution shift, MLOps, retraining, Evidently, feature drift detection