Your model didn't degrade, the world did
Model weights are frozen the day you deploy. Everything they were fitted to keeps moving, and nothing tells you unless you look.
A deployed model is a frozen artefact. Its accuracy at deployment is a fact about the data distribution at that moment, and that distribution does not hold still.
Nothing about the system reports this. Latency looks fine, error rates look fine, and the predictions get quietly worse.
Three things that drift
Input drift is the distribution of features changing. A new customer segment, a changed upstream format, a seasonal shift. The model is now extrapolating.
Concept drift is the relationship between features and outcome changing. Fraud patterns adapt, user behaviour shifts, and the mapping the model learned no longer describes reality even when inputs look the same.
Label delay is the practical problem underneath both: you often find out the truth weeks later, so accuracy is not available in real time even in principle.
What to monitor when you cannot monitor accuracy
Since ground truth is delayed, monitor proxies.
Input distributions per feature, compared against the training set. A statistical distance metric per feature, alerting on significant shifts, catches upstream changes early and is cheap.
Prediction distribution. If a model that historically flags 3% of cases suddenly flags 11%, something changed even before you know which answer was right.
Confidence distribution. A model becoming systematically less certain is telling you it is seeing unfamiliar inputs.
Null and default rates in the feature pipeline. A surprising share of drift traces to a broken join or a renamed upstream field rather than to anything about the world.
You will usually detect a broken pipeline before you detect genuine drift, and that is fine — it is the more common failure.
The registry is what makes this actionable
Detecting drift is useless if you cannot answer which model version produced a prediction.
A model registry that records version, training data, parameters and metrics — plus logging the version alongside every prediction — is what turns "results got worse" into "results got worse after version 7, which was trained on data through March".
Without it, every investigation starts from scratch and rollback is guesswork.
Retraining on a trigger, not a calendar
Scheduled retraining is common and crude: too frequent wastes compute and churns behaviour, too infrequent leaves a degraded model in place.
Triggering on measured drift is better, and the discipline is to validate the retrained model against a held-out set before promoting it. Automatic retraining that automatically deploys will eventually train on a broken pipeline and ship the result.
A human approving the promotion, with the metrics in front of them, is a reasonable amount of friction for something that changes what your product does.