The model was fine. The features were different.
A model that scores well offline and poorly in production usually isn't seeing the same inputs it was trained on.
Offline evaluation looks strong. The model ships. Production performance is noticeably worse and nothing in the training process explains it.
The usual cause is not the model. It is that the features computed at serving time are not the features it was trained on.
How the gap opens
Training features are computed in batch, often in Python over a dataframe, with the full dataset available. Serving features are computed per request, often in a different language, with only the current record to hand.
Two implementations of the same concept, written by different people at different times. They diverge in small ways: a different null default, a different rounding, a different time window, a category encoded in a different order.
Each difference is individually trivial. Collectively they mean the model is being asked about a slightly different world.
Leakage is the other half
The mirror image: training features that could not exist at prediction time.
An aggregate computed over the whole dataset including the future. A field populated after the outcome is known. A join that quietly brings in a column updated later.
This inflates offline metrics and cannot be reproduced in production, which is why the gap looks like drift or degradation rather than a bug in the pipeline.
The tell is an offline result that seems too good. It usually is.
If offline accuracy is much better than production, suspect the features before the model.
What actually closes it
One implementation, used by both paths. The strongest version of this is a feature store — compute once, serve to training and inference — but the principle applies without one: shared code, called from both places, is most of the benefit.
Point-in-time correctness in training. Every feature value must be one that was *available* at the moment of prediction, which usually means joining on both entity and timestamp rather than entity alone. This is fiddly and it is the difference between an honest evaluation and a flattering one.
Log the features you actually served. Not just the prediction — the input vector. Then you can compare production feature distributions against training ones directly, and the question stops being a debate.
The check worth running before launch
Take a sample of records. Compute their features through the training pipeline and through the serving pipeline. Compare, field by field.
They should be identical. When they are not, you have found the gap before your users did, and it is nearly always a smaller fix than the investigation it saves.