A model passes offline tests but fails in production. Is this overfitting, training-serving skew or drift?
Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.
Case briefRead the inputs, then make your call
A ticket-routing classifier is failing in production despite passing offline tests. The figures are fictional.
| Offline evaluation | 92% accuracy on 1,000 tickets. |
|---|---|
| Live audit | 160 of 200 tickets reached the correct queue. |
| Replay finding | 60 live events have different language features between training and serving. Forty have the wrong live route; ten remain wrong after feature parity is restored. |
| Unresolved evidence | The offline split method, label timing and performance on untouched post-fix traffic need verification. |
Your task. Distinguish competing causes, recommend the first fix and specify how to validate it without overstating the diagnostic replay.
0 words