Use feature parity, time-aware holdouts and delayed outcomes to decide what needs fixing before requesting another training run.
Updated Oct 2026 · Original practice questions with worked answers and teaching rubrics. Our editorial policy explains the sources, review process and limits.
A ticket-routing classifier is failing in production despite passing offline tests. The figures are fictional.
| Offline evaluation | 92% accuracy on 1,000 tickets. |
|---|---|
| Live audit | 160 of 200 tickets reached the correct queue. |
| Replay finding | 60 live events have different language features between training and serving. Forty have the wrong live route; ten remain wrong after feature parity is restored. |
| Unresolved evidence | The offline split method, label timing and performance on untouched post-fix traffic need verification. |
Your task. Distinguish competing causes, recommend the first fix and specify how to validate it without overstating the diagnostic replay.
Use feature parity, time-aware holdouts and delayed outcomes to decide what needs fixing before requesting another training run.
This reference answer is in Premium. Free signup opens other answers.
9 answers per topic instead of 5, and your progress kept · no cardor unlock all 420 remaining answersNo comments yet. How would you have answered?