Two search rankers look equally good offline. Would you use interleaving or an A/B test to choose one?
Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.
Case briefRead the inputs, then make your call
An internal document-search product has two rankers with similar offline scores. Both return permission-checked document links. The pilot figures are illustrative.
| Pilot | 200 sessions: 80 prefer B, 60 prefer A and 60 tie under a prespecified credit rule. |
|---|---|
| Assumption to examine | The arithmetic exercise treats sessions as independent. Establish whether repeated users or queries violate this in practice. |
| Product decision | Leadership wants to know whether to replace the ranker and whether search improves task completion. |
Your task. Choose an experiment sequence, interpret the pilot and explain what the result can and cannot establish.
0 words