Compare ranking preferences efficiently without mistaking a click preference for a retention or revenue result.
Updated Oct 2026 · Original practice questions with worked answers and teaching rubrics. Our editorial policy explains the sources, review process and limits.
An internal document-search product has two rankers with similar offline scores. Both return permission-checked document links. The pilot figures are illustrative.
| Pilot | 200 sessions: 80 prefer B, 60 prefer A and 60 tie under a prespecified credit rule. |
|---|---|
| Assumption to examine | The arithmetic exercise treats sessions as independent. Establish whether repeated users or queries violate this in practice. |
| Product decision | Leadership wants to know whether to replace the ranker and whether search improves task completion. |
Your task. Choose an experiment sequence, interpret the pilot and explain what the result can and cannot establish.
Compare ranking preferences efficiently without mistaking a click preference for a retention or revenue result.
This reference answer is in Premium. Free signup opens other answers.
9 answers per topic instead of 5, and your progress kept · no cardor unlock all 420 remaining answersNo comments yet. How would you have answered?