Calculate the evidence needed for a small quality gain. Running both models on the same cases changes the test; the disagreement counts determine whether pairing saves evaluation work.
free accounts read more answers per topic · no card
← Evals & Launch Decisions / 09
hardNewAnthropicOpenAIGoogle DeepMind
Model B scores 83% and model A scores 80% on our eval. How many cases do we need before we believe B is better?
Try it in the practice workspaceDraft your answer before comparing it with the rubric.Calculate the evidence needed for a small quality gain. Running both models on the same cases changes the test; the disagreement counts determine whether pairing saves evaluation work.
Updated Nov 2026 · Original practice questions with worked answers and teaching rubrics. Our editorial policy explains the sources, review process and limits.
Follow-up roundsAnswer first, then revise for each changed constraint
- Round 2.
A pilot ran both models on 200 shared cases, and they disagreed on 9.
- Round 3.
Model B costs 40% less. Product will accept up to a 1-point quality loss for that saving.
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0
No comments yet. How would you have answered?