Free account
Read this concept with a free account
Read this concept at no cost by signing in with Google.
no card needed
Core
Confidence Intervals for Eval Pass Rates: The Arithmetic Every AI PM Needs
A pass rate from an eval is an estimate. This guide gives the formula for its 95% interval, a table of widths by sample size, when to use the Wilson interval and the rule of three, how many cases you need to detect a change, and when paired comparisons of two models can reduce the sample requirement.
CONCEPT / EVALS
RELATED CONCEPTS
PRACTISE THIS CONCEPT
Evals & Launch DecisionsModel B scores 83% and model A scores 80% on our eval. How many cases do we need before we believe B is better?→Evals & Launch DecisionsYour support agent completes 82% of tasks in testing. Is it ready to launch?→AI Product SenseHow would you design an AI-first spreadsheet?→Live Prototyping & AI-Native WorkflowIn a live prototyping round, you get an AI coding tool and 40 minutes to build something. How do you approach it?→Evals & Launch DecisionsHow would you evaluate a RAG assistant that answers questions from our help center?→Live Prototyping & AI-Native WorkflowLive round: build a tool that turns a meeting transcript into action items with owners and due dates. Show me how you'd do it in 40 minutes.→