DECISION TOOL · FREE TO USE
Eval confidence calculator
The interval on your pass rate, and the cases you need to tell two models apart.
± 1.96 × √(p(1 − p) ÷ n)01 / ESTIMATE UNCERTAINTY
How much does this score tell you?
Plan for approximately 630 cases for a ±3-point margin. That is 430 more than your current set.
The size estimate uses a normal approximation. The displayed interval uses the Wilson method. Cases must be independent and representative of the traffic you want to measure.
02 / COMPARE MODELS
Could you detect the difference?
The candidate is 3 percentage points higher. Your 200-case set is below this estimate of 1,638 cases per model.
This estimate treats the two samples as independent. If both models answer the same cases, use a paired analysis based on their disagreements. A detectable difference still needs a product decision: does the gain justify the cost and latency?
FROM THE RESULT TO THE DECISION
How to reason about it
An eval score without a sample size is unreadable. Eighty-two percent on 100 cases is consistent with anything from the mid-seventies to about ninety; on 1,000 cases the range is under five points wide.
Use the left panel to report the number you have honestly, and the right one before a model migration, to find out whether your golden set can even detect the improvement you are paying for.
If your eval set cannot detect the difference you are deciding on, do not make the decision on the aggregate. Grow the set, compare models on the same cases, or decide on the segments that matter.
Questions about this tool
It depends on the precision you need. For a rate near 80%, about 250 cases give roughly ±5 points and about 630 give ±3. Comparing two models that differ by three points takes over 1,500 cases each.