statistics

Calculate the evidence needed for a small quality gain. Running both models on the same cases changes the test; the disagreement counts determine whether pairing saves evaluation work.

Audit a judge's 68% win rate with an order-swap table. Separate stable wins from unresolved verdicts, declare the denominator, and test whether the result supports the product decision.

Checking an A/B test early and stopping when it looks good inflates false positives. Run to the planned horizon unless the test was designed for early stopping, and stop early only for harm.