llm as judge

95% agreement sounds like a yes. On its own the number says almost nothing, and the thirty-second arithmetic that proves it is what separates a launch gate from a dashboard.

At this scale, judging every conversation with your best model costs more than some teams' whole inference bill. The answer is a sampling design with arithmetic behind every layer.

Audit a judge's 68% win rate with an order-swap table. Separate stable wins from unresolved verdicts, declare the denominator, and test whether the result supports the product decision.

A new judge is a new ruler. Run old and new judges in parallel on a human-labelled set, re-validate catch rates, mark the break in the trend line and pin judge versions from now on.