Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Not yet. Agreement is dominated by the easy majority class: if 5% of outputs are bad, a judge that passes everything agrees 95% of the time and catches nothing. Measure how often the judge catches real failures and how often it falsely flags good outputs, on held-out human labels, before letting it gate anything.
How to approach it
Ask two questions before answering: what share of outputs do humans label as failures, and were the 95% measured on the same examples used to write the judge prompt? The first tells you whether agreement can be trusted at all. The second tells you whether it is overfitted.
A strong answer
Start with the trap. Suppose humans labelled 1,000 outputs and 50 of them failed.
| Judge | Failures caught (of 50) | Good outputs wrongly flagged (of 950) | Agreement |
|---|---|---|---|
| Always says pass | 0 | 0 | 95.0% |
| A real judge | 30 | 20 | 96.0% |
| A strict judge | 45 | 80 | 91.5% |
The judge that never catches anything matches the headline number. The strict judge has the lowest agreement and is the most useful as a launch gate, because it finds 90% of the failures. Agreement rewards being right about the common case, which is the case you least need help with.
So the two numbers I would ask for are:
- Catch rate on failures (the true positive rate, treating "fail" as the positive class): of the outputs humans marked bad, what share did the judge flag?
- Pass rate on good outputs (true negative rate): of the outputs humans marked good, what share did the judge pass?
For a launch gate on a safety or policy failure, I care most about the first. Missing a real failure is the expensive error. A false flag costs a human a minute of review.
Then check the method. Proper judge validation splits the human labels the way you would split training data: a small set to write and tune the judge prompt, a development set to iterate on, and a test set nobody looks at until the end. If the 95% came from the examples used to write the prompt, it is an optimistic number.
Two published findings explain why agreement alone misleads. A 2024 study of judges on a question-answering benchmark found that "judges with high percent agreement can still assign vastly different scores" and that judges tended toward leniency (Thakur et al.). And the ceiling is lower than people assume: the original MT-Bench work found strong judges reach "over 80% agreement, the same level of agreement between humans" (Zheng et al., 2023), and named position, verbosity and self-enhancement biases. So also measure how often two human reviewers agree on your labels. If they agree 85% of the time, a judge claiming 95% has probably learned the easy cases, not the judgment.
There is one more use of the two rates. A judge that is imperfect but well measured can still give you an honest production estimate, using a correction epidemiologists have applied to imperfect diagnostic tests since the 1970s (the Rogan–Gladen estimator). If the judge flags 8% of live outputs, with a catch rate of 60% and a false-flag rate of about 2.1% (20 of 950), the corrected failure rate is:
(0.08 − 0.021) / (0.60 − 0.021) ≈ 10.2%
The real failure rate is higher than the raw flag rate because the judge misses 40% of failures. Leadership seeing "8% flagged" would under-estimate the problem. The corrected figure inherits the uncertainty of both rates, so report it with an interval; prediction-powered inference (Angelopoulos et al., Science, 2023) is one way to get a valid one from a small human-labelled sample plus many judged outputs.
My recommendation: use the judge for monitoring now, with its rates disclosed next to every number it produces. Make it a launch gate only when the lower end of its catch-rate interval clears the bar, not when the point estimate does. The sample size decides that:
| Failures in the held-out set | Judge catches | 95% Wilson interval on the catch rate |
|---|---|---|
| 50 | 40 (80%) | 67.0% to 88.8% |
| 50 | 45 (90%) | 78.6% to 95.7% |
| 100 | 90 (90%) | 82.6% to 94.5% |
| 200 | 180 (90%) | 85.1% to 93.4% |
With 50 failures, even a 90% observed catch rate cannot rule out a judge that catches fewer than four in five. To show at least 80% with confidence you need roughly 100 failures in the test set, which is why it should be enriched with known failures rather than sampled at the production rate. If the judge cannot get there, split the failure definition into narrower pass/fail checks, some of which code can do.
What interviewers probe next
"Labelling is expensive. How many human labels do you need?" A few hundred, deliberately enriched so failures are 20 to 30% of the set. Otherwise you spend most of the budget labelling easy passes.
"Should the judge be the same model as the product?" Prefer a different or stronger model, and test for self-preference: one study found an LLM evaluator "scores its own outputs higher than others'" when human annotators rated them equal (Panickssery et al., 2024). The real protection is still validation against humans, not the model choice.
"The judge and the domain experts disagree. Who wins?" The experts, until you read the disagreements. Sometimes the rubric is ambiguous and both are reasonable, which means the criterion needs splitting.
Round 2
"Legal wants the judge to gate a medical-advice policy, where a missed failure can harm someone. They ask for a catch rate of at least 95%."
The bar applies to the lower end of the interval, so the observed rate and the sample size decide together. A judge that catches 98% of failures needs about 210 held-out failures before its Wilson lower bound clears 95%; at an observed 97% it needs about 440, and at 96% about 1,760. The real judge here catches 60%, so no sample rescues it. I would not offer a single judge as this gate. I would split the policy into narrow checks (dosage figures, named drugs, emergency symptoms), several of them plain rules, send anything any check flags to a clinician, and measure the combined catch rate on an enriched held-out set. Legal's 95% then applies to the whole pipeline, reported with its interval.
Round 3
"Next month the product moves to a new model. The judge prompt stays the same."
The judge's catch rate and false-flag rate were measured on the old model's outputs. A new model fails in different ways, so both rates are unknown until re-measured, and every corrected failure estimate built on them stops being valid. Before the switch I would label a fresh held-out set from the new model's outputs with about 100 failures, as in the main answer. At a 5% natural failure rate that means about 2,000 random outputs, so I would enrich: label every judge-flagged output plus a random sample of unflagged ones, then weight back to production. Until the new rates exist, the dashboard shows raw flag rates marked uncorrected, and the judge gates nothing.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Reading 95% | Accepts it | Asks about class balance | Shows that a pass-everything judge scores 95% at a 5% failure rate |
| Metrics | Accuracy or agreement only | Precision and recall | Catch rate on failures and pass rate on good outputs, with held-out labels |
| Use | Judge decides launch | Judge plus spot checks | Corrects production estimates using the judge's measured rates and gates on the lower bound of the catch-rate interval |
Common mistakes
Validating the judge on the same examples used to write its prompt.
Using a 1 to 5 scale and reporting average agreement within one point. Adjacent-score agreement hides almost everything.
Asking the judge to score several criteria at once. Each failure mode deserves its own pass/fail judge and its own validation.
Background: validating LLM judges.