TL;DR: Turn each failure mode from your error analysis into a yes-or-no question with a one-line reason attached. Check with code whatever code can check, and use an LLM judge only for the rest. A pass rate per failure mode tells an engineer what to fix; an average score out of five does not.
The problem with scales
Rating outputs from 1 to 5 feels rigorous. It produces a number, the number has decimals, and it fits on a chart. In practice it fails in three ways.
It blends failures. A summary that invents a decision and a summary that is accurate but long might both get a 3. One is a trust problem and the other is a formatting preference. A falling average cannot tell you which happened.
The middle is undefined. Ask two careful reviewers to rate the same output and they will often disagree by a point. What separates a 3 from a 4 lives in each reviewer's head. A 0.2 change in the average may be pure noise from who did the rating that week.
It does not lead to action. "Quality dropped from 3.9 to 3.6" starts a meeting. "Invented action items went from 2% to 7% of summaries after Tuesday's prompt change" starts a fix.
What to do instead
For each failure mode you found in error analysis, write one question that a reviewer, a script or a model can answer yes or no.
| Failure mode | Binary question | Who checks |
|---|---|---|
| Invented facts | Is every claim in the summary supported by the transcript? | LLM judge, validated |
| Wrong language | Is the reply in the same language as the user's message? | Code (language ID) |
| Missing citation | Does every factual sentence have a citation link? | Code |
| Bad citation | Does each cited source support its sentence? | LLM judge, validated |
| Unnecessary question | Did it ask for information already in the conversation? | LLM judge, validated |
| Over length | Is it under 150 words? | Code |
Each judgment carries a short written reason: "fails: says the meeting agreed a March deadline; the transcript says April." The reason is what makes a label reviewable, lets you audit a judge, and gives engineers a concrete example to fix.
The headline numbers become a pass rate per failure mode, plus the share of outputs that pass every check. Both are easy to explain and both move when something specific changes.
Code first, model second
Many checks need no judgment at all. Is the output valid JSON? Does the cited URL exist? Is the amount in the reply the same as the amount in the order record? Did the agent call the refund tool without the approval flag? These are deterministic, cost nothing to run and never disagree with themselves. Run them on every output and every change.
LLM judges are for questions only judgment can answer: is this claim supported, is this tone appropriate, should the assistant have asked a question here. Each one is a separate judge prompt with one question, written from your labelled examples. A judge asked to score six criteria at once does all of them worse, and you cannot tell which criterion it got wrong.
When "partly right" comes up
The usual objection to binary judgments is that some outputs are partly right. That almost always means the question covers more than one thing. "Is this summary good?" is partly right. "Does it include every decision made in the meeting?" and "Is every action item assigned to the right person?" are each yes or no. If reviewers keep wanting a middle option, split the criterion.
The other place a scale can creep back in is taste: which of two rewrites sounds better, which image matches the brand. Binary is the wrong tool there, and absolute scores are worse. Use pairwise comparison: show two outputs side by side and ask which is better. Preference judgments are far more consistent than scores, and the result is a win rate with an interval.
Why binary labels help automation
The eventual goal for most failure modes is an automated judge that runs on every change. Validating a judge means comparing its labels with human labels and measuring how often it catches real failures and how often it falsely flags good outputs. That is straightforward with binary labels. With 1-to-5 scores, teams end up reporting "agreement within one point", which hides most disagreements and makes it nearly impossible to tell whether the judge can be trusted. Binary labels from the start save a migration later.
In the interview
This comes up whenever a candidate describes an eval. Saying "we'd have reviewers rate responses on a 1 to 5 scale for helpfulness and accuracy" is a common answer, and a Weak signal on most scorecards. The Strong Hire version names specific failure modes, turns each into a pass/fail question, separates code checks from judged checks, and mentions pairwise preference for comparisons that come down to taste. It shows the candidate has run an eval and had to act on its results.
Next: validating LLM judges · golden sets