Why evals became a PM job
An evaluation is the product's definition of done, written as data. Someone has to decide which examples count, what a pass looks like and which number is good enough to ship. Those are product decisions. Practising them produces evidence you can discuss: the failures you found, the rubric you revised and the scope you chose to release.
Start with error analysis
Before you write any metric, read real outputs. Take 50 to 100 recent cases, note in plain words what went wrong in each, then group the notes. The groups become your failure categories, and they are seldom the ones the team guessed in advance.
A metric invented before this step measures what was easy to imagine. The step after it measures what is actually breaking.
Build a golden set
A golden set is a fixed collection of real inputs, each with what a good output must contain. Weight it toward the cases that matter: hard ones, risky ones, and segments that are easy to neglect, such as queries in languages other than English. Version it, and add every production failure you fix so the same failure cannot come back unnoticed.
Prefer yes-or-no pass rules
A request to rate helpfulness from one to five produces numbers that drift with the grader's mood. Asking whether the reply states the correct refund amount does not. Split fuzzy qualities into several yes-or-no checks a person can apply consistently, and keep side-by-side preference comparisons for overall taste.
Check the judge before trusting it
Using a model to grade outputs makes evaluation scale, but the judge is a model with its own errors. Label a sample by hand, run the judge on the same sample and report how often the two agree, especially on failures. A judge that passes everything looks accurate on a set that is mostly passes.
Know what your sample can tell you
Small sets are good for finding failures and poor for proving improvements. If a prompt change moves your pass rate from 82% to 84% on 200 cases, you have shown nothing yet. Grow the set for that decision, or compare the same cases pair by pair.
| Examples | Pass rate | Margin either way |
|---|---|---|
| 100 | 82% | about 7.5 points |
| 200 | 82% | about 5.3 points |
| 1,000 | 82% | about 2.4 points |
Turn results into a launch gate
A useful gate has four parts: an overall pass rate, a floor no important segment may fall below, failure types that block launch whatever the average (any invented legal citation, for example), and a ceiling on cost per successful task.
Write the gate down before you see results. A bar set afterwards has a way of landing just below whatever the team achieved.
Keep evaluating after launch
Production traffic drifts away from any golden set. Sample live outputs every week for review, track the online signals that have predicted quality before, and rerun the full set on every prompt change and every provider model update.
Sources and further reading
Questions about this topic
A repeatable test of model output on a fixed set of real examples, with pass rules that encode what good means for the product. It is how a team decides whether a change is safe to ship.