Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: An evaluation measures behavior against defined criteria on chosen tasks. Software tests verify contracts and outcomes at different levels; they can also be part of an eval. For a probabilistic feature, report the task sample, repeated trials, failure severity and uncertainty rather than treating a green suite or one pass rate as launch approval.
How to approach it
Use one feature to distinguish the questions each check answers. Explain what is deterministic around the model and what varies across tasks or runs. Ask which failures are unacceptable even once before quoting an average.
A strong answer
Take a fictional assistant that summarizes unread messages. A unit test verifies that an unauthorized message is excluded by the source filter. An integration test verifies how the system handles a source timeout. A behavior evaluation checks whether generated summaries preserve owners and deadlines, including ambiguous wording. An end-to-end study checks whether users find their required actions without missing commitments.
The reason the distinction exists is stated plainly in OpenAI's evaluation best practices (checked November 2026): models "sometimes produce different output from the same input, which makes traditional software testing methods insufficient." An eval therefore has more moving parts than a test. Anthropic's agent eval guide (January 2026) names them: the task, repeated trials of it, a grader, the transcript of what happened and the outcome. The outcome is checked in the world, not in the model's words: for a booking agent, whether a reservation actually exists in the database, not whether the agent said it made one.
These checks overlap. A deterministic program can grade whether a model copied an ID correctly; a human can judge whether a software feature is useful. An eval need not produce only a pass rate, and ordinary software tests can be flaky or statistical. The distinction that helps the PM is what each piece of evidence establishes.
| Check | What a pass supports | What it does not establish |
|---|---|---|
| JSON schema validation | Fields have permitted structure | Values are true |
| Permission boundary test | The tested access rule was enforced | Every untested access path is safe |
| Graded summary set | Observed performance on these tasks and criteria | Equivalent quality on unseen traffic |
| Repeated trials | Stability on the repeated tasks | More independent user situations |
| User experiment | A measured effect under its design | Benefit for every segment or future version |
Suppose 91 of 100 independently sampled summaries pass an agreed factuality check. An approximate 95% Wilson interval is 83.8% to 95.2%, assuming the sampling design supports that calculation. The nine failures still need inspection. One exposed private message should block the affected route even if the aggregate score exceeds a quality threshold. A missing greeting has a different consequence.
Try the same independent-task example below. The chosen 90% floor is illustrative and does not replace the separate critical-failure rule.
Does the evidence clear your quality floor?
Keep the pass rate fixed and change the number of cases. Then raise the quality floor.
91 / 100 passed
83.8% to 95.2%
95% Wilson interval
● Observed rate ┊ Quality floor
The interval crosses this floor
This checks one quality criterion. Failure severity, segment coverage and recovery still belong in the launch decision.
What this experiment assumes
These are hypothetical datasets at the chosen pass rate. Changing a control does not collect new evidence. The interval assumes independently sampled cases with reliable binary labels and a fixed sample size; repeated peeking needs a different statistical plan. Inputs stay in this page and are not sent or saved.
The floor is a target you choose, not an estimated baseline. Use a paired comparison when testing two systems on the same cases. Wilson interval method (NIST).
Keep two kinds of eval apart. A regression eval guards what already works and should pass at close to 100%; a drop is a bug. A capability eval measures what the system cannot yet do reliably and should start with a low pass rate; a rise is progress. Anthropic's guide makes the same split. Mixing them produces a single score that falls when you add harder tasks and rises when you remove them.
Use representative tasks to estimate ordinary performance and a separately labelled diagnostic suite for known failures, attacks and rare hazards. Replaying three times on each of 100 tasks helps reveal variability; it does not create 300 independently sampled tasks. Record the task and trial IDs, keep the grader version fixed and compare candidate systems on paired tasks where appropriate.
Release criteria combine several kinds of evidence. The PM and domain experts specify acceptable behavior and severity; engineering owns tested enforcement and execution; the release owner resolves the result against criteria agreed before the run. Risk and safety owners participate where the use warrants it. No single job title can supply all those judgments.
What interviewers probe next
“Who writes the criteria?” Product and domain experts define the outcome and consequences with engineering, then check that graders apply those definitions consistently.
“How many cases are enough?” Choose from the effect size, failure prevalence and sampling design. A narrow interval on a biased set still measures the wrong population.
“Can the agent just tell us it succeeded?” No. Grade the outcome in the system of record. An agent's own report is part of the transcript, and it is the part most likely to be optimistic.
“Does one failed output stop release?” It can. Predefine critical failures separately from tolerable variation; inspect severity instead of averaging it away.
How it's scored
Weak: Says evals replace tests or that any green score proves readiness.
Hire: Separates contract checks from behavioral evidence and reports sample size and criteria.
Strong Hire: Handles repeated trials, diagnostic sampling and critical-failure gates without treating one aggregate as the decision; separates regression from capability evals and grades outcomes in the system of record.
Common mistakes
Checking syntax while calling it factual accuracy. Counting repeated outputs as independent tasks. Assuming every software failure is reproducible. Allowing a rare permission breach because the average looks good.
Build the AI testing pyramid, define binary judgments, and experiment with pass-rate uncertainty.