Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: "82%" is unreadable until you know the sample size, what the other 18% looks like, and which segments it averages over. Ship to the segments where failures are safe and recoverable and the agent beats today's baseline; keep the rest behind human review with a dated re-evaluation.
How to approach it
Refuse to say yes or no in the first sentence. Ask four things: how many cases produced the 82%, how those cases were sampled, what the agent did on the 18% it failed, and what happens today without the agent. Each answer can flip the decision, and the interviewer is watching whether you know that before you start building a framework on top of a number you have not inspected.
A strong answer
Start with the denominator. For independent cases sampled from the target traffic, a rough 95% interval is p ± 1.96 × √(p(1−p)/n). For reporting, use the Wilson interval shown below; the confidence-interval concept explains why it behaves better with small samples.
| Independent cases | 95% Wilson interval at 82% passing |
|---|---|
| 100 cases | about 73.3% to 88.3% |
| 200 cases | about 76.1% to 86.7% |
| 1,000 cases | about 79.5% to 84.3% |
Does the evidence clear your quality floor?
Keep the pass rate fixed and change the number of cases. Then raise the quality floor.
164 / 200 passed
76.1% to 86.7%
95% Wilson interval
● Observed rate ┊ Quality floor
The lower bound clears this floor
This checks one quality criterion. Failure severity, segment coverage and recovery still belong in the launch decision.
What this experiment assumes
These are hypothetical datasets at the chosen pass rate. Changing a control does not collect new evidence. The interval assumes independently sampled cases with reliable binary labels and a fixed sample size; repeated peeking needs a different statistical plan. Inputs stay in this page and are not sent or saved.
The floor is a target you choose, not an estimated baseline. Use a paired comparison when testing two systems on the same cases. Wilson interval method (NIST).
At 100 cases, sampling uncertainty alone leaves substantial doubt about clearing a 75% floor. A handpicked eval set can also over-represent easy intents. A confidence interval does not account for that selection bias; collect representative cases before treating the score as a production estimate.
Two more assumptions hide inside that table. The first is independence. If the 200 cases came from 40 customers or a dozen intents, they are correlated, and the honest interval is wider; Anthropic's guidance on eval statistics (November 2024) recommends clustering standard errors on the unit that was sampled. The second is consistency. One pass does not mean the agent passes every time it sees that task. On τ-bench (Yao et al., 2024), GPT-4o passed 61.2% of retail tasks and 35.2% of airline tasks in a single run, and on retail the share it passed on all eight repeated trials fell below 25%. For an agent that touches customers' money, report pass^k, the share of tasks passed on every one of k runs, next to the single-run rate. Anthropic's agent eval guide (January 2026) draws the same line: pass@k where one success is enough, pass^k where consistency matters.
Then open the failures. The 18% is not one thing. Sort it into three buckets:
- Safe failures. The agent recognised it could not help and handed off with context, or asked a clarifying question. These cost a human minute, not a customer.
- Visible failures. A wrong answer the user is likely to notice and correct.
- Silent failures. A confident wrong action the user accepts: a refund issued to the wrong order, a cancelled plan that should have been paused.
Weight each by what it costs. Expected cost per task is the sum of each failure rate times its cost. An illustrative billing queue of 1,000 tasks: 820 pass, 150 hand off safely at about $3 of human time each, and 30 issue a wrong refund that costs about $80 to find and reverse. The failures cost 150 × $3 + 30 × $80 = $450 + $2,400 = $2,850, or $2.85 per task, and the 3% of silent failures carry 84% of it. Halving the handoffs saves $225; halving the silent failures saves $1,200. Two agents with identical 82% scores can have opposite launch decisions.
Next, segment. In a hypothetical support queue, 95% on order-status questions could conceal 40% on disputes. Those segments need different launch decisions. A vendor's aggregate automation rate cannot establish quality for your difficult cases; inspect outcomes and re-contact for each intent.
Compare against the real baseline, not against 100%. If human agents resolve order-status tickets correctly 97% of the time with a four-hour wait, an agent at 95% in twenty seconds may be the better product. If humans resolve disputes at 90% and the agent manages 40%, routing disputes to the agent is a regression however good the average looks.
The recommendation I would give:
Launch autonomously only where the segment clears its baseline with the interval taken into account and the worst failure is reversible. Everything else ships as an assistant behind human approval, which still saves handle time and generates the labelled traces you need to promote the next segment. That path is not a consolation prize: a field study of 5,179 customer support agents found AI assistance raised issues resolved per hour by 14% on average, and by 34% for novice and low-skilled workers (Brynjolfsson, Li and Raymond, NBER working paper, revised November 2023; published in the Quarterly Journal of Economics in 2025). Add a kill switch, an alert on silent-failure proxies (re-contact within 48 hours, refunds reversed), and a dated re-evaluation once a thousand live traces exist.
I would hold the affected actions if their upper confidence bound for harmful errors exceeds a risk limit agreed with operations and the accountable business owner. An irreversible billing error needs a stricter limit than an unnecessary handoff. If the system cannot reliably separate those actions, hold the autonomous launch.
What interviewers probe next
"Leadership wants one number for the launch review. What do you give them?" Cost-weighted expected error per thousand tasks against the current process, by segment, with the interval. One number is fine if it carries severity.
"How would you get to 1,000 labelled cases quickly?" Shadow mode on live traffic: the agent drafts, humans act, and every human decision becomes a label. At 100 reviewed traces per working day, 1,000 labels takes ten days; check reviewer agreement before using human actions as ground truth.
"The eval says 82% but users rate it 3.1 out of 5. Which do you trust?" Neither alone. The gap usually means the eval tests correctness and users are reacting to tone, latency or unnecessary handoffs. Read fifty low-rated traces before deciding.
"The agent passed this task on Monday and failed it on Tuesday. Which run counts?" Both. Run each case several times and report pass^k for actions that move money; a task the agent gets right three times in four is not ready to run unattended.
"What if a competitor already launched at a lower bar?" Their launch tells you about their risk appetite, not your users' tolerance. Launch the safe segments fast so you are not standing still.
Round 2
"Operations can review 300 agent actions a day. Daily volume is 2,400 order-status questions, 1,200 refunds and 400 disputes."
Draft-and-approve for everything outside order status would need 1,600 reviews a day, more than five times capacity, so the plan has to change. Disputes stay with human agents and use no review capacity. Order status runs autonomously with a 1% sample, 24 reviews a day. That leaves 276 slots, too few to approve 1,200 refund drafts, and refunds are where the $80 silent failures live: at the illustrative 3% rate, autonomy would cost about $2,880 a day in reversals. So refunds stay with people for now, and the 276 slots go to shadow-labelling refund drafts. That produces 1,000 labels in four working days, the evidence the refund decision actually needs.
Round 3
"Shadow mode is done: 12 of 1,000 refund drafts would have refunded the wrong order. Operations set a limit of 1% wrong refunds."
The point estimate is 1.2%, and the 95% Wilson interval runs from about 0.7% to 2.1%. The agent misses the limit, and the upper bound is the figure the agreed risk limit applies to. In money, 1.2% of 1,200 refunds is about 14 wrong refunds and $1,152 a day in reversals, against $3,600 of human handling, so the cost case favours the agent. The risk limit still decides, because operations owns it. The design change that could pass it is a hold before payout: refunds wait two hours, reviewers check the drafts the agent marks least certain, and a wrong refund caught before settlement costs minutes instead of $80. I would re-measure the wrong-refund rate under that design on the next 1,000 drafts before any refund runs unattended.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Treats 82% as a number | Says yes or no to the number | Asks about sample size and failure types | Computes the interval, questions independence and run-to-run consistency, and shows how the decision changes with n |
| Failure analysis | "Improve the prompt" | Separates failure types | Weights failures by cost and reversibility |
| Launch plan | All or nothing | Phased rollout | Segment-level autonomy with baselines, monitors, a kill switch and a re-evaluation date |
Common mistakes
Answering "82% is good enough for a v1" without asking what the 18% did. The interviewer hears that you would ship a refund bug to save a meeting.
Proposing an LLM judge or a new eval framework before reading the failing traces. Error analysis comes first; tooling is what you build once you know the failure modes.
Comparing the agent to perfection instead of to today's process, which makes every AI launch look unshippable and every rejection look principled.
Treating "human in the loop" as a binary. The useful design question is which actions need approval, which need an undo window, and which can run unattended.