TL;DR: Ask what was attempted, what counted as success and which runs disappeared before accepting a pass rate. An agent that succeeds at least once on every task can still fail many customer attempts. Choose the measure that matches the product promise.
One task, several attempts
A task is a defined request in a starting environment. A trial is one attempt at that task. The transcript records actions and responses; the outcome is what actually happened in the system. A confident “done” message does not prove that a record changed.
An agent harness supplies the operating loop, tools, context and controls. An evaluation harness runs the test conditions and records or grades results. Both can change the score. A different retry policy, tool timeout or grader can produce a different result with the same underlying model.
Anthropic's agent evaluation article, published January 2026 and reviewed November 2026, uses these distinctions. Inspect's task documentation, reviewed November 2026, makes the dataset, solver and scorer explicit components. Neither removes the PM's responsibility to decide what success means for a customer.
Read the rows before the headline
This original teaching fixture contains ten tasks with three trials each. Each trial starts from a reset environment with the same configuration. P means the requested outcome and required constraints passed; F means a scored failure. There are no missing or infrastructure-error trials here.
| Task | Trial 1 | Trial 2 | Trial 3 |
|---|---|---|---|
| A | P | P | P |
| B | P | P | P |
| C | P | P | P |
| D | P | P | P |
| E | P | F | P |
| F | F | P | P |
| G | P | P | F |
| H | F | F | P |
| I | F | P | F |
| J | F | F | F |
The same table produces four answers:
- First attempt: 6/10 = 60%. This describes the observed first-trial result on these tasks.
- All trial outcomes pooled: 20/30 = 66.7%. This describes the attempts, with equal trials per task.
- At least one success across three trials: 9/10 = 90%. This is the observed pass@3 for this fixture.
- Success on all three trials: 4/10 = 40%. This is the observed pass^3, or all-three consistency.
Read diagram description
Original ten-task fixture: first attempt succeeds on six tasks, at least one of three succeeds on nine, and all three succeed on four. Different measures describe different product promises.
A benchmark may estimate pass@k from more than k samples rather than count one group of k directly. Check its estimator and sampling rules. Here, all ten tasks have exactly three observations, so the fractions can be read from the rows.
For one task with independent, identical success probability p, at-least-one success in k attempts is 1 − (1 − p)^k; all-k success is p^k. Real tasks have different difficulty and shared failure causes. Raising the pooled 66.7% to a power does not reproduce this table's task-level result. Nor are these thirty attempts thirty independently sampled customer tasks.
Match the measure to the decision
For a writing assistant, several drafts may be useful if a person can cheaply recognize the good one. Report the time spent choosing and editing as well as eventual success. For an agent that changes a customer record, a failed first attempt may already have caused damage. Three retries are not three free chances.
A 90% pass@3 result also assumes access to a judge that identifies success. If the product cannot tell which attempt is correct, it cannot simply deliver the benchmark's best attempt. Evaluate the actual selection policy, its errors and the cost of every run.
I would keep the record-changing agent in supervised use on this evidence. Four consistently passing tasks out of ten do not justify an unattended-product claim, and ten tasks are too few to characterize a broad workload. Inspect failures by task family and severity, then sample more representative tasks before estimating production performance.
The denominator is part of the product claim
Keep a run ledger with scheduled trials, scored successes, scored failures, infrastructure errors, missing runs and unscored runs. Record retries separately. An evaluation service outage may prevent a valid model-capability measurement; a timeout in the customer workflow can still be a product failure. State which claim each denominator answers.
As reviewed in November 2026, the tau-bench running guide documents separate infrastructure-error reporting and exclusion from its scored denominator. It also distinguishes partial observed results from complete benchmark frames. That convention is useful to inspect, but it is not permission to erase failed customer work from your launch report.
If errors cluster on long tasks or a particular language, the remaining score can describe an easier workload. Show coverage by segment and investigate the missing outcomes before comparing releases. An aggregate score with no execution ledger is incomplete evidence.
Preserve the comparison
Record task IDs and starting state, model and prompt versions, tools and permissions, budgets, retry/selection policy, and grader version. Keep the task reference answers away from the agent. Make a deliberate change and rerun a common workload. If the rubric changes, rescore both stored result sets where the necessary evidence exists; rerun when a transcript cannot establish the required outcome.
The benchmark audit case gives you a score that improves while the report loses evidence. Compare your call with the Premium reference after attempting it. For the state-changing side, try what happens after a user presses Stop.