Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Step reliability compounds. A ten-step task where each step succeeds 95% of the time completes about 60% of the time (0.95^10 ≈ 0.60). Fix it by cutting the number of model-driven steps, making the steps that can be deterministic actually deterministic, and adding verification with a retry where failures can be detected. Together those can take the same task above 90%.
How to approach it
Ask how many steps a typical task involves and how the 95% was measured: per step in isolation, or across real tasks? Then show the arithmetic, because it explains the user complaints in one line. The rest of the answer is about which levers change the exponent and which change the base.
A strong answer
The arithmetic: if a task needs k steps to all succeed, and each succeeds independently with probability p, the task succeeds with probability p^k.
| Per-step success | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 95% | 77% | 60% | 36% |
| 99% | 95% | 90% | 82% |
So "every step works 95% of the time" and "the agent fails four tasks in ten" are both true. Testing steps in isolation gave the team a reassuring number that does not describe the product.
Four levers, roughly in order of how much they usually help:
1. Fewer model-driven steps. Every step removed raises the ceiling. Merge steps that could be one call, and remove steps that exist because the agent was given too many tools. Going from ten model-driven steps to six moves the baseline from 60% to about 74% before anything else changes.
2. Make deterministic what can be. Steps like "look up the account", "format the date", "validate the schema" do not need a model choosing what to do. Code does them at 99.9% or better. If four of the ten steps become code, the task is six model steps at 95% and four code steps at 99.9%: about 73%.
3. Verify and retry. Where a failure can be detected (the tool returned an error, the output fails a schema check, a cheap verifier says the answer contradicts the source), retry the step. With 80% of failures caught and one retry, effective step reliability rises from 95% to about 98.8%, at the cost of about 4% more calls. Applied to the six model steps, the task reaches about 93%.
4. Improve the step itself. Better prompts, better tool descriptions, a stronger model on the hardest step. This raises the base, but each point of per-step accuracy is usually harder to win than the structural changes above.
| Design | Task success |
|---|---|
| Original: 10 model steps at 95% | about 60% |
| 6 model steps, 4 deterministic | about 73% |
| Plus verification and one retry on model steps | about 93% |
The last box matters as much as the arithmetic. When the agent cannot recover, it should stop and hand off with what it has done, rather than continuing with a bad intermediate result. A clean handoff is a far better experience than a confident wrong outcome, and it creates a labelled example of exactly where the agent fails.
Two caveats. Steps are not independent in practice: one confusing input can make several steps fail together, so measure task success end to end on real tasks rather than trusting the multiplication. Anthropic describes the same effect in its own research agent: "One step failing can cause agents to explore entirely different trajectories" (June 2025). And retries are not free: they add latency and cost, so cap them per step and per task.
The retry arithmetic also assumes a second attempt is independent of the first. Often it is not. A model that misread a document once is likely to misread it again given the same input, which is why the verifier should feed back what went wrong, or route the retry to a different method. Published agent benchmarks show how far apart one run and many runs can be: on τ-bench's retail tasks, GPT-4o succeeded on under half the tasks, and passed all eight repeated trials of the same task less than 25% of the time (Yao et al., 2024). Anthropic's eval guide gives the arithmetic: a 75% per-trial agent passes all three of three trials about 42% of the time. Users experience the agent many times, so consistency is part of reliability.
What interviewers probe next
"Where would you look first in the traces?" The step with the lowest success rate on real tasks, and whatever comes immediately before it. Many agent failures start with a slightly wrong intermediate result that later steps build on.
"The team wants a stronger model for every step instead." Measure it. A stronger model might take 95% to 97%, which over ten steps goes from 60% to 74%. Cutting and verifying steps usually does more for less money.
"The agent passes a task on one run and fails it on the next. Which number do you report?" Both, as pass@1 and pass^k. For tasks that move money or data, the share of tasks passed on every run is the one that predicts complaints.
"How do you report reliability to leadership?" Task success on real tasks, with the clean-handoff rate beside it. "Completes 90%, hands off cleanly 8%, fails badly 2%" is a far better product than "completes 93% and gets the rest wrong."
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Diagnosis | "Improve the prompts" | Recognises errors compound across steps | Computes p^k and shows why per-step testing misled the team |
| Levers | One fix | Fewer steps or retries | Fewer model steps, deterministic code, verification with retries, quantified with the resulting task success |
| Failure handling | Not considered | Retries | Caps retries, stops and hands off with context, notes steps are not independent |
Common mistakes
Reporting step-level accuracy as if it were task-level accuracy.
Reaching for a bigger model before reducing the number of steps.
Letting the agent continue after a detectable failure.
See reliability compounding and try the agent reliability calculator.