TL;DR: If a task needs k steps to succeed and each succeeds with probability p, the task succeeds with probability about p^k. 0.95^10 ≈ 0.60 and 0.99^10 ≈ 0.90. You raise task reliability by cutting the number of model-driven steps, making steps deterministic where possible, and verifying steps so detected failures can be retried, roughly in that order of impact.
The arithmetic
An agent completing a task makes a series of decisions: interpret the request, pick a tool, form the arguments, read the result, decide the next step, and so on. If any one goes wrong in a way the agent does not recover from, the task fails. Assuming independent steps:
P(task succeeds) = p₁ × p₂ × … × p_k, or p^k if every step has the same reliability p.
| Per-step success | 3 steps | 5 steps | 10 steps | 20 steps | 50 steps |
|---|---|---|---|---|---|
| 90% | 73% | 59% | 35% | 12% | 1% |
| 95% | 86% | 77% | 60% | 36% | 8% |
| 99% | 97% | 95% | 90% | 82% | 61% |
| 99.9% | 100% | 100% | 99% | 98% | 95% |
Two things stand out. Per-step reliability that sounds excellent (95%) gives poor task reliability once steps multiply. And deterministic code, which runs at 99.9% or better, barely dents the total, so every step converted from model judgment to code helps.
This is why teams report that "each component tests well" while users experience an unreliable product. Testing steps in isolation measures p; users experience p^k.
The levers
1. Fewer model-driven steps
The exponent matters more than the base. Removing steps raises the ceiling immediately. Common sources of unnecessary steps: an agent given too many overlapping tools, a planning step for tasks that always follow the same plan, a self-reflection step that rarely changes the outcome, and separate calls that could be one.
2. Make steps deterministic
Many steps do not need a model choosing what to do: looking up a record by ID, validating a format, computing a total, applying a policy rule. Moved into code, they run at near-perfect reliability. This is also where workflow vs agent design pays off.
3. Verify and retry
If a failed step can be detected, it can be retried. A step with base reliability p, a detection rate d, and one retry has an effective reliability of about:
p_eff = p + (1 − p) × d × p
With p = 0.95 and d = 0.8, p_eff ≈ 0.988. Ten such steps complete about 89% of the time instead of 60%. The cost is extra calls: about (1 − p) × d additional attempts per step, here 4%.
Detection is the hard part. Good verifiers are cheap and specific: a schema check on tool arguments, a check that the tool returned data rather than an error, a test suite for generated code, a comparison of an extracted amount with a source record. A vague "is this right?" verifier catches less and costs more.
4. Improve the step
Better prompts, clearer tool descriptions, a stronger model on the hardest step. This raises p but each point is usually harder won than the structural levers above.
| Design | Task success (10 steps originally) |
|---|---|
| 10 model steps at 95% | about 60% |
| 6 model steps at 95%, 4 code steps at 99.9% | about 73% |
| Plus one verified retry on model steps (d = 0.8) | about 93% |
Caveats
Steps are not independent. A confusing input can make several steps fail together, and an early mistake can make later steps fail. Real task reliability can be better or worse than the formula. Use p^k to reason about design, then measure task success end to end on real tasks.
Retries cost money and time. Cap them per step and per task, and retry only failure types where a retry plausibly helps (a timeout, yes; a permission error, no).
Silent failures are not retried. The formula assumes failures end the task. A step that produces a plausible wrong result and continues is worse: the task "completes" incorrectly. Verifiers matter most at steps where silent failure is expensive.
Recovery changes the picture. Agents can sometimes recover from a failed step by trying another approach, which is part of their value. That recovery is itself a set of extra steps with their own reliability.
Reporting reliability
A product view of reliability needs three numbers, not one:
- Tasks completed correctly.
- Tasks stopped cleanly with a handoff (the agent knew it could not finish).
- Tasks completed incorrectly or abandoned in a bad state.
"Completes 88%, hands off cleanly 10%, fails badly 2%" describes a far better product than "completes 91% and gets the rest wrong". Designing for clean failure is part of designing for reliability.
In the interview
Any agent question can turn to reliability, and computing p^k unprompted is one of the clearest signals of agent fluency. Strong answers do the arithmetic, explain why component testing misleads, rank the levers (steps first, then determinism, then verification, then model quality), and add the caveat about independence and the case for clean handoffs. The agent reliability calculator runs these numbers.