DECISION TOOL · FREE TO USE
Launch readiness simulator
Turn a pass rate into a ship, assist or hold call, with the interval and the cost of silent failures.
autonomous if low(CI) ≥ baseline, error cost ≤ today, and reversibleSTART WITH AN EXAMPLE Illustrative inputs
THE CALL FOR THIS SEGMENT
Keep human approval
Its failures cost more than today's. Keep human approval while you reduce the frequency or cost of those errors.
164 / 200 cases
Error cost per 1,000 tasks
One unit is the cost of a safe handoff.
What this calculation assumes
- Examples are illustrative. Replace them with results from one segment of your own eval set.
- Cases are independent and representative of that segment. The chart uses a 95% Wilson interval.
- A safe failure costs one unit. Every failure in today's process is priced as a safe failure; silent AI failures use your chosen multiplier.
- The result follows a simplified decision rule. It does not assess security, legal obligations or every condition of a production launch.
FROM THE RESULT TO THE DECISION
How to reason about it
A pass rate is an input to a launch decision, not the decision. The same 82% means different things on 50 cases and on 5,000, against a process that succeeds 70% of the time or 97%, and when the failures are polite handoffs or confident wrong refunds.
Start with an illustrative scenario, then adjust the evidence and the consequences. The support-agent example prices a silent failure at twenty safe handoffs. Change that ratio to reflect the harm in your workflow.
Consider autonomous use where the agent beats today's process at the bottom of its interval, its errors cost no more, and the worst action is reversible. When evidence is weak, gather more cases. When performance is worse, hold the launch. Human approval may make a narrower use case viable.
Questions about this tool
There is no universal threshold. Compare with the process the feature replaces, account for the uncertainty in the measurement, and weight failures by what they cost. A rate that is excellent for FAQ triage can be unacceptable for account changes.