TL;DR: Before choosing a model or an interface, map each failure mode by cost, visibility and recoverability. Cheap, visible, reversible errors allow automation with undo. Expensive, silent or irreversible errors require verification before action, a human in the loop, or no model at all. The map tells you what to automate, what to review, how strict the evaluation bar is, and where to launch first.
Why start here
Every AI system is wrong some of the time. Whether that is acceptable depends almost entirely on what a wrong answer does. A music app recommending a song you skip has an error rate nobody notices. A claims system paying a fraudulent claim, a scribe recording the wrong dose, or an agent deleting records has an error that matters even at one in ten thousand.
Teams often start with the model and work backwards to the risk. Starting with the cost of being wrong produces better designs, because it decides the structure: what the AI may do on its own, what must be checked, and how good it must be before launch.
Four questions per failure mode
How much does it cost? In money, time, harm, trust or regulatory exposure. A wrong restaurant suggestion costs seconds; a wrong refund costs its amount; a wrong medical note can cost far more.
Who absorbs it? The user, the business, a third party? Errors that land on third parties (a pedestrian, an applicant denied credit, a person described in a summary) carry ethical and legal weight beyond their cost to the business.
Is it visible? Visible errors (an obviously wrong answer) get caught and corrected. Silent errors (a plausible wrong figure, a confident misstatement) get acted on. Silent errors are almost always the more dangerous category.
Can it be undone? Archiving can be reversed; deleting may not. A draft can be edited; a sent email cannot be unsent. A held booking can be released; a purchased non-refundable fare cannot.
The map
Plot failure modes on two axes: cost and recoverability.
| Recoverable | Hard to recover | |
|---|---|---|
| Low cost | Automate, offer undo | Automate with confirmation for bulk actions |
| High cost | Automate with review sampling, or draft for approval | Human decides; AI prepares, or stays out |
Visibility moves a failure mode toward the bottom right: a silent error behaves as if it were more costly, because nobody catches it in time.
What the map decides
Autonomy. Low-cost recoverable actions can run automatically; high-cost irreversible ones need approval. Autonomy is assigned per action, not per product. See autonomy levels.
Review design. Where errors are catchable, put the catch in the flow: the user confirms extracted fields, the reviewer sees a diff, the clinician confirms medications. Make the check fast, or it will be skipped.
Evaluation bars. Each failure mode gets its own tolerance, set from its cost. "Under 5% generic replies" and "zero replies admitting liability" can coexist in the same feature. A single overall accuracy number hides the difference.
Launch scope. Launch first where failures are cheap and recoverable, gather evidence, then expand to costlier segments. A support agent might start with order-status questions before handling refunds.
Pricing and contracts. Where the business absorbs costly errors (a vendor guaranteeing outcomes), the cost of wrong answers belongs in the price and in the contract terms.
Expected cost
Where numbers are available, expected error cost turns the map into arithmetic:
expected error cost per task = Σ (rate of failure mode × cost of that failure)
A 3% rate of silent wrong refunds at ₹2,000 each is ₹60 per task, which may exceed the entire value the automation creates. A 15% rate of safe handoffs to humans at ₹40 each is ₹6 per task. The first failure mode, despite its lower rate, dominates the decision. This is why a single pass rate cannot decide a launch.
Common mistakes
- Treating all errors as equal and optimising an average.
- Assuming a human reviewer catches errors without measuring how many they catch.
- Ignoring silent failures because complaints are low.
- Granting autonomy to a whole agent instead of per action.
In the interview
Cost of a wrong answer is the move that turns a generic design answer into a senior one. Early in any design question, say what happens when the AI is wrong and who pays, then let that shape autonomy, review, evaluation bars and launch order. Interviewers consistently mark candidates higher when they distinguish silent from visible errors and recoverable from irreversible ones, and when they weight failure modes by cost rather than counting them.