Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Design so wrong outputs are easy to notice, easy to check, and easy to undo, in proportion to what an error costs. Show sources or evidence, present output as an editable draft rather than a final answer, let the model ask or abstain instead of guessing, make verification fast for the details that matter most, and provide undo and fallbacks. Measure edits, undos and corrections to see where the 10% lands.
How to approach it
Ask what the feature does and what a wrong output costs: a slightly off summary of a long thread, or a wrong amount in an invoice. Ask whether errors are visible (obviously wrong) or silent (plausible but false). The design depends on those two answers more than on the 90%.
A strong answer
Take an AI feature in a travel app that reads a booking confirmation email and creates an itinerary entry: flight times, hotel dates, confirmation numbers.
Where the 10% goes. Error analysis shows most mistakes are wrong times (time-zone mix-ups), missed second legs of multi-leg trips, and hotels with wrong check-out dates. All are silent and plausible, and some are costly: a wrong flight time can make someone miss a flight.
Patterns, matched to the errors.
| Pattern | Applied here |
|---|---|
| Show the source | Each field links to the exact line in the original email, highlighted |
| Editable draft | "We found these details. Check and save" for the first uses; editable fields |
| Make high-stakes details easy to verify | Flight times shown with the airport's time zone explicitly; departure date in large type |
| Abstain or ask | If the email is ambiguous ("Your flight is on the 3rd"), ask rather than guess |
| Consistency checks | Flag impossible results (arrival before departure, check-out before check-in) |
| Undo and edit | One tap to edit any field; undo after save |
| Fallback | "Couldn't read this email. Add manually" with fields prefilled where confident |
Progressive trust. For a new user, show the review step every time. Once a user has saved, say, ten itineraries with no edits, offer to save automatically and notify, with an easy way to review. Trust is earned one user at a time.
What not to do. Showing "90% confidence" next to every field. Users cannot act on a probability, and a single global number does not tell anyone which field to check. Raw model confidence is also a weak guide: the GPT-4 technical report shows the pre-trained model was well calibrated on a subset of MMLU and that post-training reduced its calibration, so the model that ships is the less calibrated one. Highlighting the specific fields that failed a check or came from ambiguous text is more useful.
Measurement. Edit rate per field (which fields the AI gets wrong), undo rate after save, user-reported problems ("missed my flight time"), and the share of itineraries saved without edits for trusted users. Feed corrected examples into the evaluation set.
The patterns are not a matter of taste. Microsoft Research's Guidelines for Human-AI Interaction (Amershi et al., CHI 2019) set out 18 guidelines and tested them with 49 design practitioners against 20 AI products. Three of them are this table in one line each: make clear how well the system can do what it does, support efficient correction, and scope services when in doubt. Citing the guideline in a design review moves the argument from opinion to a published, tested standard.
I would reverse this approach for a feature where errors are cheap and visible (suggesting playlist names): heavy verification UI would only add friction. The design intensity should track error cost.
What interviewers probe next
"Product wants the review step removed because it adds friction." Remove it for users and email formats with a strong track record, keep it for new users and unusual formats. Friction belongs where errors are likely and costly.
"How do you know users actually check?" Measure time spent on review and whether known errors (from audits) were corrected. If users approve without looking, make the risky fields more prominent or require interaction with them.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Diagnosis | Accepts 90% at face value | Asks about error types | Identifies which errors are silent and costly, from error analysis |
| Patterns | Confidence score on everything | Editable drafts and undo | Source links, targeted verification, ask or abstain, consistency checks, fallback, matched to error cost |
| Trust and measurement | Not addressed | Feedback buttons | Progressive trust per user, per-field edit and undo rates feeding evals |
Common mistakes
Showing generic confidence percentages users cannot act on.
Applying the same heavy review to cheap, visible errors.
Measuring the 90% instead of studying where the 10% lands.