Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Find out why agents distrust the summaries: a few memorable errors, summaries that omit what agents need, no way to verify quickly, or accountability for mistakes landing on them. Then fix the specific cause: make summaries checkable (claims linked to the ticket), include what agents actually look for, show a track record, and let agents flag errors that visibly get fixed. Measure trust through behaviour, such as how often agents open the full thread.
How to approach it
Do not dismiss the agents' behaviour as irrational. Experts who are accountable for outcomes are right to be cautious until a tool proves itself in their terms. Investigate with interviews and data, then address causes one by one.
A strong answer
Investigate. Sit with five agents and watch them work. Look at the data: what share of tickets do agents open in full after reading the summary, by agent, ticket type and summary length?
Typical causes, and what each looks like:
| Cause | Signal |
|---|---|
| Memorable failures | Agents cite one or two incidents ("it said the refund was approved when it wasn't") |
| Missing what they need | Summaries describe the conversation but omit order numbers, prior promises, customer sentiment |
| No fast verification | Checking a claim means scrolling through a long thread |
| Accountability | Agents are blamed for mistakes; a wrong summary is their problem, not the AI's |
| Evals measure the wrong thing | Eval says "accurate" because nothing is false, but the important detail was omitted |
The last row matters: the eval may check for false statements and miss omissions. Agents notice omissions immediately.
Fixes, by cause.
- Make claims checkable. Every line of the summary links to the message it came from; hovering shows the original. Verification drops from a minute to seconds.
- Summarise what agents need. Co-design the summary format with agents: customer's goal, what has been promised already, order and account identifiers, open questions, sentiment. Use structured fields instead of prose paragraphs.
- Fix the eval. Add omission checks for the fields agents care about and re-measure. The "accurate" claim may not hold.
- Show a track record. A visible quality indicator built from agents' own feedback ("agents flagged 1.2% of summaries last month"), and a release note when an issue they reported is fixed.
- Make flagging easy and consequential. One-click "this summary is wrong", routed to the team, with flagged cases added to the eval set.
- Address accountability. With support leadership, agree how errors caused by AI summaries are handled, so agents are not penalised for trusting a tool the company asked them to use, while still expecting them to check flagged or high-stakes tickets.
Measure trust through behaviour. Share of tickets opened in full after reading the summary, time per ticket, flag rate, and agent satisfaction in short surveys. Success looks like verification concentrated on complex or high-value tickets, and confidence elsewhere.
Two public reference points. When Apple's notification summaries drew reported complaints about inaccurate news summaries, its iOS 18.3 release (January 2025) did three things: made summaries for News & Entertainment apps "temporarily unavailable", restyled the remaining summaries in italic text with a glyph so they read as summaries and not as the original notification, and added Lock Screen controls to manage them. Pull the category where an error costs most, label the summary as a summary, hand the user a control: all three transfer to this support tool. The Stack Overflow 2026 Developer Survey points the same way for expert users: 48% said they trust AI when they can easily validate the answers, and 16% extended trust to most tasks that are not important work decisions. Trust tracks how cheap verification is, which is why the line-level source links come first in the fixes above.
What interviewers probe next
"Should we force agents to rely on summaries?" No. Forced reliance produces either silent errors or resentment. Earn reliance with evidence and better design.
"Senior agents trust it less than new agents. Why?" Senior agents know the edge cases and have seen the failures. Their feedback is the most valuable input for improving the summaries.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Stance | Agents should trust the evals | Investigates distrust | Treats expert caution as rational, observes work and analyses behaviour |
| Diagnosis | One cause | Several causes | Includes eval blind spots on omissions and accountability |
| Fixes and measurement | Training sessions | Source links | Checkable claims, co-designed structure, track record, consequential flagging, accountability policy, behavioural trust metrics |
Common mistakes
Assuming accuracy on an eval equals usefulness to experts.
Ignoring omissions, which experts notice first.
Measuring trust with surveys alone instead of behaviour.
See measuring trust and designing for uncertainty.