Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Build cost per ticket from the call up: tokens per call, calls per ticket, retries, then divide by resolution rate. Add the cost of the human who handles what the agent cannot. At typical shapes the inference bill is a few cents per ticket and the human fallback is over a dollar, so resolution rate decides whether it is worth it, not the model price.
How to approach it
State your assumptions out loud and keep them round. Ask what a human-handled ticket costs today, because that is the comparison. Then build the estimate in layers (per call, per attempt, per ticket, per month) so the interviewer can challenge any one number without the whole thing collapsing.
A strong answer
Assumptions, all illustrative:
- Each ticket attempt makes 4 model calls (classify, retrieve and answer, one tool call, a final check).
- Each call sends about 6,000 input tokens (system prompt, policies, conversation, retrieved articles) and gets about 400 output tokens.
- 10% of attempts need a retry.
- A mid-tier model at $2 per million input tokens and $10 per million output, which is Claude Sonnet 5.5's list price as of November 2026 (and, for comparison, OpenAI's GPT-6 Sol).
- The agent fully resolves 65% of tickets. The rest go to a human at $4 per ticket all-in. That figure varies a lot by market: a support centre in Bengaluru or Manila costs a fraction of one in Dublin or Austin.
Now the layers:
| Layer | Arithmetic | Result |
|---|---|---|
| Per call | 6,000 × $2/M + 400 × $10/M | $0.016 |
| Per attempt | $0.016 × 4 calls × 1.10 retries | about $0.070 |
| Per resolved ticket (AI only) | $0.070 / 0.65 | about $0.11 |
| Monthly inference | $0.070 × 1M tickets | about $70,000 |
| Effective cost per ticket | $0.070 + 0.35 × $4 | about $1.47 |
| Today, all human | $4.00 |
So the agent costs about $1.47 a ticket against $4.00 today, a saving of roughly $2.5M a month at this volume. The inference bill is under 5% of the effective cost. Human fallback is the rest.
That changes where to spend effort. Halving inference cost (a cheaper model, prompt caching on the static 4,000 tokens of instructions) saves about $35,000 a month. Raising resolution from 65% to 75% saves $0.40 per ticket, about $400,000 a month. A PM who spends the quarter negotiating token prices instead of improving resolution has optimised the wrong line.
Is it worth it? On these numbers, yes, with three conditions.
First, "resolved" has to mean resolved. If the agent closes conversations that come back as new tickets within two days, the 65% is overstated and the human cost reappears later. I would measure resolution as no re-contact within 72 hours.
Second, quality on the resolved share has to hold. A wrong refund or a frustrated customer who churns can cost more than the $2.50 saved in this estimate. Include audited correctness and subsequent re-contact in the pilot; a closed ticket alone is insufficient evidence of success.
Third, segment it. Resolution of 65% might be 90% on order status and 20% on disputes. Routing disputes straight to a human skips a failed AI attempt and its cost, and gives those customers a better experience.
What the estimate leaves out. Three costs belong in the business case even though they are not per-ticket inference. Integration and maintenance: the agent needs order, refund and identity APIs with the same permissions a human agent has. Evaluation and review: someone reads sampled conversations every week, and that is a standing cost. And shape changes: if the team later adds an agent that plans and calls several tools, token use climbs fast. Anthropic reported that its agents typically use about 4 times the tokens of chat, and multi-agent systems about 15 times (multi-agent research system, June 2025, company-reported). At 4 times, inference here goes from about $70,000 to $280,000 a month, still under a fifth of the effective cost, which is the point: resolution decides.
Check what "resolved" means before comparing with a vendor. Vendors bill on their own definitions. Zendesk's automated resolution tiers (introduced May 2026) use an LLM to read the conversation and decide whether it was resolved, and close an email conversation 72 hours after the first email; Fin counts a resolution when the customer leaves without asking for more help. Both are reasonable, and both differ from "no re-contact on any channel within 72 hours". Restate a vendor's rate on your definition before it goes in this table.
I would reverse the recommendation if the human fallback were already cheap (a mature offshore operation at under a dollar per ticket) and the agent's resolution stayed below about 50%. The margin would then be thin enough that quality risk outweighs savings.
Price source: Anthropic rate card, checked 6 November 2026.
What interviewers probe next
"The CFO says the inference bill is $70K a month and wants it halved." Show the effective-cost table. Halving inference saves $35K; ten points of resolution saves ten times that. Do both, but fund resolution first.
"What if each ticket needs 12 calls, not 4?" Inference triples to about $210K a month and effective cost rises to about $1.61. Still well below $4, which is why the resolution assumption deserves more scrutiny than the call count.
"The human team also gets an AI assistant. Does that change your comparison?" Yes, and it should be in the baseline. A field study of 5,179 support agents found AI assistance raised issues resolved per hour by 15% on average (Brynjolfsson, Li and Raymond, 2025). If the human cost per ticket falls from $4.00 to about $3.50, the agent's saving shrinks but stays large.
"How confident are you in 65%?" Not very until it is measured in shadow mode on real tickets. I would quote a range of 50 to 75% and show the effective cost at each end: about $2.07 and $1.07.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Estimate | Quotes cost per token or a single guess | Builds per-call and monthly inference cost | Layers call, attempt, resolved ticket and effective cost including human fallback |
| Insight | Optimises model price | Notices human cost matters | Shows resolution rate dominates, with the savings from each lever quantified |
| Judgment | "Yes, it's cheaper" | Mentions quality risk | Defines resolution by re-contact, segments by intent, states when the answer reverses |
Common mistakes
Stopping at the inference bill and calling it the cost of the agent.
Using cost per token as the headline. Users and CFOs buy resolved tickets, not tokens.
Assuming the resolution rate a vendor quotes. It depends on how resolution is defined.
Run your own numbers in the AI cost calculator, and see effective cost with human fallback.