Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Compare effective cost per conversation, including the human who handles what the AI does not: about $3.90 in-house against $2.85 for the vendor, assuming humans cost $6 per conversation. Then stress-test the vendor's definition of resolution. If 15% of its "resolutions" come back through another channel, the vendor's real cost rises to about $3.48 and the gap narrows sharply.
How to approach it
Refuse to compare $0.30 with $1.50. They price different things: one is cost per attempt, the other is price per success. Ask what a human-handled conversation costs, how the vendor counts a resolution, and whether the 70% was measured on traffic like ours. Then put both options on the same footing.
A strong answer
The formula that puts them on one footing is effective cost per conversation:
effective cost = AI cost + (1 − true resolution rate) × human cost
With a human-handled conversation at $6 (illustrative; it depends heavily on market and channel):
| In-house | Vendor (as claimed) | |
|---|---|---|
| AI cost per conversation | $0.30 on every conversation | $1.50 × 0.70 = $1.05 |
| Unresolved share × $6 | 0.60 × $6 = $3.60 | 0.30 × $6 = $1.80 |
| Effective cost | $3.90 | $2.85 |
At 200,000 conversations a month that is $780,000 against $570,000. The vendor looks about $210,000 a month cheaper, despite charging five times more per unit, because resolution rate dominates the cost of the human fallback.
The in-house agent breaks even at about 57.5% resolution: 0.30 + (1 − R) × 6 = 2.85 gives R ≈ 0.575. That is the number I would put in front of the team building it. If they can credibly get from 40% to 60% in a quarter, in-house wins and keeps the data, the failure traces and the ability to switch models.
Now the part that decides the deal: what does the vendor count as resolved? The published definitions differ, which is the point. Fin counts a resolution when the customer "exits the conversation without requesting further assistance", and refunds the charge only if the customer returns to the same conversation (billing rules, checked November 2026). Zendesk has an LLM read the conversation and decide, and treats an email conversation as ended 72 hours after the first email (automated resolution tiers). Both are reasonable proxies, and both are generous in the same way: a customer who gives up and emails instead, or calls the next day, can count as resolved.
The units differ too, so line the offers up before negotiating. As of November 2026, Fin lists $0.99 per outcome; Zendesk lists $1.50 per pay-as-you-go resolution; Salesforce lists Agentforce at $2 per conversation, or Flex Credits at $500 per 100,000 credits with an action costing 20 credits, about $0.10 per action (Agentforce pricing). Per resolution, per conversation and per action are three different bets on who carries the cost of failure. All are company list prices; contracts vary.
Suppose 15% of the vendor's resolutions are customers who come back through another channel within 72 hours. Then:
- We pay $1.50 for all 70% that the vendor counts: $1.05.
- Humans handle the 30% plus 15% of the 70% (another 10.5 points): 40.5% × $6 = $2.43.
- Effective cost: about $3.48, not $2.85.
The vendor still wins, but by about $0.42 a conversation rather than $1.05. Small changes in the definition move hundreds of thousands of dollars a month.
My recommendation: run the vendor on a randomly assigned 10% of traffic for a month, measure true resolution as no contact on any channel within 72 hours, and negotiate the contract on that definition, or on a price that reflects re-contact. Meanwhile, give the in-house team the 57.5% target. Choose at the end of the month on measured effective cost, then weigh the factors the arithmetic leaves out: data ownership, lock-in, and whether support is a strategic capability for us or a cost centre.
What interviewers probe next
"Volume doubles next year. Does the answer change?" Per-outcome pricing scales linearly; in-house inference cost falls per conversation with caching and routing. Higher volume favours in-house if resolution catches up.
"The vendor's 70% is an average across its customers." Our mix may be harder (more disputes, more languages). Their number is a claim until our pilot measures it.
"Finance prefers the vendor because cost is predictable." It is predictable per resolution, but total spend rises with resolution volume. Cap it in the contract.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Comparison | Picks $0.30 as cheaper | Computes effective cost with human fallback | Computes both, finds the break-even resolution rate for in-house |
| Definition risk | Accepts 70% | Questions the vendor number | Shows how re-contact changes the vendor's effective cost and proposes a cross-channel definition |
| Decision | One-off pick | Pilot | Randomised pilot with contract terms tied to measured resolution, plus non-cost factors |
Common mistakes
Comparing cost per attempt with price per success.
Taking a vendor's resolution rate at face value without reading the definition.
Ignoring the cost of humans, which is most of the bill on both sides.
See effective cost with human fallback and metric definitions.