Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Tier the system. A small, cheap model classifies every conversation by intent, language and risk. Routine intents (order status, delivery delay, missing item within policy) go to an agent that uses real order data and tools inside policy limits; risky or emotional cases go straight to people with a summary. Ground every fact in order systems, cache the static prompt, and measure resolution by re-contact. Illustratively, this runs around $10,000 a day in inference at this scale.
How to approach it
Clarify the intent mix, languages, what actions support can take (refunds, credits, re-delivery, rider contact), peak patterns (dinner rush, festivals, rain), and the current cost of a human-handled conversation. Then design tiers by intent and risk, size the cost, and plan monitoring and rollout.
A strong answer
Intent mix, illustratively. Order status and delays 40%, missing or wrong items 20%, refunds and payment issues 15%, account and app questions 10%, food quality and safety complaints 8%, rider or restaurant behaviour 7%. Users write in English, Hindi, other Indian languages, and code-mixed text.
Architecture.
1. Triage on everything. A small, fast model classifies intent, language, risk and sentiment. Rules override: any mention of food safety, allergy reactions, harassment or rider accidents routes to a person immediately.
2. Agent for routine intents. The agent looks up the order, delivery status and payment record through tools, never guesses them. For missing items within policy (photo provided, below a value threshold, customer history normal), it can issue a refund or credit through a policy engine that enforces limits in code. Every money action carries an idempotency key and an audit record.
3. Humans for the rest. Escalations arrive with a structured summary: what the customer wants, order details, what was checked, why it escalated. Agents should never need to re-ask.
4. Language. The agent replies in the user's language and script. Evaluation sets per language, built from real conversations; languages that do not meet the bar run in assist mode (drafts for human agents) until they do.
5. Peaks. Dinner rush and bad weather can triple volume within an hour, mostly delay queries. Proactive messages to affected orders ("your order is delayed by about 15 minutes due to rain") prevent many contacts. Provider rate limits and fallbacks in the gateway must handle peak load.
Cost, illustratively, at list prices rechecked on 6 November 2026.
| Component | Assumption | Daily cost |
|---|---|---|
| Triage on 1M conversations | Small model at $0.10 / $0.50 per million tokens, about 1,000 tokens in | about $110 |
| Agent on 600K routine conversations | 3 calls each, mid-tier model at $2 / $10, 4,000-token cached prefix at $0.10, 1,000 fresh tokens, 300 output | about $9,700 |
| Total inference | about $9,800 a day, roughly $300,000 a month |
Against human handling of even a fraction of those conversations, this is small. The economics are decided by true resolution rate and refund accuracy, not inference.
Measuring it. Resolution as no re-contact on any channel within 48 hours, by intent and language; refund leakage from weekly audits of agent-issued refunds; escalation quality (did humans need to re-ask?); CSAT; cost per resolved conversation; p95 response time at peak.
A public reference point. Airbnb's Q2 2025 shareholder letter, filed on 6 August 2025, reported expanding its AI customer service agent to 100% of US users and "reducing the percentage of hosts and guests who needed to contact a human agent by 15%", with more countries and languages to follow (company-reported, no baseline published). Two choices in that sentence are worth copying. The headline metric is the share of users who still needed a person, so a customer who gave up on the bot and phoned in counts against it; a bot resolution rate would have missed them. And the rollout was gated by market and language, which is the same staging this design uses.
Rollout. Start with order status in English and Hindi, add missing-item refunds below a low threshold, raise thresholds and add languages as audits support it.
What interviewers probe next
"Customers start claiming missing items to get free refunds." Customer-level history and fraud signals in the policy engine, photo requirements, and lower auto-refund limits for accounts with unusual claim rates. The agent never decides fraud; the rules and models built for it do.
"The main model provider has an outage at 8 pm on a Friday." The gateway fails over to a secondary model for the agent; triage falls back to rules; proactive delay messages reduce inbound volume; humans prioritise urgent cases.
"Your dashboard says 70% resolved. Finance says human headcount has not moved. Who is right?" Probably finance. Check how many "resolved" conversations were followed by a call or email within 48 hours, and report the share of customers who needed a person at all, the way Airbnb did. Headcount moves when that number moves.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Architecture | One chatbot for everything | Triage plus agent plus humans | Risk-based triage with rule overrides, tool-grounded agent, policy engine in code, structured handoffs |
| Scale | Not considered | Mentions peaks | Peak handling with proactive messaging, rate limits and fallbacks; daily cost computed |
| Measurement | CSAT | Resolution rate | Cross-channel re-contact by intent and language, refund audits, escalation quality, cost per resolution |
Common mistakes
Letting the model decide refund amounts without code-enforced limits.
Treating all intents the same, including safety complaints.
Ignoring proactive messaging, which removes contacts before they happen.
See the tool execution envelope and effective cost with human fallback.