Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: First explain the gap: usage grew 1.4x but cost grew 3x, so cost per request roughly doubled. Find out why (longer contexts, more calls per task, a model change, a few heavy users) before cutting. Then apply levers in order of risk: remove unnecessary work, trim context, cache stable prefixes, cap output, route easy requests to smaller models, batch what can wait.
How to approach it
Decompose the bill before proposing fixes. Total cost is requests times tokens per request times price per token, so a 3x rise with 1.4x requests means tokens or price per request rose about 2.1x. Ask what launched this quarter. New features, especially agentic ones, are the usual cause.
A strong answer
I would spend the first two days on attribution, broken down by feature, model and customer segment:
- Tokens per request, input and output separately. Input tokens creep up as conversation history grows, more documents get retrieved, or someone adds examples to the system prompt.
- Calls per user action. An agent feature that plans, calls tools and checks itself might make eight calls where the old feature made one.
- Model mix. Did a team move a high-volume feature to a larger model for a small quality gain?
- Concentration. What share of cost comes from the top 1% of users or accounts? On flat-priced plans, a few heavy users or automated scripts can dominate.
The answer is usually one or two specific things: a new agent feature and history that is never truncated, for example. Each needs a different fix. Agent features deserve suspicion first: Anthropic reported its agents typically use about 4 times the tokens of chat, and multi-agent systems about 15 times (June 2025, company-reported). One such launch on a popular feature explains a tripled bill on its own.
Then the levers, in the order I would pull them, from least to most risk to quality:
| Lever | What it does | Typical saving | Quality risk |
|---|---|---|---|
| Remove unnecessary work | Skip the model for requests rules can handle; stop calling it on every keystroke | Large on affected traffic | None if done right |
| Trim context | Summarise old conversation turns, retrieve fewer and better chunks | 20 to 50% of input tokens | Low, verify on evals |
| Prompt caching | Reuse the stable system prompt and instructions across calls | Large on the cached portion | None |
| Cap output | Ask for shorter answers, set max tokens | Output is priced several times input | Low for most features |
| Route by difficulty | Small model for easy requests, large for hard | Depends on mix | Needs per-segment evals |
| Batch | Non-urgent jobs through a batch API at a discount | 50% on eligible jobs | None, latency only |
Caching can be the biggest single lever, but only at a good hit rate. If each call sends a 4,000-token system prompt, on Claude Sonnet 5.5 that costs $0.008 per call at the $2 per million input rate. Read from cache at $0.10 per million, it costs $0.0004; writing it to the cache costs $2.50 per million, or $0.01, a premium over an uncached call (rates as of November 2026). Anthropic's cached prefixes expire after five minutes of inactivity by default (prompt caching docs), so the saving depends on traffic:
| Cache hit rate | Cost of the 4,000-token prefix per call | Saving against no cache |
|---|---|---|
| 0% (every call writes) | $0.0100 | none: 25% more expensive |
| 50% | $0.0052 | 35% |
| 90% | $0.0014 | 83% |
| 99% | $0.0005 | 94% |
A busy feature lives near the bottom rows; a feature used once an hour per customer lives near the top. It also requires the stable part of the prompt to come first and stay byte-identical. OpenAI's caching guide gives the same advice: put timestamps, user-specific content and other dynamic content at the end. That is a prompt-layout decision a PM can push for.
Every lever except removing work and caching gets checked against the eval suite before rollout, with cost per successful task as the comparison, not cost per request. A cheaper model that drops resolution by five points can raise the effective cost once human fallback is counted.
I would also fix the pricing side if concentration is the problem. If 1% of accounts drive 30% of cost on a flat plan, the answer may be fair-use limits or usage tiers rather than engineering.
Finally, put cost per active user and cost per successful task on the team's weekly dashboard, by feature, so the next tripling is caught in week two rather than at quarter end.
Price source: Anthropic rate card, checked 6 November 2026.
What interviewers probe next
"Engineering wants to self-host an open-weight model to cut cost." Possible at high, steady volume, but it adds an operations burden and needs evals on our workload. Do the cheap levers first; they often remove the case for it.
"Which lever would you never pull?" Cutting quality on high-risk flows to save money. The savings are visible immediately; the cost shows up later as refunds and churn.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Diagnosis | Switch to a cheaper model | Looks at usage and price | Decomposes into requests, tokens per request, calls per action, model mix and user concentration |
| Levers | One lever | Several levers | Orders levers by quality risk, quantifies caching, checks every change on cost per successful task |
| Durability | One-time cut | Monitoring | Cost per active user by feature on a weekly dashboard, pricing fixes for heavy-user concentration |
Common mistakes
Switching models before finding out where the money goes.
Measuring savings per request while success rate falls.
Ignoring the pricing model when a small share of users drives most of the cost.