Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Organise metrics along the developer funnel and the platform's health: acquisition and activation (sign-ups, time to first successful call), adoption and retention (developers and organisations still active after 30 and 90 days, usage growth per account), reliability and performance (error rates, latency percentiles, rate-limit hits), economics (revenue, gross margin by tier, revenue concentration), and change health (migration success when models are released or retired). Token volume matters, but as an outcome of these.
How to approach it
Clarify the business: self-serve developers, enterprises, or both. Then map metrics to stages of the developer funnel, add platform health metrics, and choose one or two primary metrics for the team.
A strong answer
Developer funnel.
| Stage | Metric | Why |
|---|---|---|
| Discover | Docs visits, sign-ups by source | Top of funnel |
| Activate | Time to first successful API call; share of sign-ups making one within a day | The single best predictor of later adoption for most developer products |
| Build | Share of developers making calls on 3 or more distinct days in their first two weeks | Moving from experiment to project |
| Ship | Organisations with production traffic (sustained daily calls, keys in production environments) | Real adoption |
| Grow | Usage and revenue per organisation over time; expansion into new models or features | Depth |
| Retain | 30 and 90-day retention of active organisations; net revenue retention | Health |
Platform health.
| Area | Metric |
|---|---|
| Reliability | Error rate by endpoint and model, with overload errors counted separately; availability against published targets |
| Performance | Time to first token and end-to-end latency at p50, p95, p99 by model and region |
| Capacity | Rate-limit responses (429s) by tier and by cause (per-minute limit, acceleration limit, spend cap); queue times at peak; cache hit rate by organisation |
| Quality | Regression reports after model updates; support tickets about output quality |
| Change | Migration completion rate before deprecation deadlines; error spikes after model releases |
| Economics | Revenue by tier and model; gross margin; share of revenue from top 10 customers |
| Developer experience | Support tickets per active developer; docs search failures; SDK issue volume |
Why one 429 count misleads. Anthropic's rate limits documentation, checked 6 November 2026, returns HTTP 429 for three different events: a per-minute rate limit, an acceleration limit triggered by a sharp rise in an organisation's usage, and a monthly spend cap ($500 on Start, $1,000 on Build, $200,000 on Scale), after which usage pauses until the first of the next month. The spend-cap response carries the error code enforced_spend_limit_reached and no retry-after header, so automatic retries keep failing until access resumes. On a single 429 chart, a customer whose product has gone dark for the rest of the month looks the same as one sending bursts too fast. Server overload has its own status, 529 overloaded_error, and belongs on the reliability row, because it is the platform's failure and not the customer's.
The same page counts only uncached input tokens toward the input-tokens-per-minute limit. Its worked example: with a 2,000,000 limit and an 80% cache hit rate, an organisation can process 10,000,000 input tokens a minute. Cache hit rate is therefore a capacity metric as well as a cost metric, and raising it is often the cheapest help you can give a customer who keeps hitting limits.
Primary metrics. For a self-serve-heavy platform: activated organisations reaching production (activation quality) and net revenue retention (depth and stickiness). Token volume and revenue follow from these. For an enterprise-heavy platform, add committed revenue and time to production for new enterprise customers.
Guarding against misleading growth. Token volume can grow from a few customers running large batch jobs, which says little about platform health. Watch concentration and the number of organisations growing, not just total volume.
What interviewers probe next
"Time to first call is 40 minutes. How do you cut it?" Watch new developers try: account setup, key creation, finding a working example, and the first error message are common blockers. Copy-paste quickstarts in popular languages and clear error messages usually help most.
"Support tickets about 429s doubled this month. Do we raise limits?" Split them by cause first. Spend-cap 429s need a clearer warning before the cap and a faster path to a higher tier; acceleration 429s need guidance on ramping traffic; true per-minute limits may need capacity. Raising every limit fixes only the third and costs the most.
"A new model release increased errors for some customers." Track error rates by customer and model version after releases, with alerts, and reach out proactively to affected customers.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Coverage | Token volume and revenue | Funnel metrics | Full developer funnel plus reliability, capacity, change health and economics |
| Prioritisation | Many equal metrics | Picks a primary | Primary metrics justified by business mix, with concentration guardrails |
| Insight | None | Time to first call | Explains activation quality, migration health and misleading volume growth; splits 429s by cause and treats cache hit rate as capacity |
Common mistakes
Using total token volume as the main success metric.
Ignoring migration health, which defines developer trust in a platform that changes models often.
Missing rate-limit and latency metrics by tier and region.