Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: A model gateway is a single internal service through which every model call passes. It handles routing to different models and providers, fallback when one fails, rate limits and budgets per team, logging and cost tracking, caching, and security policy. Its strategic value is that switching models becomes a configuration change rather than an engineering project.
How to approach it
Describe the problem it solves (many teams calling many providers directly, each in their own way), list its functions briefly, then explain the business value with one or two scenarios.
A strong answer
The problem without one. Ten teams each call model providers directly from their code. Each handles keys, retries, logging and errors differently. Nobody knows total spend by feature until the invoice arrives. When a provider has an outage, every team scrambles. When a cheaper model launches, switching means code changes in ten places.
What a gateway does.
| Function | Benefit |
|---|---|
| Routing | Send each request to the right model by task, difficulty, region or cost |
| Fallback | Retry on another provider or model when one fails or slows down |
| Budgets and rate limits | Per-team and per-feature caps; no surprise bills |
| Logging and tracing | Every prompt, response, latency and cost recorded in one place |
| Cost attribution | Spend by feature, team and customer |
| Caching | Reuse responses or prefixes where appropriate |
| Security and policy | Key management, PII redaction, blocking unapproved providers, data residency routing |
Two scenarios that show its value.
Provider outage. The main provider's error rate spikes. The gateway detects it and routes traffic to a configured fallback model within seconds. Users see slightly different answers rather than errors.
Price change. A new model matches quality on summarisation at a third of the price. The evaluation suite confirms it on the golden set. The routing rule for summarisation changes in the gateway; no application code changes.
Two published gateways. Uber described its GenAI Gateway in July 2024: close to 30 customer teams, 16 million queries a month at a peak of 25 queries per second, models from OpenAI, Vertex AI and Uber's own hosting behind one interface, audit logs for cost attribution, and a PII redactor that anonymises requests before they reach a third-party vendor and restores the values in the response (all company-reported). Wealthsimple's engineering lead told QCon San Francisco in November 2024 that about 80% of the company's LLM usage went through its gateway. Its 2023 tactic of sending Slack nudges to people who bypassed the gateway had little effect on behaviour and was dropped in 2024. The incentive it kept was free access: the company paid the API fees, so the gateway was the cheapest way to use a model.
The second case is the one a PM should remember. A gateway is an internal product with users who can walk around it, and the 20% that bypasses it is where the unredacted data and the unattributed spend are. Coverage, the share of model spend that flows through the gateway, belongs on its dashboard next to latency.
Who owns it. Usually a platform or ML infrastructure team. Product teams consume it and get visibility into cost and quality for their features.
A caution. A gateway is a critical path: if it goes down, every AI feature goes down. It must be highly available, add very little latency, and be kept simple.
What interviewers probe next
"Doesn't a gateway add latency?" A well-built one adds milliseconds, small next to model latency. Keep heavy processing (like large redaction models) optional per route.
"Can we buy one instead of building?" Yes; several products and open-source projects exist. The decision rests on control needs, data residency and how much custom routing logic you need.
"A fifth of model traffic bypasses the gateway. Do you mandate it?" Find out why first: a missing model, a latency penalty, a feature such as streaming or batch the gateway lacks, or a team with its own provider contract. Fix the reasons that are product gaps, then block direct provider keys at the network or procurement level for the rest. Wealthsimple's experience suggests that nudges alone do little; being the cheapest and easiest path does more.
How it's scored
Weak: Describes it only as "an API wrapper" with no business value.
Hire: Lists routing, fallback, logging and cost tracking.
Strong Hire: Adds budgets, security and residency, and explains the strategic value in outage and model-switch scenarios, plus the availability caution. Treats adoption as a product problem and measures coverage.
Common mistakes
Treating it as an engineering nicety rather than a lever on cost and lock-in.
Forgetting that it becomes a single point of failure.
Letting teams bypass it, which destroys cost visibility.