Your main model provider goes down for two hours during peak traffic. What should the product do, and how do you design for it in advance?

TL;DR: Decide in advance, feature by feature, what happens when the model is unavailable: fail over to a second provider or model, fall back to a simpler non-AI experience, queue work for later, or tell users honestly. Put timeouts and circuit breakers in a gateway, keep a tested secondary model for critical paths, and make sure core product functions never depend on a model call.

How to approach it

Separate the response during the incident from the design work before it. Classify features by how critical they are and whether a non-AI fallback exists, then describe the mechanisms (timeouts, circuit breakers, failover, queues) and the user experience in each degraded state.

A strong answer

Classify features. For a travel booking app:

The single most important rule: core transactions (search, booking, payment) must work without any model call. AI enhances them; it must not gate them.

Mechanisms.

• Timeouts. Every model call has a timeout appropriate to the feature. A slow provider is often worse than a failed one, because requests pile up.
• Circuit breakers. If error rates or latency cross a threshold, stop sending traffic to that provider for a period and use the fallback immediately, rather than every request waiting for its own timeout.
• Failover to a secondary model. For critical AI paths, a second provider or a self-hosted model, configured in the gateway. Prompts must be tested on the secondary model in advance; a prompt tuned for one model can behave badly on another.
• Caches. Precomputed outputs (summaries, common answers) keep working during an outage.
• Queues. Non-urgent generation waits and runs when service returns.

During the incident. Confirm failover activated, watch quality on the secondary model (answers may differ), communicate a status message where users notice changes, and keep support informed. Afterwards, review: did every feature degrade as designed, and how long did detection take?

Testing. Run failure drills: deliberately disable the primary provider in staging and periodically in production for a small share of traffic, to confirm fallbacks work. Fallbacks that have never been exercised tend to fail when needed.

Cost of resilience. A secondary provider means maintaining a second set of evaluated prompts and possibly committed capacity. Reserve it for features where an outage is costly; cheaper features can degrade to cached content or a clear message.

What it looked like for real. OpenAI's incident report for 11 December 2024 records every product degrading from 3:16 pm PST, with the API fully recovered at 7:38 pm: about four hours and twenty minutes. The stated cause was a new telemetry configuration that overloaded Kubernetes control planes and broke DNS-based service discovery. The two hours in this question are mild by that standard. Fin's engineering team wrote in August 2025 that it moved traffic from OpenAI to Anthropic models during an OpenAI outage on 10 June 2025, keeps capacity for two to three times normal traffic, and runs to a 99.8% monthly uptime target backed by an SLA (all company-reported). The same post lists deliberately caused outages as planned work and says real vendor outages had kept the failover exercised so far.

Two PM decisions sit in those numbers. An uptime target is a contract term you are held to while depending on vendors whose outages you do not control, so the secondary provider is part of the cost of selling that SLA. And a headroom multiple is capacity someone pays for on quiet days; agree with finance which features earn it.

What interviewers probe next

"The secondary model gives noticeably worse answers. Is failover still right?" For support chat, a somewhat worse answer usually beats no answer, with a lower threshold for handing off to humans. For high-stakes outputs, pausing may be better than degrading.

"How do users know something is wrong?" For visible changes, a brief notice. For invisible fallbacks (cached summaries), no notice is needed.

"Sales wants to promise enterprise customers 99.9% uptime on the assistant. Can you?" Only if you can show it with the failover path included and the drill results to back it. A single provider's bad afternoon, like the four-hour December 2024 outage, is about twice the downtime 99.9% allows in a whole quarter (about two hours and ten minutes). Price the second provider and its evaluated prompts into the enterprise tier, or promise less.

How it's scored

Common mistakes

Letting a core transaction depend on a model call.

Configuring a fallback model without testing prompts on it.

Using long timeouts that let requests pile up during a slowdown.

See fallbacks and degradation and model gateway and routing.