AI System Design Interview Questions for PMs

A RAG pipeline is the obvious first sketch. A strong answer starts with permissions instead, because one leaked salary document ends the product.

A small architectural component with outsized strategic value. The interviewer wants to hear what it does day to day and why it matters when a provider changes prices or goes down.

A full-stack AI system design with scale, money, languages and angry customers. The interviewer wants tiers: cheap triage for everything, an agent where it pays, and humans where it matters, with the cost per day worked out.

A pipeline design question with an automation trap. Writing summaries is the easy part; writing into the system of record that forecasts revenue is where the design needs care.

Every AI product depends on services it does not control. The interviewer wants a degradation plan by feature, written and tested before the outage starts.

A short question with a large payoff. A PM who knows the one rule of prompt caching can cut a feature's input bill dramatically without touching quality.

Latency-dominated system design. A suggestion that arrives after the developer has typed the next characters is worthless, so every design choice is about milliseconds, context and cost per keystroke.

A deadline you did not choose and a change that touches every output. The interviewer wants a plan that treats API compatibility and behavioural compatibility as different problems.

A design question with a constraint dropped halfway. The interviewer wants to see residency flow through every component: model calls, logs, caches, embeddings, evals and support access.

A security design question that decides whether enterprise customers will sign. The answer has to make cross-tenant leakage impossible by construction, and then prove it with tests.

A large-scale design question where every error is someone's real harm: harmful content left up, or legitimate speech taken down. Strong answers tier by severity and confidence, keep humans in the hard cases, and measure both kinds of error per language.

Without traces, every quality complaint is a guess. The interviewer wants the fields that make debugging, evaluation and cost control possible, and an awareness of what should not be stored.

Search plus generation at consumer scale. The design has to deliver fresh, cited answers in a couple of seconds, resist manipulated pages, and keep cost per query viable.

Launches fail on rate limits more often than on model quality. This question checks whether you can turn expected usage into tokens per minute, compare it with limits, and plan headroom.

Generation at scale where the model writes copy nobody reviews before it reaches millions of phones. The design is about templates, guardrails and cost, more than about the model.

Translation looks solved until a price, a date or a promise is mistranslated between two strangers doing business. The design centres on speed, protecting critical details, and letting both sides see the original.

A regulated pipeline where the model reads documents but must not be the one deciding who gets an account. The interviewer wants extraction, checks against authoritative sources, fraud signals, human review and data minimisation.

A cost lever that can silently serve wrong answers and leak one user's data to another. The interviewer wants the failure modes and a design that keeps the savings where they are safe.

A principal-level platform question. Each team building its own agent stack means dozens of security models, cost blind spots and incompatible tools. The platform has to make the safe path the easy path without slowing teams down.

A platform design question about the tooling that decides launches. The hard part is not running prompts at scale; it is making error analysis, labelling and judge validation easy enough that teams actually do them.

At a billion requests a day, a fraction of a cent per call is millions of dollars a month and every millisecond is capacity. Principal answers start from the arithmetic and reshape the feature, the models and the serving to fit.

Guardrails are often described as a box in a diagram. The interviewer wants specific checks, where each runs, what it costs in latency, and what happens when one fires.

A lawyer will not trust a summary. Design clause extraction, playbook comparison and cited deviations, with a measured miss rate on the clauses that carry liability.

The model is one box. Prompt and output filters, brand and rights rules, provenance labels, cost per kept image and a queue for peak hours decide whether the feature works.

Putting a large model in the search path can improve relevance and blow the latency budget. Use a cascade, rerank only where it helps, cache, and measure on judged queries and live tests.

Define what unread means, preserve edits and permissions, and make every action item traceable. The design must tell users when the summary covers only part of their inbox.

A model wins on connected tasks but leaves field staff without help during outages. Compare all scheduled work, then defend local data, expiring permissions and synchronization.

Twenty thousand calls a day, two languages and a telephone network you do not control. Scope the intents, budget every millisecond and price the handoff before you promise containment.

Open schema access answers more questions and produces more wrong numbers that look right. Choose the scope, the permissions and the evidence each answer must show.

Self-reported minutes saved, a randomised trial and one deployer's own records disagree by an order of magnitude. Decide what the scribe may do, how patients are told and what you will measure.

A cheap model is fine for colour and wrong for safety claims. Set a threshold per attribute, price the batch run and decide who wins when the seller disagrees.

The token bill drops by two-thirds and failures rise by one point. Price the failures before you call it a saving, and check what routing does to your cache.

Stuffing every past conversation into context is expensive and still forgets. Decide what becomes a memory, how deletion cascades and how you test updates and forgetting.

Two hundred tool calls at 99.9% each fail somewhere 18% of the time. Decide what is checkpointed, what must be idempotent and what happens to a run in flight when you deploy.

Students solve more practice problems with an AI that gives answers, then score worse on the exam. Decide what the tutor may reveal, how it gets the maths right and what you will measure.

Tokens cost under a dollar per run; reviewer time costs thirty times that. Choose the task classes, the permissions and the gates that decide what a human ever sees.

Accept rate can rise 28% while accepts stay flat. Define the reward, correct for what users never saw and decide whose data you may train on.

Cropping beats a bigger model, a smaller embedding buys back latency, and a stale index sends shoppers to items they cannot buy. Make each of those calls with numbers.

Assistant users buy 60% more often, which proves almost nothing. Ground every price in live data, separate paid placement from advice and measure lift with a holdout.

Regulators fine firms for missed crime, not slow reviews. Decide what the agent prepares, what a named analyst must decide and what a back-test must show before anything closes automatically.