latency

A strategy-sized design question. The tempting answer replaces the search bar with a chatbot. The scoring answer keeps fast keyword search for the shoppers who know what they want and uses AI for the shoppers who don't.

Faster speech can make a conversation take longer. Diagnose premature turns, design recovery after interruption and protect actions when a caller changes their mind.

The first vocabulary word of AI product management. Tokens are the unit of cost, speed and memory, and a PM who can't reason in them can't price, budget or design an AI feature.

A larger context window changes what fits in a request. Decide whether to use it by checking corpus size, freshness, permissions and cost per successful query, including cache misses.

A fluency question about the request path. The interviewer does not need the maths; they want to know you can connect each stage to latency, cost and design decisions.

A question about fairness and unit economics hiding in a tokenizer. The interviewer wants the mechanism, the arithmetic, and options that do not leave non-English users with a worse product.

A platform strategy question with hardware, privacy, cost and capability all in play. Principal-level answers design a split by task, with a privacy story users can understand and a plan for the devices that cannot run local models.

Distillation is how teams get large-model quality at small-model cost on a specific job. The interviewer wants to know when that trade is worth the engineering, and what can go wrong.

Voice adds two models and a much tighter latency budget to a text assistant. The interviewer wants the pipeline, the turn-taking problem, and the failure modes that text products never face.

Reasoning tokens are billed as output and add latency. Find the lowest effort that holds quality for each feature, and spend high effort where it measurably helps.

Translation through English helped low-resource Indian languages and hurt Hindi in one benchmark. Route by language, protect what must not be translated and measure on your own customers' messages.

A short question with a large payoff. A PM who knows the one rule of prompt caching can cut a feature's input bill dramatically without touching quality.

Latency-dominated system design. A suggestion that arrives after the developer has typed the next characters is worthless, so every design choice is about milliseconds, context and cost per keystroke.

Search plus generation at consumer scale. The design has to deliver fresh, cited answers in a couple of seconds, resist manipulated pages, and keep cost per query viable.

Translation looks solved until a price, a date or a promise is mistranslated between two strangers doing business. The design centres on speed, protecting critical details, and letting both sides see the original.

At a billion requests a day, a fraction of a cent per call is millions of dollars a month and every millisecond is capacity. Principal answers start from the arithmetic and reshape the feature, the models and the serving to fit.

A voice agent can fail at hearing, understanding, deciding or speaking. Evaluate each stage, then whole calls, segmented by language, accent and line quality.

A vocabulary question about speed. The right metric depends on whether the user is watching text stream in or waiting for a finished result.

The candidate who says 'use a faster model' in the first sentence has already lost points. Decompose the budget, find the biggest bar, and check whether two seconds is the right target at all.

The cost of the model is rarely the biggest number in this decision. The cost of a wrong answer is. One calculation makes the choice obvious in some products and wrong in others.

Batch pricing halves the bill on eligible work, and the real product question is what you can precompute before anyone asks for it.

A serving optimisation whose value depends on traffic shape. The interviewer wants the mechanism, the evidence that gains shrink under load, and a decision tied to the product's latency and throughput needs.

Model latency depends on input length, output length, load and the model itself, so a naive guarantee is either meaningless or impossible. The interviewer wants a guarantee that is defined precisely, measurable and backed by capacity.

Two labs moved from capacity commitments to a pay-per-token fast mode. Decide what speed means, how you stop the fast lane being swamped and who still needs reserved capacity.