latency budget

Putting a large model in the search path can improve relevance and blow the latency budget. Use a cascade, rerank only where it helps, cache, and measure on judged queries and live tests.

Twenty thousand calls a day, two languages and a telephone network you do not control. Scope the intents, budget every millisecond and price the handoff before you promise containment.

Cropping beats a bigger model, a smaller embedding buys back latency, and a stale index sends shoppers to items they cannot buy. Make each of those calls with numbers.

The candidate who says 'use a faster model' in the first sentence has already lost points. Decompose the budget, find the biggest bar, and check whether two seconds is the right target at all.