Why measure p95 instead of the average?

Because agent and retrieval latencies have long tails. An average of two seconds can hide one request in twenty taking eight, and those are the experiences people remember.

DECISION TOOL · FREE TO USE

Latency budget builder

Lay out the waterfall to first token, compare it with your p95 target, and find the bar worth fixing.

THE WORKING MODELE2E = retrieval + TTFT + tokens ÷ TPS + tools

Change an input. See what follows.

Network and auth

Retrieval

Reranking

Tool calls

Time to first token

Output tokens

Tokens per second

p95 target to first useful text

First token2,250 ms250 ms spare

Last token7.3 sif you stream

Biggest barTime1,200 ms

Network and auth

Retrieval

Reranking

Tool calls

Time to first token

Streaming

Dashed line: your target. Streaming bar: time to finish generating after the first token.

Inside budget to the first token. If the experience still feels slow, show progress or classic results immediately and stream the answer behind them.

FROM THE RESULT TO THE DECISION

How to reason about it

Interactive AI feels slow for reasons that are rarely the model alone. Retrieval, reranking, tool calls and network time stack up before the first token appears, and the answer then streams at a rate set by output length.

Enter your measurements and target. The builder shows what the user waits for with streaming and without it, and which component to work on first.

Optimise the biggest bar, and run independent steps in parallel before paying for a faster model. If you cannot get under budget, change what the user sees first: progress, classic results, or a shorter first answer.

Questions about this tool

Time to first token: from the request to the first piece of generated output. With streaming it is the delay a user perceives before anything appears, which makes it the latency number that matters most for chat-style products.