TL;DR: An LLM request has two phases that stress hardware differently. Prefill reads the whole prompt at once and is limited by arithmetic, so it sets time to first token. Decode produces one token at a time and must re-read the model's weights for each step, so it is limited by memory bandwidth and sets how fast text streams. That split explains the 2025 and 2026 hardware cycle: chips with more memory and bandwidth, chips built only for the prompt phase or only for decode, and serving systems that run the phases on different machines. A PM rarely buys chips, but the split decides which latency you can buy, what a fast tier costs and why long prompts hurt everyone on a shared pool.
Two phases, two bottlenecks
How LLMs generate introduces prefill and decode. The hardware view adds one question to each: what runs out first?
Prefill runs out of arithmetic. Every prompt token passes through every layer in parallel, so the chip has plenty of work to keep its multipliers busy. A forward pass costs roughly two operations per parameter per token. A 10,000-token prompt on a dense 70B model is about 2 × 70B × 10,000 = 1.4 × 10^15 operations. Google documents a peak of 4,614 FP8 teraflops per Ironwood (TPU7x) chip. At perfect utilisation that prompt would take about 0.3 seconds; real utilisation is lower, and the gap is what serving engineers spend their time on.
Decode runs out of memory bandwidth. Each new token needs the whole set of active weights read from memory again, plus the KV cache for the conversation so far. The arithmetic per step is small, so the chip waits on memory. Here is an illustrative ceiling for one stream: 70 GB of FP8 weights on a chip that moves 7.37 TB/s gives at most 7,370 ÷ 70 ≈ 105 tokens a second, before reading the KV cache and before any overhead.
This is why batching matters so much. One read of the weights can serve every stream in the batch, so throughput rises with batch size until compute or KV-cache memory runs out. It is also why a mixture-of-experts model, which reads only its active experts per token, can stream faster than its total size suggests.
The request on the hardware
Follow one request. The diagram marks the two stages where the hardware limit differs.
Follow the flow, one stage at a time.
Illustrated sequence; animation speed does not represent processing time.Read diagram description
A request arrives and is scheduled. The prefill stage processes the whole prompt at once and is limited by arithmetic, then writes the KV cache. In disaggregated serving the cache is handed to a separate decode pool. Decode generates one token at a time, re-reading the weights and cache, and streams each token to the user, repeating for every output token.
NVIDIA's inference explainer describes the same prefill, decode and KV-cache mechanics. The DistServe paper (OSDI 2024) showed that running the two phases on the same GPUs makes them interfere: a long prompt arriving mid-stream stalls everyone's next token. Its authors assigned prefill and decode to different GPUs and reported serving many more requests within the same latency targets (author-reported results). The vLLM project ships this as an experimental feature, and its documentation is careful about what it buys: operators can set the first-token target and the gap between streamed tokens independently, but the feature "DOES NOT improve throughput".
The 2025 and 2026 hardware cycle, read through the split
Specs below are vendor-published, checked on 20 November 2026. Performance multiples in vendor announcements are company claims.
| Hardware | What the vendor documents | Which phase it targets |
|---|---|---|
| Google TPU7x (Ironwood), announced April 2025 | 192 GiB HBM and about 7.37 TB/s per chip; pods up to 9,216 chips; "the first designed specifically for inference" | Both, with memory sized for large models |
| AWS Trainium3, GA 2 December 2025 | 144 GB HBM3e and 4.9 TB/s per chip; up to 144 chips per UltraServer | Training and serving |
| NVIDIA Rubin CPX, announced September 2025 | "Purpose-built for massive-context processing", 128 GB of cheaper GDDR7; expected end of 2026 | Prefill on very long contexts |
| NVIDIA Vera Rubin NVL72 | 72 Rubin GPUs per rack; NVIDIA reported it "began shipping" in its results of 17 November 2026 | General serving and training |
| Groq 3 LPX in NVIDIA's March 2026 platform | 256 LPUs with 128 GB of on-chip SRAM per rack, to "boost decode" alongside Rubin GPUs | Low-latency decode |
| Cerebras wafer-scale systems | OpenAI agreed in January 2026 to deploy 750 MW, which OpenAI described as "a dedicated low-latency inference solution" | Low-latency decode |
The pattern is specialisation. Prompt processing wants cheap arithmetic and can live with slower memory, which is why a prefill chip can use GDDR7. Decode wants bandwidth, which is why decode-focused designs keep weights in on-chip SRAM. In December 2025 Groq announced a non-exclusive licence of its inference technology to NVIDIA, with its founder joining NVIDIA. The largest buyers are also contracting for capacity in gigawatts: OpenAI announced 6 GW of AMD GPUs and 10 GW of accelerators it designs with Broadcom, both starting in the second half of 2026. Power and supply, not only chip speed, now bound how much inference exists.
Which product decisions the hardware reaches
A PM who builds on an API buys tokens, not chips. The hardware still shows up in five places.
Speed tiers. Providers now sell faster output at a higher price. Anthropic lists a fast mode for Claude Fable 5.2 at twice the standard rate, and OpenAI renamed its priority tier to Fast mode in July 2026. A decode-specialised fleet is one way such tiers are served. Whether to pay depends on whether users wait for the full output, which is a product question, worked in the question linked below.
Long prompts are not free for latency. Prefill time grows with prompt length, and on shared serving pools long prompts can slow other users' streams. Caching a stable prefix (prompt caching) removes repeated prefill work, which is why caching cuts both cost and time to first token.
Background work gets cheaper. Work that can wait fills idle capacity between peaks. OpenAI's two-hour and eight-hour background windows, added in beta on 17 November 2026, cost 30% and 40% less. Classify every workload as interactive or deferrable before you price it.
Portability is a cost. Moving a self-hosted model from GPUs to TPUs or Trainium can cut cost, but kernels, quantization formats and serving stacks differ by hardware. Budget the engineering and re-run the evaluation; do not assume a model behaves identically after a hardware or precision change.
Capacity is a roadmap risk. When supply is tight, providers ration through rate limits, regions and commitments. Read GPU economics for PMs for the cost side and plan launch-day capacity as a dependency, not an afterthought.
In the interview
Hardware questions in platform and technical loops test whether you can connect a spec to a user-visible number. A weak answer quotes teraflops. A Hire answer knows time to first token and streaming speed are different. A Strong Hire answer says prefill is compute-bound and decode is bandwidth-bound, works out a rough tokens-per-second ceiling from bandwidth and model size, explains why batching and caching help, and then decides which workloads deserve a fast tier, which can wait for cheap background capacity and which need neither.
Next: GPU economics for PMs and serving and speculative decoding.