TL;DR: A language model repeatedly predicts how likely each possible next token is and picks one, usually with some randomness controlled by temperature. Serving has two phases: prefill processes the whole prompt at once, decode produces output token by token. Sampling explains why answers vary; prefill explains why long prompts delay the first word; decode explains why long answers are slow and why output costs more.
Next-token prediction
For autoregressive text generation, the model estimates a distribution over the next token given the preceding tokens. The full product may also retrieve records, call tools or process other input types. "The capital of France is" puts very high probability on "Paris". The model then appends a chosen token and repeats, building the answer one token at a time.
This explains several behaviours product teams encounter:
- Fluency without verification. The model learns what plausible text looks like. A plausible continuation can be unsupported even on a familiar topic. Missing evidence is one cause; misreading available evidence is another.
- Sensitivity to context. Everything in the prompt shapes the probabilities: instructions, examples, retrieved documents, even formatting. Small prompt changes can change answers.
- Early choices matter. Once a token is chosen, later tokens build on it. A wrong turn early in an answer tends to persist.
Sampling and temperature
With fixed computations and inputs, greedy decoding selects the highest-scoring token at each step. A hosted service may still vary because its inputs, model version or numerical execution differ. Instead it usually samples: probable tokens are chosen most often, less probable ones occasionally.
Temperature controls how adventurous sampling is. For positive temperature, scaling the token scores before normalization changes how concentrated the sampling distribution is. Lower values concentrate it; higher values flatten it. Neither setting establishes correctness. Related settings (top-p, top-k) limit which tokens are eligible at all.
| Task | Variation wanted | Typical setting, where the API exposes one |
|---|---|---|
| Brainstorming, creative drafts | Yes | Higher temperature, several samples |
| Conversational answers | A little | Moderate |
| Factual answers from sources | No | Low |
| Extraction, classification, structured output | No | Low or zero, with schema constraints |
Where an API supports temperature 0, it commonly uses greedy selection; temperature 2 produces a flatter distribution than temperature 1 for the same scores. Two is not “twice as creative.” Supported ranges and recommended defaults vary by model. Google's generation reference, checked in October 2026, explicitly does not guarantee deterministic output. Some models constrain or omit this parameter, so test the selected endpoint rather than prescribing one universal setting. Anthropic's Messages API version 2026-10-21 rejects temperature, top_p and top_k outright, and Claude 4.7 and later models already accepted only the defaults (migration guide, checked 6 November 2026). On an endpoint like that, the table above is not a lever you have.
Even at low temperature, outputs can vary because prompts vary slightly between calls (different retrieved passages, different history) and because serving systems are not always perfectly deterministic. Consistency for factual and structured tasks is designed through grounding, schemas and moving decisions into code, not achieved by temperature alone.
Prefill and decode
Serving a request has two phases with different performance characteristics.
Prefill. The model processes the entire prompt in one pass. Work grows with prompt length. For long prompts, prefill dominates the time to first token. Prompt caching lets the server reuse work for a repeated prefix, which speeds up prefill and reduces its cost.
Decode. The model generates output tokens one at a time, each step depending on everything before. Speed is measured in tokens per second. Decode is the slower phase per token, which is part of why providers price output tokens higher than input tokens.
During decode, the server holds information about the processed prompt in memory (the KV cache). Long contexts consume more memory, which limits how many requests a server can handle concurrently, one reason long-context requests cost more to serve.
Product consequences
| Observation | Cause | Design response |
|---|---|---|
| First word slow on long prompts | Prefill | Trim prompts, cache stable prefixes, retrieve fewer chunks |
| Long answers slow and expensive | Decode | Ask for concise answers, structured output, stream |
| Different answers to the same question | Sampling and prompt variation | Ground in sources, schemas, pin model versions; low temperature only where offered |
| Confident wrong answers | Prediction of plausible text | Retrieval, abstention, verification, citations |
| Performance drops with very long context | Attention over long inputs, memory load | Summarise, retrieve selectively |
Stopping
Generation ends when the model produces a stop token, reaches a maximum length set by the application, or hits a stop sequence. Setting a sensible maximum output length is a cost and latency control. Truncated answers, however, are a poor experience, so length limits work best combined with instructions to be concise.
In the interview
For a PM explanation, connect the mechanism to a decision. Be able to explain why outputs vary and how to control it, why long prompts slow the first token and long answers slow completion, and why output tokens cost more. Connecting each mechanism to a product decision (temperature by task, prompt trimming and caching, answer length, grounding) is what scores.
For the layers beneath prediction, read transformers, attention and weights.