Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Models generate text by repeatedly choosing the next token from a probability distribution, and that choice usually involves some randomness controlled by settings like temperature. Variation is useful for creative tasks and harmful for factual or structured ones. Match the setting to the job, ground factual answers in data, constrain formats, and evaluate consistency explicitly.
How to approach it
Explain generation in plain terms, name temperature, then distinguish where variability helps from where it hurts. End with practical product measures, because "set temperature to zero" alone does not solve consistency.
A strong answer
How generation works. At each step, the model computes how likely every possible next token is, given everything so far. Then it picks one. If inputs and numerical computation were identical, greedy selection would follow the same path. A production service may not provide those conditions. Usually it samples instead: likely tokens are chosen most often, but less likely ones sometimes. Temperature controls how adventurous that sampling is: low temperature sticks close to the most likely choices; higher temperature spreads choices out, producing more varied and sometimes more creative text.
Because each choice shapes the next, a single different token early on can lead to a noticeably different answer.
Read diagram description
Two illustrative next-token distributions after the words Your refund has been. At low temperature approved takes 72 percent, processed 24, denied 3 and issued 2. At high temperature the choices spread out: approved 43, processed 30, denied 14 and issued 12. A sampled word should never decide whether a refund is approved; that belongs in code.
When variation helps and hurts.
| Task | Variation | Setting, where the API exposes one |
|---|---|---|
| Brainstorming names, marketing copy | Helpful: users want options | Higher temperature, several drafts |
| Writing assistance | Mildly helpful | Moderate |
| Factual answers from documents | Harmful: users expect the same answer | Low temperature, grounded in retrieval |
| Extracting fields, classification, JSON | Harmful: must be consistent | Low or zero temperature, structured output |
| Decisions with consequences | Harmful | Should not depend on sampling at all; use code |
Why low temperature is not the whole answer. Even at low temperature, small changes in the prompt (a different retrieved document, an extra line of history) change the output. Temperature 0 does nothing for factual accuracy, and temperature 2, where supported, flattens the distribution toward unlikely tokens. Providers update models. And serving itself varies: Thinking Machines Lab reported in September 2025 that 1,000 requests for the same prompt at temperature 0 to a large open model on a standard inference engine produced 80 different completions, identical for the first 102 tokens and then splitting. The main cause was server load: the batch a request lands in changes with traffic, and the arithmetic changes with the batch. Kernels written to give the same result whatever the batch made all 1,000 identical, at a speed cost (company-reported). So the same user, asking the same question at a busy hour, can get a different answer from an endpoint set to zero. And some APIs no longer offer the dial at all: Anthropic's Messages API version 2026-10-21 rejects temperature, top_p and top_k (migration guide, checked 6 November 2026). So consistency has to be designed:
- Ground facts in data. Retrieve the current authorized policy and check that the response preserves it. Retrieval does not guarantee faithful use of the document.
- Constrain formats. Structured output schemas constrain shape and permitted values; a valid field can still contain the wrong fact.
- Move decisions to code. Whether a refund is allowed should be a rule in code; a sampled sentence can only report it.
- Pin model versions so behaviour changes only when you decide.
- Measure consistency. Run the same inputs several times as part of evaluation and track how often the answers differ materially.
What to tell the stakeholder. "Some variation is by design and fine for creative features. For factual answers, we ground the model in our documents and use settings that keep answers consistent, and we test for it."
What interviewers probe next
"A customer got two different answers about the refund policy. Which fix comes first?" Make sure policy answers come from the retrieved policy text instead of the model's memory, then check the retrieval returned the same document both times.
"Should we offer users a 'regenerate' button?" For creative tasks, yes. For factual answers, regenerate signals the first answer may not be trustworthy, so prefer showing sources instead.
How it's scored
Weak: Says the model is "random" or "unreliable" without explaining why.
Hire: Explains sampling and temperature and matches settings to task types.
Strong Hire: Adds that consistency must be designed (grounding, structured output, decisions in code, version pinning) and measured in evaluation, and knows that serving load alone can change outputs at temperature 0.
Common mistakes
Believing temperature zero guarantees identical answers.
Using high variation for factual or structured tasks.
Letting consequential decisions depend on sampled text.
See how LLMs generate.