Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Quantization reduces the numeric precision of model weights (and sometimes activations) to cut memory, cost and latency. On Llama 3.1, published research finds 8-bit floating point effectively lossless and 4-bit weight-only quantization competitive with 8-bit, with the cheaper format depending on how the model is served; Meta found benchmarks can miss rare corrupted outputs. Decide on evaluations of your own workloads, by task and language, and roll out gradually with monitoring; consider 8-bit first if 4-bit loses quality where it matters.
How to approach it
Explain what quantization does, what published evidence says, why averages are insufficient, how you would evaluate, and how you would roll out. Tie it to economics.
A strong answer
What it is. Models store weights as numbers; standard serving often uses 16-bit formats. Quantization represents weights with fewer bits (8 or 4), shrinking memory and speeding computation. Smaller memory footprint means fewer GPUs per model replica or more requests per GPU, which cuts cost per query.
What the evidence says. Kurtic et al.'s "Give Me BF16 or Give Me Death"? (version 4, May 2026) ran more than 500,000 evaluations on Llama 3.1 models. It found FP8 (W8A8-FP) effectively lossless across model scales, well-tuned INT8 within 1 to 3% accuracy degradation, and 4-bit weight-only INT4 (W4A16) more competitive than expected. The cost winner depends on how you serve: W4A16 was the most cost-efficient for synchronous serving, while W8A8 led under asynchronous continuous batching. So "4-bit cuts cost by half" is a claim about one serving pattern, and the question back to infrastructure is which pattern our traffic follows.
Meta's Llama 3 paper adds the warning a PM needs most. When Meta served Llama 3 405B in FP8, standard benchmarks suggested FP8 matched BF16 even without mitigations, yet without an upper bound on the dynamic scaling factors the model occasionally produced corrupted responses while benchmark scores stayed strong. Meta also left the first and last Transformer layers and the self-attention parameters unquantized. It judged the final version by comparing reward-model score distributions across 100,000 responses from each precision. Replicate on your model and traffic before putting savings into a forecast; a benchmark average cannot establish your quality floor.
Why averages are not enough. "99% recovery on average" can hide larger drops on specific capabilities: long-context retrieval, mathematical reasoning, code, non-English languages, or rare tasks. Your customers' workloads may concentrate exactly there.
Evaluation plan.
| Dimension | What to test |
|---|---|
| Task segments | Your top customer use cases: summarisation, extraction, code, reasoning, chat |
| Languages | Each major language customers use |
| Long context | Retrieval and reasoning at long input lengths |
| Output format | Structured output reliability |
| Safety | Refusal behaviour and policy adherence |
| Latency and throughput | Under realistic batch sizes and load |
Compare paired outputs between full-precision and quantized versions, and read the cases that broke as well as the averages. Meta's method is worth copying: compare the distribution of scores across a large sample of real responses, so rare corrupted outputs show up in the tail.
Decision options.
- 4-bit everywhere if no segment regresses beyond agreed thresholds.
- 8-bit if 4-bit loses quality on important segments; still a meaningful saving.
- Mixed: quantized for high-volume, simpler workloads (a cheaper tier), full precision for premium or sensitive workloads.
Rollout. Shadow traffic, then a canary on a share of requests with monitoring of quality signals (customer complaints, regression reports, judge scores), then gradual ramp. Communicate if behaviour changes noticeably for customers.
Economics. If 4-bit halves GPU requirements for a model, serving cost per query drops substantially, which can fund a price cut, a new cheaper tier, or margin. Present the saving and the quality evidence together.
What interviewers probe next
"Quality drops 3 points on Hindi but nothing else." That segment's users would get a worse product. Keep higher precision for those requests or fix with targeted calibration data before rolling out.
"Does quantization affect safety?" It can shift refusal behaviour; include safety evaluations in the comparison.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Understanding | "Makes the model smaller" | Precision, memory and cost | Explains mechanism and cites evidence with caveats: format depends on serving pattern, benchmarks can miss rare corrupted outputs |
| Evaluation | Benchmark average | Own evals | Segmented by task, language, long context, format and safety, paired comparisons |
| Decision | Accept or reject | Rollout plan | Options including 8-bit and mixed serving, canary with monitoring, economics framed with quality |
Common mistakes
Accepting average recovery figures without testing your workloads.
Ignoring languages and long-context tasks where losses concentrate.
Rolling out everywhere at once.
See quantization for PMs.