Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Build a prompt matrix: take about 50 real examples with expected outcomes, run each candidate prompt on each candidate model, grade outputs pass or fail per failure mode, and compare pass rate, cost and latency in one table. Read the failures, not just the totals. Choose the cheapest combination that meets the bar on the cases that matter most.
How to approach it
Agree the success criteria first, then assemble examples, then run the grid, then decide. Keep it small enough to finish in an afternoon.
A strong answer
1. Criteria. For a feature that extracts order issues from customer messages: correct issue category, correct order ID, no invented details.
2. Examples. Pull 50 real messages (de-identified), including the hard ones: multiple issues, missing order IDs, code-mixed Hindi and English, sarcasm. Label the expected output for each.
3. The grid. Three prompts (the current one, a more detailed one with examples, a shorter structured-output one) by three models (a small, a mid and a large tier):
| Prompt × model | Category correct | Order ID correct | Invented details | Cost per 1,000 | p95 latency |
|---|---|---|---|---|---|
| A × small | 82% | 90% | 4 | lowest | fastest |
| B × small | 88% | 94% | 1 | low | fast |
| C × small | 86% | 96% | 0 | lowest | fastest |
| B × mid | 94% | 96% | 0 | medium | medium |
| C × mid | 92% | 98% | 0 | medium | medium |
| B × large | 95% | 96% | 0 | high | slow |
(Illustrative numbers.)
4. Read the failures. Where do the small-model combinations fail? If most failures are on code-mixed messages, that is a segment question, not a general one. Maybe the small model with prompt C is good enough for English and the mid model is needed for code-mixed text.
5. Decide. Pick the cheapest combination that meets the bar on high-impact cases. Here, perhaps prompt C on the mid model, or routing: small model for English, mid model for code-mixed messages. Document the decision and the evidence.
6. Keep it. The 50 examples become the start of a regression set. When a new model is released, rerun the grid in minutes.
Follow the flow, one stage at a time.
Illustrated sequence; animation speed does not represent processing time.Read diagram description
Agree pass criteria, assemble 50 real labelled examples including hard cases, run three prompts on three model tiers, tabulate pass rate per criterion with cost and latency, read the failures by segment, choose the cheapest combination that meets the bar or route by segment, and keep the examples as a regression set. When a new model ships, rerun the grid.
Tools. A notebook or a small script is enough: loop over examples, prompts and models, store outputs, grade with simple code checks plus quick manual review or a validated judge.
What interviewers probe next
"Fifty examples isn't enough to separate 94% from 92%." Agreed, and the arithmetic shows why: on 50 cases that is 47 against 46 correct, one example, with 95% intervals of roughly 84% to 98% and 81% to 97%. The matrix shows big differences and failure patterns. For close calls, expand the set and use a paired comparison on the same cases, which, as Anthropic's eval statistics guidance puts it, eliminates "the variance in question difficulty".
"Why test several prompts at all? The wording barely matters." It matters more than people expect. One study found formatting changes alone moved accuracy by up to 76 points on an open model (Sclar et al., ICLR 2024). That is the reason the matrix varies prompts and models together, and why the winning prompt is pinned in version control.
"The engineers prefer prompt B because it's more readable." Readability matters for maintenance; if B and C perform similarly, that is a fine tiebreaker.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Method | Opinions or a few tries | Test set | Grid of prompts by models on real labelled examples with defined criteria |
| Analysis | Overall winner | Pass rates and cost | Per-criterion results, cost and latency, failures read by segment |
| Decision | Best score | Cheapest acceptable | Cheapest meeting the bar on high-impact cases, routing option, kept as regression set |
Common mistakes
Comparing prompts on a handful of hand-picked examples.
Choosing the top score without looking at cost or latency.
Throwing away the examples after the decision.
See prompt-matrix evals.