Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Read real traces before building anything. Spend the first week reading and annotating a hundred or so production conversations, cluster the notes into a handful of named failure modes, and only then write pass/fail checks for the two or three modes that cost the most.
How to approach it
Clarify what the feature does, how much traffic it gets and whether traces are logged at all. If nothing is logged, logging is day one. Then say plainly that you will not pick an eval tool or an LLM judge yet, and explain why: the data decides what to measure, and the tool comes after.
A strong answer
Use this fictional exercise: the feature is an assistant inside a food-delivery app that answers "where is my order", handles missing-item complaints and offers refunds within policy.
The size of the first set is smaller than people expect. Anthropic's agent eval guide (January 2026) says "20-50 simple tasks drawn from real failures is a great start." The reading also has to come before the rubric, not after: a study of how people build LLM graders found that "users need criteria to grade outputs, but grading outputs helps users define criteria" (Shankar et al., 2024). Criteria written before reading drift once real outputs arrive.
Days 1 to 5: read. Pull a random sample of about 100 recent conversations, plus every conversation that ended in a complaint or escalation in the same period. Read each one end to end with the support lead next to you for the first twenty, because they will spot policy errors you cannot. For each conversation write one free-text note about the first thing that went wrong, if anything. Do not use a predefined checklist. The point is to find failures nobody has named yet.
Days 6 to 7: cluster. Group the notes. Use the observed notes to define a manageable set of failure modes; do not force them into a predetermined count. In this example they might be:
| Failure mode | Share of 100 traces | Cost when it happens |
|---|---|---|
| Promises a delivery time the app does not show | 9 | Re-contact, low CSAT |
| Refunds outside policy (wrong amount or no proof) | 4 | Direct money loss |
| Fails to hand off when the user is angry | 7 | Escalation, churn risk |
| Answers in English when the user wrote in Hindi or Tamil | 6 | Abandoned chat |
| Correct but slow, asks for order ID it already has | 12 | Friction |
These are illustrative counts from the random sample of 100. Keep the complaint-only cases in a separate diagnostic set so they do not inflate the frequency estimate. Here, repeated requests for an ID occur more often, but refund-policy errors have a direct monetary consequence.
Days 8 to 10: write checks for what matters. Rank by frequency times cost. Refunds outside policy come first even at 4%, because each one loses money and policy compliance can be checked mostly in code (amount against order value, presence of a photo for missing items). Language mismatch is also cheap to check in code. "Failed to hand off" needs judgment, so it gets a binary LLM judge whose prompt is written from the annotated examples, and the judge is checked against the support lead's labels before anyone trusts it.
Days 11 to 14: make it repeatable. Turn the annotated traces into the first golden set, with the failing ones kept as regression cases. Run it before every prompt or model change. Set up a weekly hour to read 30 new traces, because new failure modes appear whenever the product or the user mix changes.
The deliverable is a reviewed failure list, checks for costly mistakes and an assigned review cadence. Report rates only for samples whose selection supports that interpretation.
Follow the flow, one stage at a time.
Illustrated sequence; animation speed does not represent processing time.Read diagram description
In this illustrative plan, establish trace access, separate routine and complaint samples, read with support, classify failures, prioritize costly mistakes, implement narrow checks, test graders against labels and schedule regression and fresh-trace review. New failures update the checks.
What interviewers probe next
"Engineering wants to adopt an eval platform first. What do you say?" Fine after we know what to measure. A platform before error analysis produces generic scores like "helpfulness" that nobody can act on. OpenAI's own best-practice guide lists "vibe-based evals" and "not calibrating your automated metrics against human evals" among the anti-patterns; the reading is how you avoid both.
"There are 50,000 conversations a day. Is 100 enough?" It is a starting review budget. Inspect coverage by intent, language and severity, and expand if new modes keep appearing. Complaint and rare-intent cases help diagnosis; a representative sampling plan is needed for a production rate.
"How do you know when you have read enough?" When successive batches stop changing the taxonomy, turn the current findings into checks. Keep reviewing new segments and incidents; saturation in this sample does not establish rare-failure coverage.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| First move | Picks a tool or metric | Proposes reviewing outputs | Reads ~100 real traces with a domain expert and writes open notes |
| Prioritisation | Fixes the most frequent issue | Considers impact | Ranks failure modes by frequency times cost and starts with the cheapest reliable check |
| Durability | One-off audit | Builds a test set | Golden set with regression cases plus a standing review cadence |
Common mistakes
Starting with generic criteria such as helpfulness, coherence and tone. They come from a blog post, not from this product's failures.
Asking an LLM to do the first pass of error analysis. You will get the model's idea of what can go wrong, not what did.
Reading only escalated conversations. They over-represent angry users and miss the confident wrong answers nobody complained about.
The method is laid out in error analysis first.