TL;DR: Before choosing a metric, a judge or a platform, read about a hundred real traces, write a free-text note on the first thing that went wrong in each, and group the notes into fewer than ten named failure modes. Rank those by frequency times cost. That list is your eval spec.
Why teams skip it, and what it costs
The usual sequence on a new AI feature goes like this. Someone proposes an eval platform. The platform ships with default metrics: helpfulness, relevance, coherence, toxicity. A dashboard appears, the numbers look reasonable, and for a while everyone feels the feature is measured.
Then users complain about something the dashboard does not show: the assistant promises delivery dates it cannot know, or answers in English when the user wrote in Hindi, or asks for an order number it already has. None of those are "helpfulness". They are specific to this product, and no generic metric was ever going to find them.
Error analysis reverses the order. You find out what goes wrong first, by looking, and then decide what to measure. It can start with a bounded review session; the investment matters because every later decision (which checks to write, which judge to build, what to put in the golden set) depends on it.
The method
1. Get traces. A trace is the full record of one interaction: the user input, any retrieved context, tool calls and results, intermediate model outputs and the final response. If the product does not log these, logging is the first task. You cannot evaluate what you cannot see.
2. Sample deliberately. Start with around 100 randomly selected recent traces as a review budget, not a guarantee of coverage. Add a smaller set of traces you have reason to suspect: conversations followed by a complaint, an escalation, a thumbs-down, or a repeated question. Random sampling can miss rare failures; suspect-only selection distorts estimates of how often things go wrong. Label each sample source and keep the denominators separate.
3. Write open notes. For each trace, write one or two sentences about the first thing that went wrong, if anything. Do not use a predefined checklist. The whole point is to notice failures nobody has named. "Gave refund policy for the wrong country." "Correct answer, but asked for the order ID twice." "Cited an article that doesn't mention the topic." Note only the first failure, since later errors often cascade from it.
4. Read with a domain expert. For at least the first twenty or thirty traces, sit with someone who knows what correct looks like: a support lead, a lawyer, a clinician, an accountant. They will spot policy and domain errors you would read straight past.
5. Cluster. Group the notes into categories. You can use a model to propose groupings once the notes exist, but you name and check the groups yourself. Let the observed failures determine the categories; five to eight is a workable starting size for this exercise, not a property of every product. Write a one-line definition for each, plus a passing and failing example.
6. Count and rank. Tally how many traces hit each mode, then weight by what each costs when it happens.
| Failure mode | Traces (of 100) | Cost when it happens | Priority |
|---|---|---|---|
| Asks for information it already has | 14 | Annoyance, slower resolution | Medium |
| Cites a source that doesn't support the claim | 6 | Trust damage, wrong action | High |
| Wrong policy applied (country, plan tier) | 4 | Refund errors, complaints | High |
| Replies in the wrong language | 5 | Abandoned conversation | Medium |
| Overly long answer | 11 | Mild | Low |
These counts are illustrative and refer to the random sample of 100, with additional complaint cases kept separate. Frequency alone would prioritize repeated questions; the policy errors may deserve earlier action because of their consequences.
7. Set the next review. When another batch adds no new categories, turn the current findings into checks. This is a stopping heuristic for exploratory work, not evidence that rare failures are covered. Continue sampling new languages, intents and policy changes; deliberate adversarial cases serve a different purpose from estimating a traffic-wide rate.
What you get out of it
Three things come directly from a finished error analysis.
A prioritised fix list. Some failure modes are fixable immediately with a prompt change, a retrieval fix or a missing tool. Fix those first; you do not need an eval to justify fixing an obvious bug.
An eval spec. Each remaining failure mode becomes a pass/fail check. Many can be code checks (language ID, schema validation, citation exists). The rest become narrow LLM judges, each written from the labelled examples and validated against them.
The first golden set. The annotated traces, especially the failing ones, become regression cases. Every future change runs against them.
Follow the flow, one stage at a time.
Illustrated sequence; animation speed does not represent processing time.Read diagram description
Collect complete traces, separate representative and diagnostic samples, inspect the first failure, review with a domain expert, name failure modes, assess frequency and consequence, write checks, then rerun and review new traces. Diagnostic cases cannot estimate a production failure rate.
Keep doing it
Error analysis is not a launch task. User behaviour changes, new features change what reaches the model, providers update models. A standing hour a week reading 20 to 30 fresh traces, especially from new segments or after a change, catches new failure modes before they show up in complaints. Teams that stop reading traces gradually end up with evals that measure last year's product.
In the interview
Our practice questions ask you to apply error analysis when "you inherit an AI feature with no evals", "how would you start evaluating X", or "quality dropped and you don't know why". Our teaching rubric distinguishes whether the candidate's first move is to read real outputs with a domain expert, or to name a tool, a metric or an LLM judge. Saying "I'd start by reading a hundred traces and writing down what goes wrong" and then explaining how those notes become checks makes the proposed evaluation specific enough to inspect.
Method reading: Hamel Husain’s account of building product evals discusses inspecting traces and choosing task-specific checks. It is practitioner guidance; the budgets and examples above are our teaching choices.
Try the method in the two-week evaluation case, then inspect the launch-readiness tool to connect measured quality to a product decision.