Anthropic Evals & Launch Decisions interview questions

Decide which support tasks an agent can safely own. An 82% pass rate needs a denominator, a breakdown of failures and a comparison with today's service.

Explain what a passing test suite establishes, what a behavior evaluation adds and why one critical failure can still block an otherwise strong result.

The strongest first move costs nothing and involves no tooling: read the traces. What you do with the notes, and in what order, is the rest of the answer.

95% agreement sounds like a yes. On its own the number says almost nothing, and the thirty-second arithmetic that proves it is what separates a launch gate from a dashboard.

Calculate the evidence needed for a small quality gain. Running both models on the same cases changes the test; the disagreement counts determine whether pairing saves evaluation work.

A short question about roles that reveals whether a candidate sees evals as a product spec or as an engineering chore.

Two teams pulling in opposite directions, each with a real point. The strong answer gives them one shared measurement so the argument becomes a decision.

A larger labelled set can make a broken quality bar look more precise. Allocate a fixed review budget, resolve disputed criteria and decide which results can support a launch.

Single-turn test sets miss what breaks over a conversation: forgotten constraints, contradictions and drift. Build conversation-level cases, simulated users with goals, and session metrics.

The model stayed the same, some runs disappeared and the grader changed. Reconcile the report before deciding what the result supports.

Synthetic cases add coverage cheaply and inherit the generator's blind spots. Use them for named slices and adversarial variants, keep real traffic as the launch gate, and measure the gap between the two.

When a measure becomes a target, teams tune to it. Take eval scores out of targets, keep a sealed holdout teams cannot tune against, separate capability from regression evals, and tie OKRs to user outcomes.

A new judge is a new ruler. Run old and new judges in parallel on a human-labelled set, re-validate catch rates, mark the break in the trend line and pin judge versions from now on.

A RAG answer can fail in the retriever or in the generator, and one accuracy score cannot say which. The interviewer wants two evals, a production loop and a launch bar tied to what a wrong answer costs.

A quick question with a firm right answer. A 3.8 average feels like data, but nobody can tell you what to fix on Monday.

Everyone says 'build a golden set'. The interviewer wants to know what goes in it, where it comes from, who maintains it and how you stop it from rotting.

A diagnosis question with five plausible causes. Strong candidates rank them by likelihood and name the cheapest check for each instead of reciting all five.

Leaderboards move every few weeks and every team feels the pull. The interviewer is checking whether you have a repeatable switching process or just enthusiasm.

Agent evals are where answers that worked for chatbots stop working. A correct final booking can hide a path that broke policy, spent money twice or took nine minutes.