AI Evals & Launch Decision Interview Questions

Decide which support tasks an agent can safely own. An 82% pass rate needs a denominator, a breakdown of failures and a comparison with today's service.

A RAG answer can fail in the retriever or in the generator, and one accuracy score cannot say which. The interviewer wants two evals, a production loop and a launch bar tied to what a wrong answer costs.

Explain what a passing test suite establishes, what a behavior evaluation adds and why one critical failure can still block an otherwise strong result.

The strongest first move costs nothing and involves no tooling: read the traces. What you do with the notes, and in what order, is the rest of the answer.

95% agreement sounds like a yes. On its own the number says almost nothing, and the thirty-second arithmetic that proves it is what separates a launch gate from a dashboard.

A quick question with a firm right answer. A 3.8 average feels like data, but nobody can tell you what to fix on Monday.

Everyone says 'build a golden set'. The interviewer wants to know what goes in it, where it comes from, who maintains it and how you stop it from rotting.

A diagnosis question with five plausible causes. Strong candidates rank them by likelihood and name the cheapest check for each instead of reciting all five.

Calculate the evidence needed for a small quality gain. Running both models on the same cases changes the test; the disagreement counts determine whether pairing saves evaluation work.

Leaderboards move every few weeks and every team feels the pull. The interviewer is checking whether you have a repeatable switching process or just enthusiasm.

Agent evals are where answers that worked for chatbots stop working. A correct final booking can hide a path that broke policy, spent money twice or took nine minutes.

'Hallucination rate' sounds like one number. In practice it is three different failures, and the one that matters most is the hardest to see.

Candidates often say subjective quality cannot be measured, or reach for a 1-to-10 score. Both lose points. Taste can be measured; you just measure it differently.

Green dashboards and angry users at the same time. The strong answer treats the complaints as evidence about the eval, not noise to explain away.

A principal-level question because every simple answer is dangerous. The doctor signs the note, so the tempting conclusion is that errors get caught. The strongest candidates show why that assumption fails.

An eval nobody runs is worth nothing. This question tests whether you can do the cost arithmetic and redesign the workflow, rather than ordering people to comply.

A short question about roles that reveals whether a candidate sees evals as a product spec or as an engineering chore.

A translated English eval set can miss code-switching, transliteration and local support needs. Decide what to sample in each language and who can judge the replies.

Two teams pulling in opposite directions, each with a real point. The strong answer gives them one shared measurement so the argument becomes a decision.

A classic experiment-design question with AI-specific traps: novelty that fades, quality that differs by segment, and a cost line that classic A/B tests never had.

A portfolio question. Standardise too much and teams stop evaluating what matters to their users; too little and nobody can compare quality, swap models or answer an auditor.

At this scale, judging every conversation with your best model costs more than some teams' whole inference bill. The answer is a sampling design with arithmetic behind every layer.

A larger labelled set can make a broken quality bar look more precise. Allocate a fixed review budget, resolve disputed criteria and decide which results can support a launch.

Choose an operating threshold using prevalence, error costs and review capacity. A good ranking score can still produce an unusable queue.

Compare ranking preferences efficiently without mistaking a click preference for a retention or revenue result.

Separate reference overlap from faithful meaning. Build a decision from claim errors, omissions and the reader's task.

Single-turn test sets miss what breaks over a conversation: forgotten constraints, contradictions and drift. Build conversation-level cases, simulated users with goals, and session metrics.

A voice agent can fail at hearing, understanding, deciding or speaking. Evaluate each stage, then whole calls, segmented by language, accent and line quality.

The model stayed the same, some runs disappeared and the grader changed. Reconcile the report before deciding what the result supports.

Audit a judge's 68% win rate with an order-swap table. Separate stable wins from unresolved verdicts, declare the denominator, and test whether the result supports the product decision.

Synthetic cases add coverage cheaply and inherit the generator's blind spots. Use them for named slices and adversarial variants, keep real traffic as the launch gate, and measure the gap between the two.

Checking an A/B test early and stopping when it looks good inflates false positives. Run to the planned horizon unless the test was designed for early stopping, and stop early only for harm.

When a measure becomes a target, teams tune to it. Take eval scores out of targets, keep a sealed holdout teams cannot tune against, separate capability from regression evals, and tie OKRs to user outcomes.

Compare the same request with and without the profile, judged by people who see the profile, and test the three failures users notice: ignoring a stated preference, using a stale one and over-personalising.

A new judge is a new ruler. Run old and new judges in parallel on a human-labelled set, re-validate catch rates, mark the break in the trend line and pin judge versions from now on.

Accuracy alone rewards guessing. Set a penalty for wrong answers from what errors cost in your product, score both models under it, and the choice usually flips.

Rate each quality separately, on your users' own prompts, with side-by-side human comparisons. Use automated prompt-adherence checks as the regression gate and kept-image rate as the product outcome.

Short-answer benchmarks say little about ten-page reports. Check each statement against the source it cites, score coverage against an expert outline, test balance on contested topics and measure whether users act without redoing the work.

Benchmarks die from leakage, label errors and saturation. Hold back a private split, audit labels with experts, score abstention, watch the public-private gap and publish when you will retire it.

Published systems measure whether comments lead to code changes, aim for a precision target and filter non-actionable comments. Compare against human comments, seed known bugs for recall and run a randomised rollout.