Meta Evals & Launch Decisions interview questions

A diagnosis question with five plausible causes. Strong candidates rank them by likelihood and name the cheapest check for each instead of reciting all five.

A short question about roles that reveals whether a candidate sees evals as a product spec or as an engineering chore.

Two teams pulling in opposite directions, each with a real point. The strong answer gives them one shared measurement so the argument becomes a decision.

A classic experiment-design question with AI-specific traps: novelty that fades, quality that differs by segment, and a cost line that classic A/B tests never had.

At this scale, judging every conversation with your best model costs more than some teams' whole inference bill. The answer is a sampling design with arithmetic behind every layer.

Audit a judge's 68% win rate with an order-swap table. Separate stable wins from unresolved verdicts, declare the denominator, and test whether the result supports the product decision.

When a measure becomes a target, teams tune to it. Take eval scores out of targets, keep a sealed holdout teams cannot tune against, separate capability from regression evals, and tie OKRs to user outcomes.

Compare the same request with and without the profile, judged by people who see the profile, and test the three failures users notice: ignoring a stated preference, using a stale one and over-personalising.

Decide which support tasks an agent can safely own. An 82% pass rate needs a denominator, a breakdown of failures and a comparison with today's service.

A RAG answer can fail in the retriever or in the generator, and one accuracy score cannot say which. The interviewer wants two evals, a production loop and a launch bar tied to what a wrong answer costs.

Explain what a passing test suite establishes, what a behavior evaluation adds and why one critical failure can still block an otherwise strong result.

The strongest first move costs nothing and involves no tooling: read the traces. What you do with the notes, and in what order, is the rest of the answer.

95% agreement sounds like a yes. On its own the number says almost nothing, and the thirty-second arithmetic that proves it is what separates a launch gate from a dashboard.

A quick question with a firm right answer. A 3.8 average feels like data, but nobody can tell you what to fix on Monday.

Everyone says 'build a golden set'. The interviewer wants to know what goes in it, where it comes from, who maintains it and how you stop it from rotting.

Calculate the evidence needed for a small quality gain. Running both models on the same cases changes the test; the disagreement counts determine whether pairing saves evaluation work.

Leaderboards move every few weeks and every team feels the pull. The interviewer is checking whether you have a repeatable switching process or just enthusiasm.

Agent evals are where answers that worked for chatbots stop working. A correct final booking can hide a path that broke policy, spent money twice or took nine minutes.

'Hallucination rate' sounds like one number. In practice it is three different failures, and the one that matters most is the hardest to see.

Candidates often say subjective quality cannot be measured, or reach for a 1-to-10 score. Both lose points. Taste can be measured; you just measure it differently.

Green dashboards and angry users at the same time. The strong answer treats the complaints as evidence about the eval, not noise to explain away.

A principal-level question because every simple answer is dangerous. The doctor signs the note, so the tempting conclusion is that errors get caught. The strongest candidates show why that assumption fails.

An eval nobody runs is worth nothing. This question tests whether you can do the cost arithmetic and redesign the workflow, rather than ordering people to comply.