Google DeepMind Evals & Launch Decisions interview questions

95% agreement sounds like a yes. On its own the number says almost nothing, and the thirty-second arithmetic that proves it is what separates a launch gate from a dashboard.

Calculate the evidence needed for a small quality gain. Running both models on the same cases changes the test; the disagreement counts determine whether pairing saves evaluation work.

A larger labelled set can make a broken quality bar look more precise. Allocate a fixed review budget, resolve disputed criteria and decide which results can support a launch.

Accuracy alone rewards guessing. Set a penalty for wrong answers from what errors cost in your product, score both models under it, and the choice usually flips.

Rate each quality separately, on your users' own prompts, with side-by-side human comparisons. Use automated prompt-adherence checks as the regression gate and kept-image rate as the product outcome.

Benchmarks die from leakage, label errors and saturation. Hold back a private split, audit labels with experts, score abstention, watch the public-private gap and publish when you will retire it.

Decide which support tasks an agent can safely own. An 82% pass rate needs a denominator, a breakdown of failures and a comparison with today's service.

A RAG answer can fail in the retriever or in the generator, and one accuracy score cannot say which. The interviewer wants two evals, a production loop and a launch bar tied to what a wrong answer costs.

Explain what a passing test suite establishes, what a behavior evaluation adds and why one critical failure can still block an otherwise strong result.

The strongest first move costs nothing and involves no tooling: read the traces. What you do with the notes, and in what order, is the rest of the answer.

A quick question with a firm right answer. A 3.8 average feels like data, but nobody can tell you what to fix on Monday.

Everyone says 'build a golden set'. The interviewer wants to know what goes in it, where it comes from, who maintains it and how you stop it from rotting.

A diagnosis question with five plausible causes. Strong candidates rank them by likelihood and name the cheapest check for each instead of reciting all five.

Leaderboards move every few weeks and every team feels the pull. The interviewer is checking whether you have a repeatable switching process or just enthusiasm.

Agent evals are where answers that worked for chatbots stop working. A correct final booking can hide a path that broke policy, spent money twice or took nine minutes.

'Hallucination rate' sounds like one number. In practice it is three different failures, and the one that matters most is the hardest to see.

Candidates often say subjective quality cannot be measured, or reach for a 1-to-10 score. Both lose points. Taste can be measured; you just measure it differently.

Green dashboards and angry users at the same time. The strong answer treats the complaints as evidence about the eval, not noise to explain away.

A principal-level question because every simple answer is dangerous. The doctor signs the note, so the tempting conclusion is that errors get caught. The strongest candidates show why that assumption fails.

An eval nobody runs is worth nothing. This question tests whether you can do the cost arithmetic and redesign the workflow, rather than ordering people to comply.

A short question about roles that reveals whether a candidate sees evals as a product spec or as an engineering chore.