evals

A classic AI product sense prompt. The trap is a chat box bolted onto a grid. The strong answer picks a user, runs every job through an AI-fit test, and treats a silently wrong number as the main risk.

Often the first question in an AI PM loop. Generic answers ('AI is uncertain') score poorly; concrete changes to specs, testing, cost and launches score well.

The most common story in AI product development. A demo proves the model can do something; a product needs it to do that thing reliably on inputs nobody chose. The strong answer separates those milestones and resets expectations without losing the sponsor.

A craft question that separates PMs who have shipped AI features from those who have read about them. The answer is a document organised around failure modes, thresholds and evidence.

Explicit ratings are rare, biased and vague. The strongest feedback signals are things users already do. The interviewer wants a design that captures them and turns a few explicit reports into labelled examples.

Fine-tuning is sometimes the best decision a team makes and often a costly detour. The interviewer wants the test that tells them apart, with numbers.

Everyone knows models make things up. The interviewer wants to hear why, which kinds of hallucination exist, and a layered set of product defences, without promising to eliminate it.

A principal-level question that combines technical fluency, skepticism and business judgment. The two claims in the prompt are exactly the ones to test hardest.

A deadline you did not choose and a change that touches every output. The interviewer wants a plan that treats API compatibility and behavioural compatibility as different problems.

A platform design question about the tooling that decides launches. The hard part is not running prompts at scale; it is making error analysis, labelling and judge validation easy enough that teams actually do them.

Research agents are judged on whether an analyst can trust and reuse the output. The design centres on source quality, verifiable citations and showing the plan before spending ten minutes.

Decide which support tasks an agent can safely own. An 82% pass rate needs a denominator, a breakdown of failures and a comparison with today's service.

A RAG answer can fail in the retriever or in the generator, and one accuracy score cannot say which. The interviewer wants two evals, a production loop and a launch bar tied to what a wrong answer costs.

Explain what a passing test suite establishes, what a behavior evaluation adds and why one critical failure can still block an otherwise strong result.

The strongest first move costs nothing and involves no tooling: read the traces. What you do with the notes, and in what order, is the rest of the answer.

95% agreement sounds like a yes. On its own the number says almost nothing, and the thirty-second arithmetic that proves it is what separates a launch gate from a dashboard.

A quick question with a firm right answer. A 3.8 average feels like data, but nobody can tell you what to fix on Monday.

Everyone says 'build a golden set'. The interviewer wants to know what goes in it, where it comes from, who maintains it and how you stop it from rotting.

A diagnosis question with five plausible causes. Strong candidates rank them by likelihood and name the cheapest check for each instead of reciting all five.

Calculate the evidence needed for a small quality gain. Running both models on the same cases changes the test; the disagreement counts determine whether pairing saves evaluation work.

Leaderboards move every few weeks and every team feels the pull. The interviewer is checking whether you have a repeatable switching process or just enthusiasm.

Agent evals are where answers that worked for chatbots stop working. A correct final booking can hide a path that broke policy, spent money twice or took nine minutes.

'Hallucination rate' sounds like one number. In practice it is three different failures, and the one that matters most is the hardest to see.

Candidates often say subjective quality cannot be measured, or reach for a 1-to-10 score. Both lose points. Taste can be measured; you just measure it differently.

Green dashboards and angry users at the same time. The strong answer treats the complaints as evidence about the eval, not noise to explain away.

A principal-level question because every simple answer is dangerous. The doctor signs the note, so the tempting conclusion is that errors get caught. The strongest candidates show why that assumption fails.

An eval nobody runs is worth nothing. This question tests whether you can do the cost arithmetic and redesign the workflow, rather than ordering people to comply.

A short question about roles that reveals whether a candidate sees evals as a product spec or as an engineering chore.

A translated English eval set can miss code-switching, transliteration and local support needs. Decide what to sample in each language and who can judge the replies.

A portfolio question. Standardise too much and teams stop evaluating what matters to their users; too little and nobody can compare quality, swap models or answer an auditor.

A larger labelled set can make a broken quality bar look more precise. Allocate a fixed review budget, resolve disputed criteria and decide which results can support a launch.

Routing is one of the biggest cost levers in AI products, and one of the easiest to get subtly wrong. The interviewer wants the savings arithmetic and the misrouting risk in the same answer.

A global preference win can conceal a regression for the people doing the hardest work. Reweight the evidence, inspect the behavior change and choose a release policy.

Rehearse a 40-minute build with time reserved for testing. Choose a small user task, make it work and explain the failures you found before proposing a launch.

A worked live-round walkthrough. The trap is building a pretty summariser. The scoring path is a narrow extractor with structured output, a small test set that probes the hard cases, and an honest readout.

The part of the prototyping round that separates candidates. Ten minutes is enough for a small, sharp evaluation if you choose the cases well and grade them pass or fail.

Opinions about prompts are cheap. A small grid of prompts by models on fifty real examples turns a debate into a table, and the interviewer wants to see you build it.

The handoff is where many AI prototypes die or get rebuilt wrongly. The interviewer wants the artefacts that carry learning forward and a clear list of what the prototype did not prove.

Agent prompts tempt candidates into ambitious builds that collapse. The strong answer builds the least autonomous version that demonstrates the idea, with mocked tools, clear limits and tests of the hard paths.

RAG is the most common live-build topic. Strong candidates choose the simplest approach that works for 30 documents, make citations real, and test retrieval and answers separately.

An exact Given/When/Then cannot test a model that words things differently every time. The scored answer shows which criteria stay exact and how the rest become measurable.

A behavioural question with a technical core: how you turned a readiness argument into agreed criteria and shared evidence, what you were wrong about, and whether the relationship survived.