An eval nobody runs is worth nothing. This question tests whether you can do the cost arithmetic and redesign the workflow, rather than ordering people to comply.
Updated Nov 2026 · Original practice questions with worked answers and teaching rubrics. Our editorial policy explains the sources, review process and limits.
Schema checks would reject 10% of outputs before judging, and a judge at a tenth of the price matches the expensive one on two of the four criteria, validated against human labels.
A model upgrade lands. Against the current model, the full suite shows 41 newly failing and 63 newly passing cases out of 2,000, and the release is due tomorrow.
An eval nobody runs is worth nothing. This question tests whether you can do the cost arithmetic and redesign the workflow, rather than ordering people to comply.
This reference answer is in Premium. Free signup opens other answers.
9 answers per topic instead of 5, and your progress kept · no cardor unlock all 420 remaining answersNo comments yet. How would you have answered?