The model stayed the same, some runs disappeared and the grader changed. Reconcile the report before deciding what the result supports.
Updated Nov 2026 · Original practice questions with worked answers and teaching rubrics. Our editorial policy explains the sources, review process and limits.
An internal agent report claims a quality gain after an evaluation-runner upgrade. These fictional runs use the same task IDs, reset environments, model weights and application prompt, with one attempt per task.
| Scheduled tasks | 100 before; 100 after |
|---|---|
| Scored successes | 72 before; 72 after |
| Scored failures | 28 before; 8 after |
| Infrastructure errors, no score | 0 before; 15 after |
| Missing runs, no artifact | 0 before; 5 after |
| Grader change | Before: checks final database state. After: checks the assistant's completion message. |
Your task. Decide whether to announce the improvement. Reconstruct the report's denominators, identify what evidence is missing and propose the comparison needed for a release decision.
The model stayed the same, some runs disappeared and the grader changed. Reconcile the report before deciding what the result supports.
This reference answer is in Premium. Free signup opens other answers.
9 answers per topic instead of 5, and your progress kept · no cardor unlock all 420 remaining answersNo comments yet. How would you have answered?