Our agent score rose from 72% to 90% after a runner upgrade. Can we announce the improvement?
Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.
Case briefRead the inputs, then make your call
An internal agent report claims a quality gain after an evaluation-runner upgrade. These fictional runs use the same task IDs, reset environments, model weights and application prompt, with one attempt per task.
| Scheduled tasks | 100 before; 100 after |
|---|---|
| Scored successes | 72 before; 72 after |
| Scored failures | 28 before; 8 after |
| Infrastructure errors, no score | 0 before; 15 after |
| Missing runs, no artifact | 0 before; 5 after |
| Grader change | Before: checks final database state. After: checks the assistant's completion message. |
Your task. Decide whether to announce the improvement. Reconstruct the report's denominators, identify what evidence is missing and propose the comparison needed for a release decision.
0 words