Our summarizer's ROUGE score improved, but users find more factual errors. What would you measure and ship?
Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.
Case briefRead the inputs, then make your call
A meeting summarizer has a higher reference-overlap score, but users report factual errors. This is a fictional paired evaluation on the same transcripts.
| Evaluation | 200 transcripts, unchanged references and ROUGE implementation. |
|---|---|
| Old model | Mean ROUGE-L F1 0.42; 8 summaries contain a critical factual error. |
| New model | Mean ROUGE-L F1 0.47; 18 summaries contain a critical factual error. |
| Missing information | Per-transcript paired error labels and the severity policy still need inspection. |
Your task. Make a release decision, design the error review and identify what evidence would justify reversing your call.
0 words