Separate reference overlap from faithful meaning. Build a decision from claim errors, omissions and the reader's task.
Updated Oct 2026 · Original practice questions with worked answers and teaching rubrics. Our editorial policy explains the sources, review process and limits.
A meeting summarizer has a higher reference-overlap score, but users report factual errors. This is a fictional paired evaluation on the same transcripts.
| Evaluation | 200 transcripts, unchanged references and ROUGE implementation. |
|---|---|
| Old model | Mean ROUGE-L F1 0.42; 8 summaries contain a critical factual error. |
| New model | Mean ROUGE-L F1 0.47; 18 summaries contain a critical factual error. |
| Missing information | Per-transcript paired error labels and the severity policy still need inspection. |
Your task. Make a release decision, design the error review and identify what evidence would justify reversing your call.
Separate reference overlap from faithful meaning. Build a decision from claim errors, omissions and the reader's task.
This reference answer is in Premium. Free signup opens other answers.
9 answers per topic instead of 5, and your progress kept · no cardor unlock all 420 remaining answersNo comments yet. How would you have answered?