Our reviewers score every AI response from 1 to 5. Is that a good eval?

Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.