Model B scores 83% and model A scores 80% on our eval. How many cases do we need before we believe B is better?

Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.