An AI reply-suggestion feature in our messaging app will get a billion requests a day. How do you make cost and latency work?

Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.