YOUR PRACTICE DESK
Make the call.
Then defend it.
Write your reasoning before opening the answer. Compare it with the rubric, then come back to the decisions that need another attempt.
Explore the question bankA DECISION TO REHEARSE
Evals & Launch Decisions · hard
Your support agent completes 82% of tasks in testing. Is it ready to launch?
- 01Draft your reasoning
- 02Compare the rubric
- 03Choose what to revisit
NEW HERE? MAKE YOUR FIRST CALL.
One agent. Four decisions. Your launch plan.
Try the free launch case, see the consequences of each choice and find your next exercise.
YOUR WORKING SET
Pick up your preparation.
One starting scenario from each topic's study path. Each reference answer is available with your current access.
12 questions
How would you double usage of an AI image-generation feature in a consumer app, with a team of three engineers?
Answer availableEvals say our AI ticket summaries are accurate, but support agents don't trust them and re-read every ticket anyway. What do you do?
Answer availableThe model provider we build on just launched a feature that does our core use case, for free inside their assistant. What do you do?
Answer availableNew models accept enormous context windows. Should we drop RAG and just put all our documents in the prompt?
Answer availableDesign an AI assistant that answers employees' questions from all of a company's internal documents.
Answer availableEach step of our agent works 95% of the time in testing, yet users say it fails constantly. Explain, and fix it.
Answer availableYour support agent completes 82% of tasks in testing. Is it ready to launch?
Answer availableEstimate what it costs to run an AI support agent on a million tickets a month. Is it worth it?
Answer availableOur assistant reads users' email and can send messages and share files. How could an attacker abuse it, and how do you stop that?
Answer availablePlan the launch of a new, cheaper and faster model tier for your API.
Answer availableLive round: build a tool that turns a meeting transcript into action items with owners and due dates. Show me how you'd do it in 40 minutes.
Answer availableTell me about a time you disagreed with engineering or research about whether an AI feature was ready to launch.
Answer available