Free account
Read this concept with a free account
Read this concept at no cost by signing in with Google.
no card needed
Core
Experimenting With AI Features: A/B Tests, Holdouts and the Traps Specific to Models
Online experiments decide whether an AI feature helps users, but AI features break common assumptions: early novelty fades, users in shared workspaces affect each other, the model can change mid-test, and engagement can rise while quality falls. How to size a test, choose the unit, pair metrics and keep a long-term holdout.
CONCEPT / AI UNIT ECONOMICS
RELATED CONCEPTS
PRACTISE THIS CONCEPT
Evals & Launch DecisionsHow would you A/B test an AI reply-suggestion feature before rolling it out to everyone?→Evals & Launch DecisionsHow would you evaluate a RAG assistant that answers questions from our help center?→Behavioral & LeadershipAI projects are full of uncertainty. How do you handle ambiguity as a PM?→Evals & Launch DecisionsThe new model scores 6 points higher on our offline eval, but the A/B test shows no change in user metrics. What happened?→Evals & Launch DecisionsModel B scores 83% and model A scores 80% on our eval. How many cases do we need before we believe B is better?→Metrics & Unit EconomicsYour AI assistant edits shared documents. How would you A/B test it without misleading yourself?→