experimentation

Hiding the bot is off the table in many markets and was never a good bet. Disclose on the first turn, then win back the lost sales by testing how you disclose and what the assistant does next.

If you don't answer the question, someone else's assistant will. Ship behind a long holdout with revenue per session as the guardrail, and build the replacement monetisation before scaling.

Students solve more practice problems with an AI that gives answers, then score worse on the exam. Decide what the tutor may reveal, how it gets the maths right and what you will measure.

Assistant users buy 60% more often, which proves almost nothing. Ground every price in live data, separate paid placement from advice and measure lift with a holdout.

A classic experiment-design question with AI-specific traps: novelty that fades, quality that differs by segment, and a cost line that classic A/B tests never had.

Compare ranking preferences efficiently without mistaking a click preference for a retention or revenue result.

Checking an A/B test early and stopping when it looks good inflates false positives. Run to the planned horizon unless the test was designed for early stopping, and stop early only for harm.

One person's AI output becomes another person's input. Choose the experiment unit, account for shared work and decide what a large user count can actually prove.

Separate saved meeting time from lost participation, and test outcomes at the team level when the feature changes shared behavior.

In one randomised trial, developers believed AI made them 20% faster while it made them 19% slower. Run staggered access on real work, measure time to merge and defects, and renew team by team.

The landmark study of AI in a call centre found the same pattern: big gains for new agents, small speed gains and small quality losses for the best. Report by skill tier and give experts a lighter mode.

Shadow-bill everyone first, roll out to new customers before existing ones, protect annual plans and default overage to zero. Set stop rules on bill-shock tickets, refunds and downgrades before you start.

Six configurations and a dozen sessions leave two sessions per cell. Keep the findings that hold across versions, drop comparisons between them, and stamp every future session with its full configuration.

A common behavioural prompt with an AI twist: model capability itself is uncertain. A good answer turns ambiguity into small, time-boxed experiments with clear decisions at the end.