TL;DR: Name the uncertainty before choosing the prototype. A person behind the interface can test whether the help is valuable; they cannot establish automated reliability or marginal cost. A successful model replay cannot establish that customers will use or buy the product.
Three experiments that look like the same demo
Original teaching case: a small manufacturer wants an assistant that turns a technician's fault description into a maintenance checklist. The team could show the same screen in three different experiments:
| Experiment | What happens behind the screen | What it can establish | What remains unknown |
|---|---|---|---|
| Manual service | A qualified specialist prepares the checklist | Whether the output fits the task and users return | Automated quality, latency and staffing at scale |
| Offline replay | A model processes previously recorded cases | Behavior on those cases, against a defined reference | Whether live users supply comparable information |
| Bounded live pilot | The system uses approved inputs; a specialist reviews before delivery | Actual review effort, failures, usage and integration friction | Generalization beyond the pilot cohort and sustained demand |
Call a test an MVP if that is your team's vocabulary. The label does not change its evidence. A paid manual service can be a useful first product; a deployed model can still be an experiment with no demonstrated demand.
Write a contract for the experiment
Specify the participant, decision, input and baseline. Include what the participant is told, how errors are contained and who checks the result. When humans prepare or review the output, disclose that involvement and account for its work. A prototype should not create a false impression of unattended automation.
For the manufacturer, the initial question is whether a structured checklist helps a technician find the approved procedure. Start with a non-operational task or supervised exercise. Keep decisions about live equipment with qualified staff. A manually prepared checklist can test the format without pretending the automation is ready.
Make the hidden labor visible
Illustrative pilot: 60 checklists take a specialist eight minutes each to prepare. That is eight hours of work. If the interface returns the result instantly during a presentation because the team prepared it yesterday, the demo has not measured eight-minute latency or live staffing. Record the preparation honestly.
Now suppose a model produces a draft in ten seconds, but checking it takes six minutes and correcting it takes two more. The workflow has not saved specialist time on that task. It may still improve consistency or let a different role do part of the work, but those are separate hypotheses to test.
Count the complete job: collecting inputs, finding documents, reviewing, correcting and handing the result to the next person. Do not compare an automated subtask with a human's entire job.
Preserve the comparison
Recruit for the intended workflow, including people with poor inputs and little spare time. Early enthusiasts can reveal possibilities; they do not represent every future user. Record declined participation and abandoned tasks.
For a causal claim about time saved or contacts avoided, use a suitable randomized comparison where feasible. If a small pilot uses observational comparisons, report them as signals and describe the selection differences. A participant who volunteered for the assistant may already work differently from one who did not.
A convincing pilot readout contains failures. Keep a short evidence register: observed result, denominator, cohort, version, missing evidence and next decision. This is also how to answer an interviewer who asks what you would validate in the first two weeks.
What earns a larger release
Choose the next investment according to what remains uncertain. Demand evidence with weak automation may justify a narrow assisted product. Strong automation with weak demand may justify more discovery. Neither result requires a bigger model by default.
Link the decision to data readiness, then use the prototype-to-production checks to inspect integrations and operating ownership. For a complete sequence, read the AI product lifecycle.
Further reading: Marily Nika's Building AI-Powered Products, chapter 2, discusses prototypes and early products. This page uses an original case to distinguish the evidence each experiment supports.