TL;DR: Write two kinds of scenario. Rules the product must never break (approval before sending, no data from another tenant, a refusal in a named situation) get ordinary Given/When/Then scenarios with exact assertions, and one failure blocks release. The model's behaviour (a good summary, a correct classification) gets a scenario outline run over a judged example set, where the Then step names a property a grader can check and the acceptance line states a pass rate, a sample size and the failures that block regardless of the rate.
What BDD gives a product team
Behaviour-driven development writes acceptance criteria as scenarios in plain language that a product manager, an engineer and a tester can all read and that a test runner can execute. The common syntax is Gherkin: a Given sets up the context, a When names the action and a Then states the outcome that must be true. A Scenario Outline repeats one scenario over the rows of an Examples table. The Gherkin reference documents the keywords.
For deterministic software, a Then step is an exact assertion: the total is 42.50, the button is disabled. That is where AI features break the habit. Ask the same model the same question twice and you may get two different, equally acceptable answers. A Then step that demands one exact string fails good outputs, and a Then step that says "the summary is good" cannot be tested at all.
Split the spec by what can be exact
The fix is to sort every acceptance criterion into one of two kinds before writing it.
| Kind | Example | How the Then step is written | Release rule |
|---|---|---|---|
| Rule | The assistant never sends an email without the user's approval | Exact assertion on system state | Any failure blocks |
| Behaviour | The summary lists every action item with its owner | A property a grader can judge pass or fail on one example | A pass rate over a set, plus blocking failure types |
Rules live outside the model. They are enforced by code (a permission check, a filter, a schema), so they can be tested like any other software. Behaviour comes from the model, so it is measured across many examples.
A worked example: meeting action items
A team is building a feature that extracts action items from meeting transcripts.
A rule scenario:
Scenario: Action items are never posted without approval
Given a transcript with three action items
When the assistant finishes extracting them
Then nothing is posted to the team channel
And the draft waits for the meeting owner's approval
A behaviour scenario outline:
Scenario Outline: Every action item is captured with its owner
Given the transcript "<transcript_id>" from the judged set
When the assistant extracts action items
Then every action item the reviewers marked appears in the output
And each appears with the owner the reviewers marked
And no item appears that the reviewers did not mark
Examples:
| transcript_id |
| T-001 |
| T-002 |
The Examples table points at a judged set, usually 100 to 300 transcripts drawn from real meetings, where two reviewers marked the correct items. The acceptance line sits beside the outline:
Accept when at least 90% of 200 transcripts pass all three Then steps, with zero transcripts where an item is assigned to the wrong person.
The arithmetic behind the threshold
A pass rate from a sample is an estimate. If 184 of 200 transcripts pass, the observed rate is 92%. A rough 95% interval is plus or minus 1.96 times the square root of 0.92 × 0.08 ÷ 200, about 3.8 points, so the true rate is probably between 88% and 96%. The feature clears a 90% bar on the point estimate and might still sit just below it. Two decisions follow:
- Set the sample size from the margin you can live with. To halve the margin you need four times the examples.
- Name blocking failures separately. A wrong owner on one action item can cause a missed deadline, so it blocks even at a 99% pass rate.
The confidence intervals concept covers the calculation in more depth.
Writing Then steps a grader can judge
Each Then step in a behaviour scenario should be something a person or a calibrated model judge can mark pass or fail on one example, without a scale. "Is concise" fails that test. "Is under 120 words" passes it. "Captures the decision the meeting reached, as the reviewers recorded it" passes it. Keep one property per step, so a failure tells you which property broke, and avoid 1 to 5 scores, which hide disagreement (see binary pass/fail judgments).
| Weak Then step | Gradeable Then step |
|---|---|
| The summary is accurate | Every number in the summary appears in the transcript |
| The tone is appropriate | The reply contains no promise of a refund amount |
| The answer is helpful | The answer cites the policy section that decides the question |
Where the scenarios run
Rule scenarios run in the ordinary test suite on every change. Behaviour outlines run in the evaluation harness against the golden set, on every prompt, model or retrieval change, and the acceptance line becomes a regression gate. The same plain-language scenarios also make a good brief for a coding agent: they state what done means in a form it can turn into tests.
In the interview
Expect this when a loop asks how you would write acceptance criteria for an AI feature, or when an interviewer says that Given/When/Then "does not work for LLMs". The answer that scores separates rules from behaviour, writes one exact rule scenario and one scenario outline aloud, and states the acceptance line with a sample size and a blocking failure. Weak answers either force exact outputs onto a model or retreat to "we will review quality manually".
Next: golden sets, writing an AI PRD.