Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Pick one outcome metric tied to why the business built it (retention or paid conversion among writers), support it with behaviour metrics that show the AI is helping (drafts kept with light edits, repeat use), and guard it with quality, cost and risk metrics. "AI generations per day" is an activity count, not success.
How to approach it
Ask why the company is building it. Retention of existing customers, an upsell to a paid AI tier, and defending against a competitor each lead to a different top metric. Ask also who the user is (a student, a team writing specs, a marketing team) because "helpful" looks different for each. Then build the metrics as a tree from the business goal down.
A strong answer
Assume the goal is retention and expansion of team workspaces, with AI included in the business plan.
I would structure the metrics in five layers, because AI features fail in ways classic metric sets do not catch.
| Layer | Metric | What it tells you |
|---|---|---|
| Business | 90-day workspace retention, seat expansion among teams using AI vs similar teams not using it | Whether the feature is worth building |
| Product behaviour | Weekly active writers using AI; repeat use within 7 days of first use | Whether people come back to it |
| Value | Share of AI drafts kept after light edits; time from blank page to first shared draft | Whether the output is useful |
| Quality and trust | Undo rate; drafts deleted immediately; regenerate rate; factual error reports | Whether it is getting things wrong |
| Cost and risk | Inference cost per active writer; share of outputs flagged by safety filters | Whether it is sustainable and safe |
The value layer is where most of the signal is. "Kept after light edits" needs a definition: say, the AI-written text survives into the shared version with less than 30% of characters changed. Users who keep drafts nearly as written are getting value; users who rewrite everything are using the AI as a starting prompt at best. Tracking the distribution of edit distance, not just the mean, shows whether there is a group the feature fails completely.
The quality layer catches what activity metrics hide. A high regenerate rate can look like engagement on a usage dashboard. It usually means the first answer was not good enough.
Cost per active writer matters because AI features carry a marginal cost classic features do not. If heavy users generate thousands of drafts a month on a flat plan, the feature can be popular and lose money. I would set a budget per active writer and watch the 95th-percentile user, not the average.
Published studies are useful here because they disagree in instructive ways. In a randomised experiment on professional writing tasks, access to a chat assistant cut the average time taken by 40% and raised rated quality by 18% (Noy and Zhang, Science, 2023), which supports time to first shared draft as a value metric. GitHub's telemetry study of code completion found that how often suggestions were accepted, "rather than more specific metrics regarding the persistence of completions", tracked developers' own sense of productivity (Ziegler et al., 2022). That is code, not prose, and perceived productivity is not retention, but it is a warning that the persistence metric I prefer has to earn its place against simpler acceptance. And a randomised field experiment giving workers an AI assistant found they spent about two fewer hours a week on email but detected no shift in the quantity or composition of their tasks (Dillon et al., 2025): time can fall without the business outcome moving.
For the primary metric I would choose share of weekly active writers who keep at least one AI draft, because it moves faster than retention, is hard to inflate with empty activity, and should lead retention if the feature works. I would validate that link after a quarter by comparing retention of keepers against similar non-keepers, and run acceptance rate alongside it. If acceptance predicts retention as well as keeping does, switch to the simpler metric; if neither moves retention, the feature is a convenience, and the business case should say so.
What interviewers probe next
"Usage doubled after launch. Success?" Check whether kept-draft rate held. If usage doubled and kept drafts fell, people are trying it more and liking it less.
"How do you separate the AI's effect on retention from selection?" Teams that adopt AI may already be the engaged ones. Use a holdout group or compare matched cohorts before and after access.
"Which single guardrail would you alert on?" Undo rate by feature. A jump right after a model or prompt change is the fastest sign of a quality regression.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Primary metric | Generations or clicks | Engagement tied to a goal | A value metric (drafts kept with light edits) with a stated link to the business outcome, to be validated |
| Coverage | Usage only | Usage plus satisfaction | Business, behaviour, value, quality and cost layers |
| AI-specific risk | None | Mentions quality | Regenerate and undo as failure signals, cost per user with tail awareness, selection bias in retention claims |
Common mistakes
Treating regenerations or total generations as engagement.
Leaving cost out of the success definition for an included-in-plan feature.
Claiming retention impact from users who self-selected into the feature.
See the AI metric tree.