How would you measure a voice assistant's usefulness without collecting users' raw conversations?
Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.
Case briefRead the inputs, then make your call
A fictional voice assistant creates timers and private reminders. The product promises not to collect raw conversations in routine telemetry.
| Opted-in cohort | 1,000 eligible users; 5,000 task attempts. |
|---|---|
| Action records | 4,300 confirmed executions; 200 linked and deduplicated immediate reversals. |
| Unknowns | Intent correctness, non-participant outcomes and semantic failure details are not fully observable. |
Your task. Design useful measurement and a separate diagnostic path while preserving the collection limit.
0 words