Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Scope to one job: extract action items as structured data (task, owner, due date, source quote) from a pasted transcript. Build with a schema-constrained model call and a simple display. Spend real time on a test set that covers tentative commitments, missing dates, relative dates and multiple speakers. Report pass rates per field and name the next fixes.
How to approach it
Clarify the user (a team lead who wants follow-ups tracked), confirm the input (a transcript with speaker labels), and agree success criteria: correct items, correct owners, correct dates, nothing invented. Then commit to a narrow build.
A strong answer
Minutes 0 to 5: scope. "I'll build an extractor that takes a transcript and returns a list of action items as JSON with task, owner, due date and a quote from the transcript supporting each. Out of scope: audio, integrations with task tools, multiple languages. Success: no invented items, correct owner, dates correct or marked unknown."
Minutes 5 to 22: build.
- A page or notebook with a text box for the transcript and a button.
- One model call with instructions and a JSON schema: an array of items with
task,owner,due_date(ISO date or null),confidence_note, andquote. - Pass today's date so relative dates ("next Friday") can be resolved.
- Render the items in a table, with the quote under each.
Minutes 22 to 34: evaluate. Write eight short test transcripts targeting known hard cases:
| Case | Expected behaviour |
|---|---|
| Clear commitment with date | Correct item, owner, date |
| Tentative ("I'll try to look at it") | Either excluded or flagged as tentative |
| No date mentioned | Due date null, not invented |
| Relative date ("by end of next week") | Resolved correctly from today's date |
| Two people discuss, one commits | Owner is the one who committed |
| Item assigned to someone absent | Owner recorded as named |
| Small talk only | Empty list |
| Decision, not action ("we'll go with vendor B") | Not an action item |
Run all eight; mark each field pass or fail. A realistic result: owners correct in 7 of 8, dates in 6 of 8 (one invented date, one relative-date error), tentative handling inconsistent.
Minutes 34 to 40: readout. "Owners are reliable. Dates are the weak spot: one invented date and one relative-date error. Next I'd compute relative dates in code from extracted phrases rather than asking the model, add a 'tentative' flag to the schema, and expand to 50 real transcripts. Before any real use, users should confirm items before they go into a task tool."
Follow the flow, one stage at a time.
Illustrated sequence; animation speed does not represent processing time.Read diagram description
Minutes 0 to 5: scope to one job. Minutes 5 to 22: pass the transcript and today's date to one model call with a JSON schema, validate the schema, and show a table of task, owner, date and supporting quote. Minutes 22 to 34: run eight targeted test transcripts and mark each field pass or fail. Minutes 34 to 40: read out what works and the next fixes.
Why this scores. Structured output shows engineering judgment; quotes show traceability; the test set targets the real failure modes; and the readout separates what works from what does not with a concrete plan.
Format is not truth. Structured-output features now guarantee the shape of the response: Anthropic's documentation describes guaranteeing "schema-compliant responses through constrained decoding" (structured outputs), with listed exceptions such as refusals. A valid JSON object can still carry an invented due date. That is why the tests check field values, not just parsing, and why shipped products keep a person in the loop: Google Meet's note-taking presents suggested next steps that users "review and edit" before assigning (Workspace update, February 2025).
What interviewers probe next
"The model invented a due date. How do you stop that?" Make the field nullable and instruct that missing dates are null; better, extract the date phrase and resolve it in code, so the model never fabricates a date.
"Our meetings switch between Hindi and English. Does that change the plan?" Yes: add code-mixed transcripts to the test set before anything else. Even mature products limit this; Google Meet's help page says multiple languages in the same meeting "aren't currently supported" for its notes (checked November 2026).
"How would this integrate with a task tool?" As proposed tasks the user confirms, never auto-created, until accuracy on real meetings is proven.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Scope | Generic summariser | Action items | Structured extraction with owner, date and supporting quote, explicit non-goals |
| Build | Free-text output | JSON output | Schema-constrained, nullable fields, today's date passed, quotes for traceability |
| Evaluation and readout | One demo transcript | A few tests | Targeted hard-case set, per-field results, concrete fixes such as dates resolved in code |
Common mistakes
Building a summary instead of structured items.
Testing only a clean transcript.
Letting the model fill in dates that were never said.