You have $30,000 to improve a task assistant. Buy demonstrations, preference labels or execution feedback?
Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.
Case briefRead the inputs, then make your call
You own a task assistant that updates customer records in a sandbox. A review of 200 failed tasks assigns each failure to one primary cause. The prices and acceptance rates below are fictional planning inputs.
| Available budget | $30,000 in total |
|---|---|
| Failure review | 90 incorrect action sequences; 60 false completion claims; 30 unsuitable responses despite correct task results; 20 tool-contract or authorization defects. |
| Demonstration quote | $20 per collected example; 80% passed the pilot acceptance review. |
| Preference quote | $3 per collected comparison; 90% passed the pilot acceptance review. |
| Execution-trace quote | $5 per collected trace; 60% passed the pilot acceptance review. |
| Budget scope | Collection, acceptance review, model adaptation and independent evaluation must all fit. Other costs have not yet been quoted. |
Your task. Recommend the first investment, calculate usable-data costs and explain how you will decide whether to spend the remaining budget. State any additional assumptions.
0 words