You need preference labels for a model update. How would you design the work, pay annotators and control quality?
Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.
Case briefRead the inputs, then make your call
You are commissioning preference labels for a support assistant. All time, pay and budget figures are fictional planning inputs, not market wage benchmarks.
| Label order | 2,000 response pairs; two independent labels per pair. |
|---|---|
| Pilot timing | 30 seconds per production label, plus eight total worker-hours of required training and ten total worker-hours of review and dispute work. |
| Compensation proposal | $0.08 per submitted label, assumed to reach workers in full. Compare with an illustrative $24 per hour target for all required work. |
| Total budget | $1,600, including worker compensation, platform fees, management and other costs. |
| Judgment problem | Some responses are more polite but contradict the refund policy. Workers disagree on how to rank them. |
Your task. Design a paid pilot, calculate the full labor cost and define how you will distinguish useful labels from agreement on the wrong objective.
0 words