Budget for the whole labeling job, distinguish disagreement from poor work and test whether the resulting data improves the model.
Updated Nov 2026 · Original practice questions with worked answers and teaching rubrics. Our editorial policy explains the sources, review process and limits.
You are commissioning preference labels for a support assistant. All time, pay and budget figures are fictional planning inputs, not market wage benchmarks.
| Label order | 2,000 response pairs; two independent labels per pair. |
|---|---|
| Pilot timing | 30 seconds per production label, plus eight total worker-hours of required training and ten total worker-hours of review and dispute work. |
| Compensation proposal | $0.08 per submitted label, assumed to reach workers in full. Compare with an illustrative $24 per hour target for all required work. |
| Total budget | $1,600, including worker compensation, platform fees, management and other costs. |
| Judgment problem | Some responses are more polite but contradict the refund policy. Workers disagree on how to rank them. |
Your task. Design a paid pilot, calculate the full labor cost and define how you will distinguish useful labels from agreement on the wrong objective.
Budget for the whole labeling job, distinguish disagreement from poor work and test whether the resulting data improves the model.
This reference answer is in Premium. Free signup opens other answers.
9 answers per topic instead of 5, and your progress kept · no cardor unlock all 420 remaining answersNo comments yet. How would you have answered?