Our fraud model has a 0.96 ROC-AUC, but reviewers cannot handle its alerts. Would you launch it?
Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.
Case briefRead the inputs, then make your call
All figures are fictional. You own a fraud alert system that sends transactions to trained reviewers; full automatic blocking is also being proposed.
| Daily population | 10,000 transactions: 100 confirmed frauds and 9,900 legitimate transactions. |
|---|---|
| Candidate A | ROC-AUC 0.96; selected threshold detects 80% of frauds and flags 2% of legitimate transactions. |
| Candidate B threshold | 60 true positives and 40 false positives on the same population. |
| Review capacity | 200 cases per day. |
| Illustrative error costs | $100 per missed fraud; $5 per unnecessary review. Other operational and customer costs need investigation. |
| Calibration sample | Among 200 representative cases scored at about 80% fraud probability, 100 were fraud. |
Your task. Choose an operating policy and rollout scope. Calculate the confusion matrix, precision and workload; explain which evidence would change your decision.
0 words