Users approve 93% of our coding agent's permission prompts. Should we replace the prompts with an automatic classifier?
Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.
Case briefFacts and constraints for your answer
All figures are an illustrative fixture. A coding agent raises 2,000 permission prompts a day and users approve 93% of them. Engineering proposes an automatic classifier instead.
| Risky actions | About 3% of prompts, roughly 60 a day, are actions a careful reviewer would refuse; users deny 45 of them. |
|---|---|
| Published example | One vendor reports 0.4% false positives and 17% false negatives for its own classifier pipeline, on its own traffic. |
| Environment | Agents run in a container that can reach staging systems. |
Your task. Decide whether to replace the prompts, for which actions, how you would test the classifier first, and what must hold whatever it decides.
0 words