24Our fraud model has a 0.96 ROC-AUC, but reviewers cannot handle its alerts. Would you launch it?▼hardNewGoogleMicrosoftAmazon◆ premiumChoose an operating threshold using prevalence, error costs and review capacity. A good ranking score can still produce an unusable queue.Practise this question →Read answer →
40We turned on an AI code reviewer that comments on every pull request. How do you tell whether it catches real problems or just adds noise?▼hardNewAtlassianGoogleMicrosoft◆ premiumPublished systems measure whether comments lead to code changes, aim for a precision target and filter non-actionable comments. Compare against human comments, seed known bugs for recall and run a randomised rollout.Practise this question →Read answer →