calibration

A deceptively simple design question. Numbers feel transparent, but most users cannot act on them, and the numbers are often less meaningful than they look.

Turn confidence scores into a defensible approval policy. Work through selective accuracy, uncertainty, correlated errors and review capacity before deciding which invoice fields can bypass a human.

Choose an operating threshold using prevalence, error costs and review capacity. A good ranking score can still produce an unusable queue.

A new judge is a new ruler. Run old and new judges in parallel on a human-labelled set, re-validate catch rates, mark the break in the trend line and pin judge versions from now on.