TL;DR: A rate is a weighted average of the populations contributing to it. Check event definitions and observation windows, then reconstruct counts within stable segments. Hold the mix fixed to understand composition, but use suitable evidence to determine cause. Severe failures still need containment while analysis continues.
The chart that invites a premature rollback
Fictional case: a quote assistant's task success falls from 77% to 60% after a model release and a campaign. Success means a submitted eligible request produces an accepted quote with no correction during the following seven days. Both periods have complete observation windows.
Segments describe the source package at submission, using the same classification rules in each period. “Structured” requests have the required fields; “unstructured” requests arrive as free-form notes. A task belongs to exactly one segment.
| Source segment | Before tasks | Before successes | After tasks | After successes |
|---|---|---|---|---|
| Structured | 900 | 720 | 200 | 168 |
| Unstructured | 100 | 50 | 800 | 432 |
| Total | 1000 | 770 | 1000 | 600 |
Structured success improves from 80% to 84%; unstructured success improves from 50% to 54%. The total falls because the lower-performing segment becomes a much larger share of requests. Do not average 84% and 54% equally: after-period weights are 20% and 80%, producing 60%.
Hold one thing fixed
At the original 90% / 10% mix, the new segment rates produce 0.9 × 84% + 0.1 × 54% = 81%. That is a standardized comparison, not the actual customer experience after the campaign.
Another decomposition applies the old rates to the new mix: 0.2 × 80% + 0.8 × 50% = 56%. In that ordering, mix contributes −21 percentage points relative to 77%, while within-segment improvement adds four points to reach 60%.
In general, the allocated mix and performance contributions can depend on the order of calculation when weights and rates both change. State the method. These calculations also cannot rule out changes within a segment, selection effects or an unrelated safety regression.
Follow the flow, one stage at a time.
Illustrated sequence; animation speed does not represent processing time.Read diagram description
Define the outcome, check observation windows, keep segment rules stable, reconstruct segment rates, inspect changed weights, standardize the comparison, test competing causes and choose the next action. Revisit the segment assumptions as new evidence arrives.
Arithmetic explains the total; it does not establish the cause
The release may have helped, harmed or had little effect compared with what would otherwise have happened. Both periods differ in more than model version. Inspect matched task samples, versioned traces and an appropriately designed comparison before attributing the four-point changes to the model.
The lower actual success rate still matters. More customers are experiencing unsuccessful tasks. Investigate whether the campaign promised support for source packages the product handles poorly. Review missing-field recovery, onboarding and the cost of human follow-up. A mix explanation does not excuse a bad expansion decision.
Order the investigation
First confirm numerator, denominator, eligibility and logging completeness. Then compare stable segments and inspect representative failures. Keep a separate urgent path for unauthorized actions or other consequential failures; statistical diagnosis need not delay containment.
Practise the improving-segments case. For missing telemetry, continue to the diagnosis lesson. Experimenting with AI features covers the causal question that this table leaves open.