Fictional churn planning exercise. A time-held-out set has mature 30-day churn labels; all features were available at prediction time. Each model selects the same outreach population size.
Scenario inputs| Accounts | 10,000 |
|---|
| Observed churners | 800 |
|---|
| Outreach capacity | 500 |
|---|
| Usage rule | 150 true churners in the 500 selected accounts. |
|---|
| Tabular model | 220 true churners in the 500 selected accounts. |
|---|
| LLM on serialized records | 190 true churners in the 500 selected accounts. |
|---|
| Unknown | The effect of outreach on retention has not been measured. |
|---|
Your task. Choose a model and an intervention pilot. Calculate precision, recall and F1 at the supplied capacity, identify leakage risks and explain what would change your decision.