Churn models and what to do with them

The model is the easy half

A churn model with a respectable area under the curve is roughly two weeks of work for anyone competent, given clean data. What follows is harder and rarely planned for: turning a ranked list into a set of interventions that change behaviour, aimed at accounts where the intervention can plausibly work, within a retention team's real capacity, measured against a control group that was deliberately left alone. Most churn programs skip three of those four, and then cannot say whether anything was saved.

The label is a decision, not a fact

In a contractual business there is an event: a cancellation, a non-renewal, a downgrade to zero. Even there, choices remain. Does a customer who cancels and returns forty days later count as churned? Does a downgrade from twelve seats to one count? Does a payment failure that is never resolved count as churn or as a billing problem, and if you call it churn, your model will spend most of its capacity learning to predict expiring credit cards.

In a non-contractual business — retail, marketplaces, most consumer apps — no event exists at all. Churn is something you invent by declaring an inactivity window, and the window you choose moves both the label and the answer. Ninety days of silence in a weekly-purchase category is a strong signal. Ninety days in a category people buy twice a year is a Tuesday. Set the threshold from the observed distribution of gaps between purchases: a common approach is to pick a percentile of that distribution, and to compute it per segment rather than globally, because a single global window will label an entire class of infrequent-but-loyal customers as churned every cycle.

Write the definition down, with the window, the exclusions, and the treatment of returns, and put it next to the model. Almost every argument about a churn number between two teams turns out to be an argument about definitions that neither wrote down.

Leakage is the defect that shows up in every review

Churn data leaks more easily than almost any other kind, because the events surrounding a departure are recorded in the same systems as the features. The cancellation is preceded by a support conversation about cancelling, a visit to the billing page, an export of the customer's own data, a removal of seats. Include those and the model looks extraordinary. It has learned to predict cancellation from cancellation.

The discipline is a strict feature cutoff and a gap. Pick a prediction date, use only data available on or before that date, and predict churn in a window that starts some distance after it. That gap is not a technicality; it is what makes the prediction actionable. If you predict churn tomorrow, nobody can do anything about it. Predict thirty to ninety days out, depending on how long the retention motion takes, and now there is room to act.

Then check the top features by hand. Anyone can read a list of twenty and ask, for each one, whether it could have been recorded after the customer decided. That review takes an hour and it catches more real defects than any amount of cross-validation, because leakage does not look like overfitting — it looks like success in every offline metric you have.

Metrics that survive a three percent base rate

At a monthly churn rate of three percent, a model that predicts nobody churns is ninety-seven percent accurate. Accuracy is not merely a weak metric here, it is actively misleading, and it is still what gets reported to executives.

Use three numbers together. Area under the precision-recall curve, which unlike the ROC variant responds to the imbalance and moves when the positives are ranked badly. Lift at the operating point you will actually use, which for a retention team calling the top decile means precision in that decile against the base rate. And calibration, if you intend to multiply the probability by anything.

That last one matters more than teams expect. Ranking accounts by risk is one job; ranking them by expected value at risk means multiplying probability by margin by remaining horizon, and a product is only meaningful if the probability is on a real scale. Gradient-boosted models are frequently miscalibrated, often overconfident at the extremes. Fit an isotonic or Platt calibration on a held-out slice and check the reliability curve before anybody multiplies anything.

Honest ranges, so you have something to compare against. On subscription businesses with reasonable behavioural data, churn models commonly land between 0.75 and 0.85 area under the ROC curve, with two to four times lift in the top decile. Anything materially above that on a first attempt is worth investigating for leakage before it is celebrated. And the number that surprises people most: a retention intervention that moves churn among the treated by one to three percentage points is a good result, not a disappointing one.

High risk and persuadable are different populations

This is the part that decides whether the program earns anything, and it is not a modelling subtlety. It is arithmetic about who is in the top of your list.

The accounts with the highest churn probability are frequently the ones furthest gone: the champion left the company, the product was replaced, the business closed, the use case ended. No offer reaches them. Meanwhile some of the accounts a retention call would genuinely save sit in the middle of the distribution, hesitant rather than decided. A budget spent strictly top-down on probability lands disproportionately on the first group.

There is a worse version. Offering a discount to a customer who was going to renew at full price is a direct margin loss, and an unsolicited retention outreach can introduce the idea of leaving to someone who had not been thinking about it. Both effects are real and both are invisible unless somebody is holding a control group.

The proper treatment is to model the effect of the intervention rather than the probability of the outcome. Randomise the treatment across a range of risk levels, then estimate, for each account, the difference in retention between being treated and not. That is uplift modelling, and it splits the population into four useful groups: those you save, those who stay regardless, those who leave regardless, and the ones the outreach pushes out the door. Only the first group is worth spending on.

The cost is real. Uplift estimates a difference between two noisy quantities, so it needs substantially more data than a response model — a working rule of thumb is around four times as much — and it requires genuine random assignment rather than an observational comparison of who happened to get called. If you cannot support that yet, the intermediate move is cheap and useful: hold out a random slice, treat the rest, and simply compare retention by risk decile between treated and untreated. That tells you which deciles the intervention helps in, which is most of the value without any of the machinery.

Targeting rule	What it optimises	Failure mode	Needs
Top N by probability	Catching the most churners	Spends on accounts nothing can save	A ranking model only
Top N by value at risk probability × margin × horizon	Currency exposure	Still ignores whether the account is reachable	Calibrated probabilities
Risk band with a cap exclude the very top	A crude proxy for persuadability	The cut point is a guess until measured	Ranking plus a holdout
Uplift ranking	Incremental retention per dollar	Noisy on small samples; easy to fool yourself	Randomised treatment, roughly 4× the data
Cause-based routing	Matching the action to the reason	Reason codes are often wrong or missing	Reliable diagnostic signals

The capacity constraint decides the design

Ask early how many accounts the retention team can genuinely treat in a month. The answer is usually smaller than anyone expects: a team of four making meaningful calls handles a few hundred, not a few thousand. That number sets what the model has to be good at.

If capacity is two hundred accounts against a base of forty thousand, only the top half-percent of the ranking matters. Overall discrimination is close to irrelevant; precision in that thin slice is the whole game. Optimise and report at that operating point, not at the aggregate, and be aware that most model selection procedures will happily choose a model that is better on average and worse at the very top.

Capacity is also why a menu of interventions beats a better model. A personal call is expensive and can be excellent. An in-product prompt costs nearly nothing and scales to everyone. A discount costs margin and can be given automatically. With one intervention, the model can only choose who. With four, it can choose what, and the cheap ones cover the long tail the expensive one never reaches. Most churn programs we are asked to review have exactly one intervention, and it is a discount.

When "when" is more useful than "whether"

A binary model answers one question at one horizon. A time-to-event model answers a different and often more operational one: given that this account is still here today, what is the risk over each of the next several months.

That shape helps in three ways. It handles censoring correctly, meaning customers who have not churned yet contribute information rather than being labelled negative and forgotten. It produces a schedule rather than a snapshot, so outreach can be timed instead of batched. And it lets you say something about remaining tenure, which is what a value calculation actually needs when it multiplies margin by remaining horizon.

The classical proportional-hazards model is a reasonable place to start and is easy to explain. Tree-based survival models handle interactions better. Either way, check the proportional-hazards assumption if you use a model that makes it, because it fails routinely in subscription data where early-tenure and late-tenure customers behave nothing alike.

The feedback loop nobody notices for a year

Once the program runs, the training data stops being a record of natural behaviour. Accounts the model flagged got treated, some of them stayed because of it, and next year's model learns that customers with those characteristics tend to stay. The signal it was built on is being erased by the system built on top of it.

The untreated holdout fixes this too, which is a second reason to keep it: train and evaluate on the holdout where behaviour is undisturbed, and reserve the treated population for measuring the effect of treatment. Log the treatment on every account, with what was offered and when, so the two populations can always be separated. Retrofitting that log a year later is usually impossible, and the absence of it is one of the most common reasons a mature retention program cannot answer basic questions about itself.

The mistakes we are called in to fix

• Cancellation-adjacent features in the model — billing page visits, seat removals, tickets about leaving
• Accuracy reported at a three percent base rate, which makes a useless model look excellent
• Uncalibrated scores multiplied by revenue to produce a value-at-risk number nobody can defend
• An inactivity window chosen globally, labelling every infrequent-but-loyal customer as churned
• Every at-risk account treated, so the program's effect is permanently unmeasurable
• One intervention, and it is a discount, handed to people who would have renewed anyway
• A prediction horizon shorter than the retention motion, so the alert arrives too late to act on
• No treatment log, making next year's training data quietly contaminated

A six-week pass

Step five is where most of the learning is, and it is the step that gets cut when a quarter is going badly. Randomising treatment across bands for one cycle costs a small amount of retention effort and produces the only evidence that will ever tell you where the budget belongs.

Before you turn it on

• The churn definition is written down, with window, exclusions and return handling
• A feature cutoff date and a gap exist, and the gap exceeds the retention motion
• The top twenty features have been read by a person for post-decision signals
• Metrics fit the base rate: precision-recall and lift, never bare accuracy
• Probabilities are calibrated if anything is multiplied by them
• The operating point matches the retention team's real monthly capacity
• More than one intervention exists, each with a cost written beside it
• A permanent untreated holdout is in place and sized for the effect you want to see
• Every treatment is logged against the account with what and when
• Someone owns the question of whether outreach itself increases churn

Bottom line

A churn model is a ranking device, and a ranking device is worth exactly as much as the actions attached to it. The label definition determines what you are predicting, leakage determines whether the offline score means anything, calibration determines whether you can convert risk into currency, and the holdout determines whether you will ever be able to prove the program earned its cost. Spend the effort there. The choice between one gradient-boosting library and another will not decide this, and in the projects we get called into it never has.

Frequently asked questions