Interactive demonstration

The retraining trap

A churn model predicts; a matchmaking policy acts on the prediction; players respond; the model retrains on their response. Below is the smallest world where that loop can run — one number for the model, one for the players, and a causal arrow we planted between them. Watch where retraining lands. Then check what the landing spot is worth. Companion to Working Paper WP-2026-01

[ Illustrative synthetic data — demonstration only ]

base churn rate p₀

performativity ε (prediction → churn)

performative risk PR(θ) — the score each model would earn on the world it makes today’s valley — the risk curve on the world the current model already made θ_PS — where retraining parks θ_PO — the performative optimum

the model’s validation score (Brier, on its own induced data — lower is better) realized churn rate in the population

–

θ_PS = p₀/(1−ε) — the retraining fixed point

–

θ_PO — best model accounting for its own effect

–

the gap — visible only to a method that models the loop

–

contraction per retrain = |ε| — how fast the trap closes

–

calibration at θ_PS: predicted vs realized churn

What you are looking at

The world: players churn at base rate p₀. The model publishes a churn prediction θ; the policy acting on it shifts realized churn to p(θ) = p₀ + εθ. That ε is the whole story — the strength of the arrow from prediction to outcome. Retraining is the obvious move, repeated: fit the model to the churn you observed, i.e. θ_t+1 = p(θ_t). Press Retrain once and watch the gray valley — the risk curve on the data the current model generated — move each time the model moves. The model chases the valley; the valley moves because the model moved; the chase contracts, at rate |ε| per round, onto the red fixed point θ_PS = p₀/(1−ε).

And here is the property that makes the fixed point so hard to distrust from the inside: at θ_PS, predicted churn equals realized churn — check the calibration readout. The model is perfectly calibrated. Calibration is the equilibrium condition: the fixed point is defined as the model the induced world confirms. Every internal check passes there. The blue curve shows what the checks can’t: the performative risk each model would earn on the world it would make, whose minimum θ_PO sits somewhere else (green). Slide ε and watch the two pull apart — they coincide only on the razor’s edge 2p₀ = 1−ε, and nothing in the retraining loop knows the edge exists.

Meanwhile the lower chart runs the two dashboards. The blue line is the model’s validation score on its own induced data — improving, converging, publishable. The amber line is realized churn in the population. With ε > 0 the loop is self-fulfilling: each retrain toward the fixed point drags realized churn upward while the validation dashboard reports steady progress. Both lines are true. They answer different questions, and only one of them is about the players.

The formal floor under the cartoon

Nothing here is invented for the occasion. This is the standard toy of performative prediction (Perdomo, Zrnic, Mendler-Dünner & Hardt, ICML 2020): the fixed point is a performatively stable point, the green marker is the performative optimum, and the theory’s results govern exactly what you see — retraining converges linearly to stability when the loop is a contraction; stable and optimal are distinct solution concepts; and the gap between them can be made arbitrarily large in problems no less well-behaved than this one (Miller, Perdomo & Zrnic, ICML 2021). Reaching θ_PO requires estimating how the model moves the world — the one term retraining never computes. The working paper maps this machinery onto engagement-optimized matchmaking, where the arrow from prediction to churn runs through match assembly and is documented; the paper states what would falsify the mapping on real telemetry.

One honest note. We planted ε, so the trap is visible by construction — a cartoon, built to be seen through. In a live pipeline the arrow’s existence is documented but its strength is an empirical unknown, and whether a real retraining loop sits in the tame regime or the trap regime turns on a ratio — the world’s sensitivity to the model, against the objective’s curvature — that standard pipelines do not estimate, because they contain no object that represents the loop at all. Measuring that ratio on real telemetry is identification work, and it is exactly the work a validation dashboard cannot do for you.

Illustrative synthetic data — demonstration only. Model: Bernoulli churn with linear response p(θ) = clip(p₀ + εθ), Brier loss; retraining θ_t+1 = p(θ_t); PR(θ) = p(θ)(1−θ)² + (1−p(θ))θ²; ε kept below ½ so PR stays strictly convex. References: Perdomo, Zrnic, Mendler-Dünner & Hardt (2020), ICML, PMLR 119; Miller, Perdomo & Zrnic (2021), ICML, PMLR 139. Companion: The Matchmaker’s Fixed Point, Dividual Working Paper WP-2026-01.