Efficient Multi-Cohort Inference for Long-Term Effects and Lifetime Value in A/B Testing with User Learning
2026-04-22 • Machine Learning
Machine Learning
AI summaryⓘ
The authors point out that streaming platforms often lose users (churn), which is costly, but A/B tests usually only look at short-term results and might miss important long-term effects. They created a new method to better estimate how changes impact user retention and value over time, even with limited data from multiple user groups. Their approach combines data more efficiently to reduce errors and models how effects fade, helping to understand the lasting impact of changes. This method helps make better decisions by showing when short-term or long-term measures alone could be misleading.
churnA/B testinguser retentionlong-term treatment effectsinverse-variance weightingmulti-cohort analysisparametric decaystreaming platformsexperimental horizon
Authors
Dario Simionato, Andrea Tonon, Mingxue Wang, Weiguo Wang, Tong Gui, Xiaoyue Li
Abstract
In streaming platforms churn is extremely costly, yet A/B tests are typically evaluated using outcomes observed within a limited experimental horizon. Even when both short- and predicted long-term engagement metrics are considered, they may fail to capture how a treatment affects users' retention. Consequently, an intervention may appear beneficial in the short term and neutral in the long term while still generating lower total value than the control due to users churn. To address this limitation, we introduce a method that estimates long-term treatment effects (LTE) and residual lifetime value change ($ΔERLV$) in short multi-cohort A/B tests under user learning. To estimate time-varying treatment effects efficiently, we introduce an inverse-variance weighted estimator that combines multiple cohorts estimates, reducing variance relative to standard approaches in the literature. The estimated treatment trajectory is then modeled as a parametric decay to recover both the asymptotic treatment effect and the cumulative value generated over time. Our framework enables simultaneous evaluation of steady-state impact and residual user value within a single experiment. Empirical results show improved precision in estimating LTE and $ΔERLV$ and identify scenarios in which relying on either short-term or long-term metrics alone would lead to incorrect product decisions.