Abstract

A/B tests are standard tools for estimating the average treatment effect in online controlled experiments (OCEs). The majority of OCE theory relies on the Stable Unit Treatment Value Assumption, which presumes the response of individual users depends only on the assigned treatment, not the treatments of others. Violations of this assumption occur when users are subjected to network interference, a common phenomenon in social media platforms. Standard methods for estimating the average treatment effect typically ignore network effects and produce heavily biased results. Additionally, unobserved user covariates, such as offline information or variables hidden due to privacy restrictions, that influence user response and network structure also bias current estimators of the average treatment effect. In this paper, we demonstrate that network-influential lurking variables can heavily bias popular network clustering-based methods, thereby making them unreliable. To address this problem, we propose a two-stage design and estimation technique called HODOR (Hold-Out Design for Online Randomized experiments). We show that HODOR is unbiased for the average treatment effect, has minimizable variance, and provides reliable estimation even when the underlying network is unknown.