Title: Robust and efficient multiple-unit switchback experimentation Show Chinese title

Title: 鲁棒且高效的多单元回溯实验设计

Abstract: User-randomized A/B testing has emerged as the gold standard for online experimentation. However, when this kind of approach is not feasible due to legal, ethical or practical considerations, experimenters have to consider alternatives like item-randomization. Item-randomization is often met with skepticism due to its poor empirical performance. To fill this gap, in this paper we introduce a novel and rich class of experimental designs, "Regular Balanced Switchback Designs" (RBSDs). At their core, RBSDs work by randomly changing treatment assignments over both time and items. After establishing the properties of our designs in a potential outcomes framework, characterizing assumptions and conditions under which corresponding estimators are resilient to the presence of carryover effects, we show empirically via both realistic simulations and real e-commerce data that RBSDs systematically outperform standard item-randomized and non-balanced switchback approaches by yielding much more accurate estimates of the causal effects of interest without incurring any additional bias. Show Chinese abstract

Abstract: 用户随机化的 A/B 测试已成为在线实验的黄金标准。然而，由于法律、伦理或实际考虑等因素，当这种方法不可行时，实验者必须考虑替代方案，如项目随机化。由于其较差的经验表现，项目随机化常常受到质疑。为填补这一空白，在本文中我们介绍了一类新颖且丰富的实验设计——“正则平衡切换设计”(RBSDs)。本质上，RBSDs 通过在时间和项目上随机改变处理分配来运作。在潜在结果框架下建立我们的设计属性后，确定对应估计量在存在残留效应的情况下具有鲁棒性的假设和条件，我们通过现实模拟和真实的电子商务数据实证表明，RBSDs 在得出感兴趣的因果效应更准确的估计值方面系统性地优于标准的项目随机化和非平衡切换方法，且不会产生额外的偏差。