Title: Collaborative Analysis for Paired A/B Testing Experiments Show Chinese title

Title: 配对A/B测试实验的协作分析

Abstract: With the extensive use of digital devices, online experimental platforms are commonly used to conduct experiments to collect data for evaluating different variations of products, algorithms, and interface designs, a.k.a., A/B tests. In practice, multiple A/B testing experiments are often carried out based on a common user population on the same platform. The same user's responses to different experiments can be correlated to some extent due to the individual effect of the user. In this paper, we propose a novel framework that collaboratively analyzes the data from paired A/B tests, namely, a pair of A/B testing experiments conducted on the same set of experimental subjects. The proposed analysis approach for paired A/B tests can lead to more accurate estimates than the traditional separate analysis of each experiment. We obtain the asymptotic distribution of the proposed estimators and demonstrate that the proposed estimators are asymptotically the best linear unbiased estimators under certain assumptions. Moreover, the proposed analysis approach is computationally efficient, easy to implement, and robust to different types of responses. Both numerical simulations and numerical studies based on a real case are used to examine the performance of the proposed method. Show Chinese abstract

Abstract: 随着数字设备的广泛使用，在线实验平台常用于开展实验以收集数据来评估产品、算法和界面设计的不同变体，即所谓的 A/B 测试。 在实践中，基于同一平台的共同用户群体通常会执行多个 A/B 测试实验。 由于用户的个体效应，同一用户对不同实验的响应在一定程度上可能是相关的。 本文提出了一种新颖的框架，用于协作分析成对的 A/B 测试数据，即在同一组实验对象上进行的一对 A/B 测试实验。 所提出的成对 A/B 测试分析方法可以比传统分别分析每个实验的方法获得更准确的估计值。 我们获得了所提出估计量的渐近分布，并证明了在某些假设下，所提出的估计量在渐近意义下是最优无偏线性估计量。 此外，所提出的分析方法计算效率高，易于实现，并且对不同类型响应具有鲁棒性。 数值模拟和基于真实案例的数值研究均被用来检验所提出方法的性能。