Title: Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests Show Chinese title

Title: 通过提升排名：大规模A/B测试的成本效益方法

Abstract: A/B testers that conduct large-scale tests often prioritize lifts as the main outcome metric and want to be able to control costs resulting from false rejections of the null. This work develops a decision-theoretic framework for maximizing profits subject to false discovery rate (FDR) control. We build an empirical Bayes solution for the problem via a greedy knapsack approach. We derive an oracle rule based on ranking the ratio of expected lifts and the cost of wrong rejections using the local false discovery rate (lfdr) statistic. Our oracle decision rule is valid and optimal for large-scale tests. Further, we establish asymptotic validity for the data-driven procedure and demonstrate finite-sample validity in experimental studies. We also demonstrate the merit of the proposed method over other FDR control methods. Finally, we discuss an application to data collected by experiments on the Optimizely platform. Show Chinese abstract

Abstract: 从事大规模测试的A/B测试人员通常会优先考虑提升作为主要的结果指标，并希望能够控制因错误拒绝零假设而导致的成本。 本文开发了一种基于决策论的框架，用于在控制虚假发现率（FDR）的同时最大化利润。 我们通过贪婪背包方法为该问题构建了一个经验贝叶斯解决方案。 我们基于局部虚假发现率（lfdr）统计量，推导出一个基于预期提升与错误拒绝成本比率排序的最优规则。 我们的最优决策规则在大规模测试中是有效的且是最优的。 此外，我们证明了数据驱动过程的渐近有效性，并在实验研究中展示了有限样本的有效性。 我们还展示了所提出的方法相对于其他FDR控制方法的优势。 最后，我们讨论了该方法在Optimizely平台实验收集的数据中的应用。