Title: Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests Show Chinese title

Title: 按提升排名：大规模A/B测试的成本效益方法

Abstract: A/B testing is a core tool for decision-making in business experimentation, particularly in digital platforms and marketplaces. Practitioners often prioritize lift in performance metrics while seeking to control the costs of false discoveries. This paper develops a decision-theoretic framework for maximizing expected profit subject to a constraint on the cost-weighted false discovery rate (FDR). We propose an empirical Bayes approach that uses a greedy knapsack algorithm to rank experiments based on the ratio of expected lift to cost, incorporating the local false discovery rate (lfdr) as a key statistic. The resulting oracle rule is valid and rank-optimal. In large-scale settings, we establish the asymptotic validity of a data-driven implementation and demonstrate superior finite-sample performance over existing FDR-controlling methods. An application to A/B tests run on the Optimizely platform highlights the business value of the approach. Show Chinese abstract

Abstract: A/B测试是商业实验中用于决策的核心工具，特别是在数字平台和市场中。 从业者在寻求控制假发现成本的同时，通常优先考虑性能指标的提升。 本文开发了一个决策理论框架，在成本加权假发现率（FDR）约束下最大化预期利润。 我们提出了一种经验贝叶斯方法，使用贪心背包算法根据预期提升与成本的比率对实验进行排序，并将局部假发现率（lfdr）作为关键统计量。 所得的Oracle规则是有效且排名最优的。 在大规模情况下，我们建立了数据驱动实现的渐近有效性，并展示了在现有FDR控制方法中的优越有限样本表现。 在Optimizely平台上运行的A/B测试应用突显了该方法的商业价值。