标题： 优化实验计划的回报 显示英文标题

标题： Optimizing Returns from Experimentation Programs

摘要： 在线数字平台中的实验用于辅助决策制定。具体来说，许多实验的目标是优化感兴趣的指标。然而，零假设显著性检验可能不适合此任务，因为它对效应量的大小和机会成本不敏感。鉴于可以访问相关历史实验的池子，我们讨论了当目标是优化时，实验实践应该如何改变。我们回顾了关于A/B测试组合的经验贝叶斯分析的文献，并单独提出了A/B测试问题（Azevedo等人，2020年）作为起点，该问题将实验视为一个受约束的优化问题。我们展示了解决这一框架可以通过动态规划实现，并通过适当调整$p$-值阈值来实施。此外，我们开发了A/B测试问题的几个扩展，并讨论了这些结果对工业实验项目的影响。例如，在没有成本假设的情况下，公司应该测试更多的想法，减少测试分配的规模，并放松$p$-值阈值远离$p = 0.05$。 显示英文摘要

摘要： Experimentation in online digital platforms is used to inform decision making. Specifically, the goal of many experiments is to optimize a metric of interest. Null hypothesis statistical testing can be ill-suited to this task, as it is indifferent to the magnitude of effect sizes and opportunity costs. Given access to a pool of related past experiments, we discuss how experimentation practice should change when the goal is optimization. We survey the literature on empirical Bayes analyses of A/B test portfolios, and single out the A/B Testing Problem (Azevedo et al., 2020) as a starting point, which treats experimentation as a constrained optimization problem. We show that the framework can be solved with dynamic programming and implemented by appropriately tuning $p$-value thresholds. Furthermore, we develop several extensions of the A/B Testing Problem and discuss the implications of these results on experimentation programs in industry. For example, under no-cost assumptions, firms should be testing many more ideas, reducing test allocation sizes, and relaxing $p$-value thresholds away from $p = 0.05$.