标题： 同行评审的随意性：对NIPS实验的贝叶斯分析 显示英文标题

标题： Arbitrariness of peer review: A Bayesian analysis of the NIPS experiment

摘要： 同行评审的原则是研究评估的核心，通过确保只有高质量的项目得到资助或发表。 但同行评审也受到了批评，因为评审人员的选择可能在系统中引入偏见。 2014年，"神经信息处理系统\rq \rq{} 会议"的组织者进行了一项实验，其中$10\%$份提交的手稿（166项）经历了两次评审过程。 随意性被测量为在第二次委员会审查的情况下，已被接受的稿件被拒绝的条件概率。 这个数字等于$60\%$，总接受率等于$22.5\%$。 在这里，我们通过对这两个数字进行贝叶斯分析，引入了一个隐藏参数，用于衡量一份稿件符合基本质量标准的概率。 通常的标准质量标准包括新颖性、清晰度、可重复性、正确性以及不存在任何形式的不当行为，并且这些标准被大量提交的稿件所满足。 贝叶斯估计对于隐藏参数等于$56\%$ ($95\%$置信区间: $ I = (0.34, 0.83)$)，并且有明确的解释。 结果表明为了减少未来评审过程中的随意性估计，应该提高总的接受率。 显示英文摘要

摘要： The principle of peer review is central to the evaluation of research, by ensuring that only high-quality items are funded or published. But peer review has also received criticism, as the selection of reviewers may introduce biases in the system. In 2014, the organizers of the ``Neural Information Processing Systems\rq\rq{} conference conducted an experiment in which $10\%$ of submitted manuscripts (166 items) went through the review process twice. Arbitrariness was measured as the conditional probability for an accepted submission to get rejected if examined by the second committee. This number was equal to $60\%$, for a total acceptance rate equal to $22.5\%$. Here we present a Bayesian analysis of those two numbers, by introducing a hidden parameter which measures the probability that a submission meets basic quality criteria. The standard quality criteria usually include novelty, clarity, reproducibility, correctness and no form of misconduct, and are met by a large proportions of submitted items. The Bayesian estimate for the hidden parameter was equal to $56\%$ ($95\%$CI: $ I = (0.34, 0.83)$), and had a clear interpretation. The result suggested the total acceptance rate should be increased in order to decrease arbitrariness estimates in future review processes.