标题： QRA++：自然语言处理中常见类型结果的量化可再现性评估 显示英文标题

标题： QRA++: Quantified Reproducibility Assessment for Common Types of Results in Natural Language Processing

摘要： 在自然语言处理领域中报告的再现性研究提供了单独的数据点，这些数据点组合在一起表明该领域的再现性水平令人担忧地低。 由于每项再现性研究都是基于其自身通常未明确说明的再现成功/失败标准得出定量结论，因此得出的结论难以解释、比较和学习。 在本文中，我们提出了QRA++，这是一种定量的再现性评估方法，它（i）在三个粒度级别上生成连续值的再现性评估；（ii）使用可以直接在不同研究之间比较的再现性度量；（iii）将对再现性程度的期望建立在实验之间的相似性程度基础上。 QRA++使得可以进行更具信息量的再现性评估，并得出关于导致再现性更好或更差的原因的结论。 我们通过将QRA++应用于三个可比较的实验集来说明这一点，揭示了清晰的证据，表明再现性程度取决于实验属性的相似性，但也取决于系统类型和评估方法。 显示英文摘要

摘要： Reproduction studies reported in NLP provide individual data points which in combination indicate worryingly low levels of reproducibility in the field. Because each reproduction study reports quantitative conclusions based on its own, often not explicitly stated, criteria for reproduction success/failure, the conclusions drawn are hard to interpret, compare, and learn from. In this paper, we present QRA++, a quantitative approach to reproducibility assessment that (i) produces continuous-valued degree of reproducibility assessments at three levels of granularity; (ii) utilises reproducibility measures that are directly comparable across different studies; and (iii) grounds expectations about degree of reproducibility in degree of similarity between experiments. QRA++ enables more informative reproducibility assessments to be conducted, and conclusions to be drawn about what causes reproducibility to be better/poorer. We illustrate this by applying QRA++ to three example sets of comparable experiments, revealing clear evidence that degree of reproducibility depends on similarity of experiment properties, but also system type and evaluation method.