标题： 量化误差一致性中的不确定性：迈向分类器行为比较的可靠性 显示英文标题

标题： Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers

摘要： 基准模型是机器学习（ML）研究快速进步的关键因素。 因此，进一步的进步取决于改进基准测试指标。 衡量机器学习模型与人类观察者行为一致性的标准指标是错误一致性（EC）。 EC比其他指标如准确性等允许更细致的行为比较，并已被用于有影响力的Brain-Score基准中，以根据与人类的行为一致性对不同的DNN进行排名。 以前报告的EC值没有置信区间。 然而，经验测量的EC值通常是噪声的——因此，在没有置信区间的情况下，有效的基准测试结论是有问题的。 在这里，我们从两个方面改进了标准的EC：首先，我们展示了如何使用自助法技术获得EC的置信区间，使我们能够为EC得出显著性检验。 其次，我们提出了一种新的计算模型，将两个分类器之间的EC与其中一个复制另一个响应的隐含概率联系起来。 这种对EC的看法使我们能够就进行足够强大、结论明确的实验所需的试验次数向科学家提供实用指导。 最后，我们使用我们的方法重新审视流行的NeuroAI结果。 我们发现，虽然人类和机器之间行为差异的一般趋势经得起审查，但许多深度视觉模型之间的报告差异在统计上不显著。 我们的方法使研究人员能够设计出充分有力的实验，可以可靠地检测模型之间的行为差异，为更严格的的行为一致性基准测试提供了基础。 显示英文摘要

摘要： Benchmarking models is a key factor for the rapid progress in machine learning (ML) research. Thus, further progress depends on improving benchmarking metrics. A standard metric to measure the behavioral alignment between ML models and human observers is error consistency (EC). EC allows for more fine-grained comparisons of behavior than other metrics such as e.g. accuracy, and has been used in the influential Brain-Score benchmark to rank different DNNs by their behavioral consistency with humans. Previously, EC values have been reported without confidence intervals. However, empirically measured EC values are typically noisy -- thus, without confidence intervals, valid benchmarking conclusions are problematic. Here we improve on standard EC in two ways: First, we show how to obtain confidence intervals for EC using a bootstrapping technique, allowing us to derive significance tests for EC. Second, we propose a new computational model relating the EC between two classifiers to the implicit probability that one of them copies responses from the other. This view of EC allows us to give practical guidance to scientists regarding the number of trials required for sufficiently powerful, conclusive experiments. Finally, we use our methodology to revisit popular NeuroAI-results. We find that while the general trend of behavioral differences between humans and machines holds up to scrutiny, many reported differences between deep vision models are statistically insignificant. Our methodology enables researchers to design adequately powered experiments that can reliably detect behavioral differences between models, providing a foundation for more rigorous benchmarking of behavioral alignment.