标题： 名义尺度的编码者一致性：基于模型的方法 显示英文标题

标题： Inter-Coder Agreement for Nominal Scales: A Model-based Approach

摘要： 一致率度量（如Cohen's kappa）修正了编码员之间同意的相对频率，以考虑纯粹由机会导致的同意。 然而，在某些情况下，这些度量表现出行为，使得其值难以解释。 这些特性，例如“注释者偏差”或“流行性问题”，指的是这些度量中的某些倾向于根据许多研究人员认为与编码员一致性无关的条件，指示出反直觉的高可靠性或低可靠性值。 然而，并非所有研究人员都认同这一观点，由于没有普遍接受的编码员一致性的正式定义，很难判断这是否取决于可靠性的不同概念，还是仅仅取决于测量算法中的缺陷。 因此，在本文中我们采取公理化方法：我们引入一个模型，用于多个编码员根据名义尺度对项目进行评级。 基于此模型，我们将编码员一致性定义为以确定性分配类别到项目的概率。 然后我们讨论在何种条件下，这种编码员一致性的概念在典型的实验结果（即不同编码员的类别分配的相对频率）下是唯一确定的。 此外，我们提供了一个算法并进行了数值模拟，展示了在不同的模型参数设置下该算法的准确性。 显示英文摘要

摘要： Inter-coder agreement measures, like Cohen's kappa, correct the relative frequency of agreement between coders to account for agreement which simply occurs by chance. However, in some situations these measures exhibit behavior which make their values difficult to interprete. These properties, e.g. the "annotator bias" or the "problem of prevalence", refer to a tendency of some of these measures to indicate counterintuitive high or low values of reliability depending on conditions which many researchers consider as unrelated to inter-coder reliability. However, not all researchers agree with this view, and since there is no commonly accepted formal definition of inter-coder reliability, it is hard to decide whether this depends upon a different concept of reliability or simply upon flaws in the measuring algorithms. In this note we therefore take an axiomatic approach: we introduce a model for the rating of items by several coders according to a nominal scale. Based upon this model we define inter-coder reliability as a probability to assign a category to an item with certainty. We then discuss under which conditions this notion of inter-coder reliability is uniquely determined given typical experimental results, i.e. relative frequencies of category assignments by different coders. In addition we provide an algorithm and conduct numerical simulations which exhibit the accuracy of this algorithm under different model parameter settings.