标题： 带有未知参考的数据标注 显示英文标题

标题： Labelling Data with Unknown References

摘要： 当存在某种被普遍接受的方式来衡量评估器作为标记器的性能时，该评估器被认为是可信赖的。 建立可信度的两种方法要么是通过测试它，要么是假设评估器“知道”某种标注语料库的方式。 然而，如果没有标注参考（例如开发集），这两种方法都无法使用：前者需要数据，而后者是一种假设，而非证据。 为了解决这个问题，我们引入了一种算法（即“无数据算法”），通过这种方法可以在没有任何现有参考的情况下建立对评估器的信任。 我们的算法通过向所述评估器逐步提出挑战来工作。 我们证明了这种方法在大多数情况下足以建立可信度，使得当评估器确实知道如何标注语料库时，“无数据算法”会接受其输出；相反，当这些评估器无法证明这一点时，会将其标记为不可信。 我们提供了正确性的正式证明和有限的实验结果。 显示英文摘要

摘要： An evaluator is trustworthy when there exists some agreed-upon way to measure its performance as a labeller. The two ways to establish trustworthiness are either by testing it, or by assuming the evaluator `knows' somehow the way to label the corpus. However, if labelled references (e.g., a development set) are unavailable, neither of these approaches work: the former requires the data, and the latter is an assumption, not evidence. To address this, we introduce an algorithm (the `No-Data Algorithm') by which to establish trust in an evaluator without any existing references. Our algorithm works by successively posing challenges to said evaluator. We show that this is sufficient to establish trustworthiness w.h.p., in such a way that when the evaluator actually knows the way to label the corpus, the No-Data Algorithm accepts its output; and, conversely, flags untrustworthy evaluators when these are unable to prove it. We present formal proofs of correctness and limited experiments.