Title: Confident Learning: Estimating Uncertainty in Dataset Labels Show Chinese title

Title: Confident Learning: 估计数据集标签中的不确定性

Abstract: Learning exists in the context of data, yet notions of confidence typically focus on model predictions, not label quality. Confident learning (CL) is an alternative approach which focuses instead on label quality by characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. Whereas numerous studies have developed these principles independently, here, we combine them, building on the assumption of a class-conditional noise process to directly estimate the joint distribution between noisy (given) labels and uncorrupted (unknown) labels. This results in a generalized CL which is provably consistent and experimentally performant. We present sufficient conditions where CL exactly finds label errors, and show CL performance exceeding seven recent competitive approaches for learning with noisy labels on the CIFAR dataset. Uniquely, the CL framework is not coupled to a specific data modality or model (e.g., we use CL to find several label errors in the presumed error-free MNIST dataset and improve sentiment classification on text data in Amazon Reviews). We also employ CL on ImageNet to quantify ontological class overlap (e.g., estimating 645 "missile" images are mislabeled as their parent class "projectile"), and moderately increase model accuracy (e.g., for ResNet) by cleaning data prior to training. These results are replicable using the open-source cleanlab release. Show Chinese abstract

Abstract: 学习存在于数据的背景下，然而置信度的概念通常关注的是模型预测，而不是标签质量。 自信学习（CL）是一种替代方法，它转而通过根据修剪噪声数据的原则、基于概率阈值计数以估计噪声以及对示例进行排序以有把握地训练，来关注标签质量，并识别数据集中存在的标签错误。 尽管许多研究已经独立开发了这些原则，但在这里，我们结合了它们，基于类别条件噪声过程的假设，直接估计嘈杂（已知）标签和未受污染（未知）标签之间的联合分布。 这导致了一种广义的CL，它在理论上是始终一致的，并且在实验上表现出色。 我们提出了CL精确找到标签错误的充分条件，并展示了CL性能在CIFAR数据集上使用噪声标签进行学习时，超过了七种最近的竞争方法。 独特的是，CL框架并不局限于特定的数据模态或模型（例如，我们用CL在被认为无误的MNIST数据集中找到了几个标签错误，并改进了Amazon评论中文本数据的情感分类）。 我们还在ImageNet上应用CL来量化本体论类别的重叠（例如，估计645张“导弹”图片被错误地标记为其父类别“投射物”），并通过在训练前清理数据，适度提高了模型准确性（例如，对于ResNet）。 这些结果可以通过开源的cleanlab发布版本进行复制。