Title: Foundations of data imbalance and solutions for a data democracy Show Chinese title

Title: 数据不平衡的基础及数据民主的解决方案

Abstract: Dealing with imbalanced data is a prevalent problem while performing classification on the datasets. Many times, this problem contributes to bias while making decisions or implementing policies. Thus, it is vital to understand the factors which cause imbalance in the data (or class imbalance). Such hidden biases and imbalances can lead to data tyranny and a major challenge to a data democracy. In this chapter, two essential statistical elements are resolved: the degree of class imbalance and the complexity of the concept; solving such issues helps in building the foundations of a data democracy. Furthermore, statistical measures which are appropriate in these scenarios are discussed and implemented on a real-life dataset (car insurance claims). In the end, popular data-level methods such as random oversampling, random undersampling, synthetic minority oversampling technique, Tomek link, and others are implemented in Python, and their performance is compared. Show Chinese abstract

Abstract: 处理不平衡数据是在对数据集进行分类时常见的问题。 很多时候，这个问题在做出决策或实施政策时会导致偏差。 因此，了解导致数据不平衡（或类别不平衡）的因素至关重要。 这种隐藏的偏见和不平衡可能导致数据暴政，并成为数据民主的重大挑战。 在本章中，解决了两个基本的统计要素：类别不平衡的程度和概念的复杂性；解决这些问题有助于建立数据民主的基础。 此外，讨论并实现了在这些情况下适用的统计度量，并在真实的数据集（汽车保险索赔）上进行了测试。 最后，在Python中实现了流行的数据级方法，如随机过采样、随机欠采样、合成少数过采样技术、Tomek链接等，并比较了它们的性能。