Title: PandasBench: A Benchmark for the Pandas API Show Chinese title

Title: PandasBench: 一个用于 Pandas API 的基准测试

Abstract: The Pandas API has been central to the success of pandas and its alternatives. Despite its importance, there is no benchmark for it, and we argue that we cannot repurpose existing benchmarks (from other domains) for the Pandas API. In this paper, we introduce requirements that are necessary for a Pandas API enchmark, and present the first benchmark that fulfills them: PandasBench. We argue that it should evaluate the real-world coverage of a technique. Yet, real-world coverage is not sufficient for a useful benchmark, and so we also: cleaned it from irrelevant code, adapted it for benchmark usage, and introduced input scaling. We claim that uniform scaling used in other benchmarks (e.g., TPC-H) is too coarse-grained for PandasBench, and use a non-uniform scaling scheme. PandasBench is the largest Pandas API benchmark to date, with 102 notebooks and 3,721 cells. We used PandasBench to evaluate Modin, Dask, Koalas, and Dias. This is the largest-scale evaluation of all these techniques to date. Prior works report significant speedups using constrained benchmarks, but we show that on a larger benchmark with real-world code, the most notebooks that got a speedup were 8/102 (~8%) for Modin, and 0 for both Koalas and Dask. Dias showed speedups in up to 55 notebooks (~54%), but it rewrites code incorrectly in certain cases, which had not been observed in prior work. Second, we identified many failures: Modin runs only 72/102 (~70%) notebooks, Dask 4 (~4%), Koalas 10 (~10%), and Dias 97 (95%). Show Chinese abstract

Abstract: Pandas API 对 pandas 及其替代品的成功起到了核心作用。尽管其重要性毋庸置疑，但目前尚无针对它的基准测试，我们认为无法将其他领域的现有基准测试重新用于 Pandas API。本文介绍了 Pandas API 基准测试所需的必要要求，并提出了首个满足这些要求的基准测试：PandasBench。我们主张它应评估技术在实际应用中的覆盖率。然而，实际覆盖率不足以构建有用的基准测试，因此我们还对其进行了清理（移除无关代码）、适配为基准测试使用，并引入了输入缩放机制。我们认为，其他基准测试（例如 TPC-H）中使用的统一缩放方案对于 PandasBench 来说过于粗糙，因此采用了非统一缩放方案。PandasBench 是迄今为止最大的 Pandas API 基准测试，包含 102 个笔记本和 3,721 个单元格。我们利用 PandasBench 对 Modin、Dask、Koalas 和 Dias 进行了评估，这是迄今为止对这些技术的最大规模评估。先前的研究报告称，在受限基准测试中获得了显著加速，但我们发现，在更大规模的实际代码基准测试中，获得加速的笔记本数量仅为：Modin 为 8/102（约 8%），而 Koalas 和 Dask 均为 0。Dias 在多达 55 个笔记本中（约 54%）显示了加速，但在某些情况下错误地重写了代码，这一点在之前的工作中未被观察到。其次，我们发现了许多失败案例：Modin 仅运行了 72/102（约 70%）的笔记本，Dask 仅运行了 4 个（约 4%），Koalas 运行了 10 个（约 10%），而 Dias 运行了 97 个（95%）。