Title: Not All Explanations for Deep Learning Phenomena Are Equally Valuable Show Chinese title

Title: 并非所有对深度学习现象的解释都具有同等价值

Abstract: Developing a better understanding of surprising or counterintuitive phenomena has constituted a significant portion of deep learning research in recent years. These include double descent, grokking, and the lottery ticket hypothesis -- among many others. Works in this area often develop ad hoc hypotheses attempting to explain these observed phenomena on an isolated, case-by-case basis. This position paper asserts that, in many prominent cases, there is little evidence to suggest that these phenomena appear in real-world applications and these efforts may be inefficient in driving progress in the broader field. Consequently, we argue against viewing them as isolated puzzles that require bespoke resolutions or explanations. However, despite this, we suggest that deep learning phenomena do still offer research value by providing unique settings in which we can refine our broad explanatory theories of more general deep learning principles. This position is reinforced by analyzing the research outcomes of several prominent examples of these phenomena from the recent literature. We revisit the current norms in the research community in approaching these problems and propose practical recommendations for future research, aiming to ensure that progress on deep learning phenomena is well aligned with the ultimate pragmatic goal of progress in the broader field of deep learning. Show Chinese abstract

Abstract: 近年来，深入理解令人惊讶或违反直觉的现象已成为深度学习研究的重要组成部分。 这些包括双重下降、领悟和彩票票根假设——还有许多其他现象。 该领域的研究通常会提出临时假设，试图以孤立的、逐个案例的方式解释这些观察到的现象。 本文认为，在许多显著的情况下，几乎没有证据表明这些现象出现在现实应用中，这些努力可能在推动更广泛领域进展方面效率低下。 因此，我们反对将它们视为需要定制解决方案或解释的孤立谜题。 然而，尽管如此，我们认为深度学习现象仍然具有研究价值，因为它们提供了独特的环境，使我们能够完善对更一般的深度学习原理的广泛解释理论。 通过对近期文献中这些现象的一些显著例子的研究成果进行分析，进一步支持了这一观点。 我们重新审视了研究社区在处理这些问题时的当前规范，并提出了未来研究的实用建议，旨在确保深度学习现象的进展与更广泛深度学习领域进步的最终实际目标保持一致。