Title: Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches Show Chinese title

Title: 跨回归范式的尺度定律现象：多重和核方法

Abstract: Recently, Large Language Models (LLMs) have achieved remarkable success. A key factor behind this success is the scaling law observed by OpenAI. Specifically, for models with Transformer architecture, the test loss exhibits a power-law relationship with model size, dataset size, and the amount of computation used in training, demonstrating trends that span more than seven orders of magnitude. This scaling law challenges traditional machine learning wisdom, notably the Oscar Scissors principle, which suggests that an overparametrized algorithm will overfit the training datasets, resulting in poor test performance. Recent research has also identified the scaling law in simpler machine learning contexts, such as linear regression. However, fully explaining the scaling law in large practical models remains an elusive goal. In this work, we advance our understanding by demonstrating that the scaling law phenomenon extends to multiple regression and kernel regression settings, which are significantly more expressive and powerful than linear methods. Our analysis provides deeper insights into the scaling law, potentially enhancing our understanding of LLMs. Show Chinese abstract

Abstract: 最近，大型语言模型（LLMs）取得了显著的成功。 成功的关键因素是OpenAI观察到的缩放定律。 具体而言，对于具有Transformer架构的模型，测试损失与模型大小、数据集大小以及训练中使用的计算量之间呈现出幂律关系，显示出跨越七个数量级以上的趋势。 这一缩放定律挑战了传统的机器学习智慧，尤其是奥斯卡剪刀原理，该原理表明过度参数化的算法会对训练数据集过拟合，从而导致测试性能不佳。 最近的研究还发现在更简单的机器学习情境中存在缩放定律，例如线性回归。 然而，完全解释大型实际模型中的缩放定律仍然是一个难以实现的目标。 在本工作中，我们通过证明缩放定律现象扩展到多种回归和核回归设置，这些设置比线性方法更具表现力和强大，从而推进了我们的理解。 我们的分析提供了对缩放定律的更深入见解，可能有助于增强我们对LLMs的理解。