标题： 密集和稀疏大语言模型的缩放定律推广 显示英文标题

标题： Generalizing Scaling Laws for Dense and Sparse Large Language Models

摘要： 在过去几年中，语言模型的规模呈指数级增长，训练这些大型模型的计算成本也随之增加。这种快速增长促使研究人员开发新的技术，以提高训练过程的效率。尽管取得了这些进展，但最优地预测模型规模或分配最优资源仍然是一个挑战。一些研究通过提出不同的缩放定律来解决这一挑战，但几乎所有这些定律都是针对特定架构的（密集或稀疏）。在本工作中，我们重新审视现有的缩放定律，并提出一种通用的缩放定律，以提供一个适用于密集和稀疏大型语言模型的统一框架。我们评估并比较了所提出的缩放定律与现有缩放定律，以证明其有效性。 显示英文摘要

摘要： Over the past few years, the size of language models has grown exponentially, as has the computational cost to train these large models. This rapid growth has motivated researchers to develop new techniques aimed at enhancing the efficiency of the training process. Despite these advancements, optimally predicting the model size or allocating optimal resources remains a challenge. Several efforts have addressed the challenge by proposing different scaling laws, but almost all of them are architecture-specific (dense or sparse). In this work we revisit existing scaling laws and propose a generalized scaling law to provide a unified framework that is applicable to both dense and sparse large language models. We evaluate and compare our proposed scaling law with existing scaling laws to demonstrate its effectiveness.