标题： 领域专业化与模型大小之间的相互作用 显示英文标题

标题： The interplay between domain specialization and model size

摘要： 语言模型的缩放定律通常关注从头开始训练时找到最佳模型大小和标记数量。 然而，由于从随机初始化的权重训练模型时需要大量数据，达到这种最佳平衡需要大量的计算资源。 持续预训练提供了一种成本效益更高的替代方案，利用预训练模型的计算投入来融入新知识，而无需大量新数据。 最近的研究表明，数据质量会影响缩放定律中的常数，从而改变最佳参数-标记分配比例。 基于这一见解，我们在计算受限的情况下，研究了持续预训练中领域专业化与模型大小之间的相互作用。 我们的目标是确定这种场景下的最佳训练制度，并检测这种相互作用中的模式，这些模式可以在不同模型大小和领域中进行推广。 为了比较通用和专业训练，我们过滤了一个基于网络的数据集，以提取来自三个领域：法律、医学和会计的数据。 我们在未过滤和过滤后的数据集上对15亿、30亿、70亿和140亿参数的模型进行了预训练，然后在领域特定的考试上评估了它们的性能。 结果表明，随着模型规模的增加，专业模型的表现优于通用模型，同时所需的训练计算更少。 此外，它们日益提高的计算效率导致了之前学到的知识遗忘减少。 显示英文摘要

摘要： Scaling laws for language models have often focused on finding the optimal model size and token count for training from scratch. However, achieving this optimal balance requires significant compute resources due to the extensive data demands when training models from randomly-initialized weights. Continued pretraining offers a cost-effective alternative, leveraging the compute investment from pretrained models to incorporate new knowledge without requiring extensive new data. Recent findings suggest that data quality influences constants in scaling laws, thereby altering the optimal parameter-token allocation ratio. Building on this insight, we investigate the interplay between domain specialization and model size during continued pretraining under compute-constrained scenarios. Our goal is to identify an optimal training regime for this scenario and detect patterns in this interplay that can be generalized across different model sizes and domains. To compare general and specialized training, we filtered a web-based dataset to extract data from three domains: legal, medical, and accounting. We pretrained models with 1.5B, 3B, 7B, and 14B parameters on both the unfiltered and filtered datasets, then evaluated their performance on domain-specific exams. Results show that as model size increases, specialized models outperform general models while requiring less training compute. Additionally, their growing compute efficiency leads to reduced forgetting of previously learned knowledge.