标题： 变色龙：一种灵活的数据混合框架，用于语言模型的预训练和微调 显示英文标题

标题： Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning

摘要： 训练数据混合方式对大型语言模型的泛化性能有重大影响。现有的领域重新加权方法通常依赖于昂贵的权重计算，并且在引入新数据时需要重新训练。为此，我们引入了一个灵活高效的混合框架——变色龙（Chameleon），它利用杠杆分数来量化学习嵌入空间中的领域重要性。我们首先在领域嵌入上构建一个领域亲和矩阵。由此产生的杠杆分数决定了一个混合方式，该方式会增加共享嵌入空间中共同表示的领域的权重。这种公式允许通过计算新的领域嵌入直接转移到新数据。在实验中，我们在三个关键场景中展示了改进：(i) 我们计算出的权重以现有方法的一小部分计算成本提高了预训练领域的性能；(ii) 变色龙可以适应数据变化而无需代理重新训练，在转移到新数据时提高了少量样本推理的准确性；(iii) 我们的方法在微调中实现了高效的领域重新加权，一致地降低了所有微调领域的测试困惑度。我们的代码可在 https://github.com/LIONS-EPFL/Chameleon 获取。 显示英文摘要

摘要： Training data mixtures greatly impact the generalization performance of large language models. Existing domain reweighting methods often rely on costly weight computations and require retraining when new data is introduced. To this end, we introduce a flexible and efficient data mixing framework, Chameleon, that employs leverage scores to quantify domain importance within a learned embedding space. We first construct a domain affinity matrix over domain embeddings. The induced leverage scores determine a mixture that upweights domains sharing common representations in embedding space. This formulation allows direct transfer to new data by computing the new domain embeddings. In experiments, we demonstrate improvements over three key scenarios: (i) our computed weights improve performance on pretraining domains with a fraction of the compute of existing methods; (ii) Chameleon can adapt to data changes without proxy retraining, boosting few-shot reasoning accuracies when transferred to new data; (iii) our method enables efficient domain reweighting in finetuning, consistently improving test perplexity on all finetuning domains over uniform mixture. Our code is available at https://github.com/LIONS-EPFL/Chameleon.