标题： OpenCSG 中文语料库：一系列用于 LLM 训练的高质量中文数据集 显示英文标题

标题： OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training

摘要： 大型语言模型（LLMs）已经展示了惊人的能力，但它们的成功在很大程度上依赖于预训练语料库的质量。 对于中文的LLMs，高质量中文数据集的缺乏是一个重大挑战，常常限制了它们的性能。 为了解决这个问题，我们提出了OpenCSG中文语料库，这是一个专门为LLM预训练、后训练和微调设计的一系列高质量数据集。 该语料库包括Fineweb-edu-chinese、Fineweb-edu-chinese-v2、Cosmopedia-chinese和Smoltalk-chinese，每个数据集都有其独特的特点：Fineweb-edu数据集专注于从多样化的中文网络来源中过滤出的高质量内容；Cosmopedia-chinese提供合成的、教科书风格的数据，用于知识密集型训练；而Smoltalk-chinese则强调风格多样且格式为聊天的数据。 OpenCSG中文语料库的特点是高质量的文本、跨领域的广泛覆盖以及可扩展且可重复的数据整理过程。 此外，我们进行了广泛的实验分析，包括对较小参数模型的评估，这些评估在C-Eval等任务中展示了显著的性能提升，证明了该语料库在训练中文LLMs方面的有效性。 显示英文摘要

摘要： Large language models (LLMs) have demonstrated remarkable capabilities, but their success heavily relies on the quality of pretraining corpora. For Chinese LLMs, the scarcity of high-quality Chinese datasets presents a significant challenge, often limiting their performance. To address this issue, we propose the OpenCSG Chinese Corpus, a series of high-quality datasets specifically designed for LLM pretraining, post-training, and fine-tuning. This corpus includes Fineweb-edu-chinese, Fineweb-edu-chinese-v2, Cosmopedia-chinese, and Smoltalk-chinese, each with distinct characteristics: Fineweb-edu datasets focus on filtered, high-quality content derived from diverse Chinese web sources; Cosmopedia-chinese provides synthetic, textbook-style data for knowledge-intensive training; and Smoltalk-chinese emphasizes stylistic and diverse chat-format data. The OpenCSG Chinese Corpus is characterized by its high-quality text, diverse coverage across domains, and scalable, reproducible data curation processes. Additionally, we conducted extensive experimental analyses, including evaluations on smaller parameter models, which demonstrated significant performance improvements in tasks such as C-Eval, showcasing the effectiveness of the corpus for training Chinese LLMs.