标题： 微调方法在弱到强泛化中的局限性 显示英文标题

标题： Limitations of refinement methods for weak to strong generalization

摘要： 对齐大型语言模型（LLMs）的标准技术利用人类生成的数据，这可能限制任何对齐的LLM的能力至人类水平。标签精炼和弱训练已成为解决这一超级对齐问题的有希望的策略。在本工作中，我们采用常用于研究标签精炼的概率假设，并分析精炼是否可以被其他方法超越，包括计算上不可行的oracle方法。我们表明，弱训练和标签精炼都存在不可减少的误差，导致标签精炼与oracle之间存在性能差距。这些结果促使未来的研究开发替代方法，以实现从弱到强的泛化，结合标签精炼或弱训练的实用性以及oracle过程的最优性。 显示英文摘要

摘要： Standard techniques for aligning large language models (LLMs) utilize human-produced data, which could limit the capability of any aligned LLM to human level. Label refinement and weak training have emerged as promising strategies to address this superalignment problem. In this work, we adopt probabilistic assumptions commonly used to study label refinement and analyze whether refinement can be outperformed by alternative approaches, including computationally intractable oracle methods. We show that both weak training and label refinement suffer from irreducible error, leaving a performance gap between label refinement and the oracle. These results motivate future research into developing alternative methods for weak to strong generalization that synthesize the practicality of label refinement or weak training and the optimality of the oracle procedure.