标题： EE-Tuning：一种经济且可扩展的早期退出大型语言模型调优方案 显示英文标题

标题： EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models

摘要： 这项工作介绍了EE-Tuning，这是一种轻量且经济的解决方案，用于训练/调整早期退出大型语言模型（LLMs）。 与常见的全参数预训练方法不同，EE-Tuning通过添加额外的早期退出层来增强任何预训练（可能经过微调）的标准LLM，并以参数高效的方式进行调整，这需要显著更少的计算资源和训练数据。 我们的EE-Tuning实现通过广泛的性能优化实现了出色的训练效率，并由于其与3D并行性的完全兼容性而具有可扩展性。 系统实验的结果验证了EE-Tuning的有效性，确认在有限的训练预算下可以实现有效的早期退出LLM推理。 希望使早期退出LLMs对社区可用，我们将在https://github.com/pan-x-c/EE-LLM发布我们的EE-Tuning实现的源代码。 显示英文摘要

摘要： This work introduces EE-Tuning, a lightweight and economical solution to training/tuning early-exit large language models (LLMs). In contrast to the common approach of full-parameter pre-training, EE-Tuning augments any pre-trained (and possibly fine-tuned) standard LLM with additional early-exit layers that are tuned in a parameter-efficient manner, which requires significantly less computational resources and training data. Our implementation of EE-Tuning achieves outstanding training efficiency via extensive performance optimizations, as well as scalability due to its full compatibility with 3D parallelism. Results of systematic experiments validate the efficacy of EE-Tuning, confirming that effective early-exit LLM inference can be achieved with a limited training budget. In hope of making early-exit LLMs accessible to the community, we release the source code of our implementation of EE-Tuning at https://github.com/pan-x-c/EE-LLM.