标题： 大型语言模型的指令跟随剪枝 显示英文标题

标题： Instruction-Following Pruning for Large Language Models

摘要： 随着大型语言模型（LLMs）的快速发展，结构化剪枝已成为一种广泛使用的技术，用于从更大的模型中学习更高效、更小的模型，并且与从头开始训练同样大小的模型相比，表现出色。 在本文中，我们超越了传统静态剪枝方法，即为模型确定固定剪枝掩码的方法，提出了一种动态的结构化剪枝方法。 在我们的方法中，剪枝掩码是输入相关的，并根据用户指令描述的信息动态调整。 我们称之为“指令跟随剪枝”的方法引入了一个稀疏掩码预测器，该预测器以用户指令作为输入，并动态选择与给定任务最相关的模型参数。 为了识别和激活有效的参数，我们联合优化稀疏掩码预测器和LLM，利用指令跟随数据和预训练语料库。 实验结果表明，我们的方法在广泛的评估基准上表现出了有效性。 例如，我们的3B激活模型在数学和编码等领域比3B密集模型提高了5到8个百分点，并且与9B模型的表现相当。 显示英文摘要

摘要： With the rapid scaling of large language models (LLMs), structured pruning has become a widely used technique to learn efficient, smaller models from larger ones, delivering superior performance compared to training similarly sized models from scratch. In this paper, we move beyond the traditional static pruning approach of determining a fixed pruning mask for a model, and propose a dynamic approach to structured pruning. In our method, the pruning mask is input-dependent and adapts dynamically based on the information described in a user instruction. Our approach, termed "instruction-following pruning", introduces a sparse mask predictor that takes the user instruction as input and dynamically selects the most relevant model parameters for the given task. To identify and activate effective parameters, we jointly optimize the sparse mask predictor and the LLM, leveraging both instruction-following data and the pre-training corpus. Experimental results demonstrate the effectiveness of our approach on a wide range of evaluation benchmarks. For example, our 3B activated model improves over the 3B dense model by 5-8 points of absolute margin on domains such as math and coding, and rivals the performance of a 9B model.