Title: The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models Show Chinese title

Title: 压缩的成本：研究压缩对语言模型中参数知识的影响

Abstract: Compressing large language models (LLMs), often consisting of billions of parameters, provides faster inference, smaller memory footprints, and enables local deployment. Two standard compression techniques are pruning and quantization, with the former eliminating redundant connections in model layers and the latter representing model parameters with fewer bits. The key tradeoff is between the degree of compression and the impact on the quality of the compressed model. Existing research on LLM compression primarily focuses on performance in terms of general metrics like perplexity or downstream task accuracy. More fine-grained metrics, such as those measuring parametric knowledge, remain significantly underexplored. To help bridge this gap, we present a comprehensive analysis across multiple model families (ENCODER, ENCODER-DECODER, and DECODER) using the LAMA and LM-HARNESS benchmarks in order to systematically quantify the effect of commonly employed compression techniques on model performance. A particular focus is on tradeoffs involving parametric knowledge, with the goal of providing practitioners with practical insights to help make informed decisions on compression. We release our codebase1 to enable further research. Show Chinese abstract

Abstract: 压缩大型语言模型（LLMs），这些模型通常由数十亿个参数组成，可以提供更快的推理速度、更小的内存占用，并支持本地部署。 两种标准的压缩技术是剪枝和量化，前者消除模型层中的冗余连接，后者用更少的位数表示模型参数。 关键的权衡在于压缩程度与对压缩后模型质量的影响之间。 现有的LLM压缩研究主要集中在通用指标如困惑度或下游任务准确率方面的性能上。 更细粒度的指标，例如衡量参数知识的指标，仍远未得到充分探索。 为了帮助弥合这一差距，我们使用LAMA和LM-HARNESS基准对多个模型家族（ENCODER、ENCODER-DECODER和DECODER）进行了全面分析，以系统地量化常用压缩技术对模型性能的影响。 特别关注涉及参数知识的权衡，旨在为从业者提供实用的见解，帮助他们在压缩方面做出明智的决策。 我们发布了我们的代码库1以促进进一步的研究。