标题： 关于模型压缩中的信息几何和迭代优化：算子分解 显示英文标题

标题： On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization

摘要： 深度学习模型的参数数量不断增加，这需要有效的压缩技术以便在资源受限的设备上部署。 本文探讨了信息几何学在模型压缩空间中现有方法的应用，信息几何学是研究参数空间上的密度诱导度量的学科，主要关注操作符分解。 采用这种观点突出了核心挑战：定义一个最优的低计算子流形（或子集）并将其投影到该子流形上。 我们认为，许多成功的模型压缩方法可以被理解为对此投影的隐式近似信息散度。 我们指出，在压缩预训练模型时，使用信息散度对于实现改进的零样本准确率至关重要，但当模型进行微调时，这种情况可能不再成立。 在这些情况下，瓶颈模型的可训练性对于实现高压缩比且性能下降最小更为重要，这需要采用迭代方法。 在此背景下，我们证明了在软秩约束下训练神经网络的迭代奇异值阈值法的收敛性。 为了进一步说明这一观点的实用性，我们展示了通过更柔和的秩减少对现有方法进行简单修改，可以在固定压缩率下实现更好的性能。 显示英文摘要

摘要： The ever-increasing parameter counts of deep learning models necessitate effective compression techniques for deployment on resource-constrained devices. This paper explores the application of information geometry, the study of density-induced metrics on parameter spaces, to analyze existing methods within the space of model compression, primarily focusing on operator factorization. Adopting this perspective highlights the core challenge: defining an optimal low-compute submanifold (or subset) and projecting onto it. We argue that many successful model compression approaches can be understood as implicitly approximating information divergences for this projection. We highlight that when compressing a pre-trained model, using information divergences is paramount for achieving improved zero-shot accuracy, yet this may no longer be the case when the model is fine-tuned. In such scenarios, trainability of bottlenecked models turns out to be far more important for achieving high compression ratios with minimal performance degradation, necessitating adoption of iterative methods. In this context, we prove convergence of iterative singular value thresholding for training neural networks subject to a soft rank constraint. To further illustrate the utility of this perspective, we showcase how simple modifications to existing methods through softer rank reduction result in improved performance under fixed compression rates.