Title: Model Preserving Compression for Neural Networks Show Chinese title

Title: 模型保持压缩 for 神经网络

Abstract: After training complex deep learning models, a common task is to compress the model to reduce compute and storage demands. When compressing, it is desirable to preserve the original model's per-example decisions (e.g., to go beyond top-1 accuracy or preserve robustness), maintain the network's structure, automatically determine per-layer compression levels, and eliminate the need for fine tuning. No existing compression methods simultaneously satisfy these criteria $\unicode{x2014}$ we introduce a principled approach that does by leveraging interpolative decompositions. Our approach simultaneously selects and eliminates channels (analogously, neurons), then constructs an interpolation matrix that propagates a correction into the next layer, preserving the network's structure. Consequently, our method achieves good performance even without fine tuning and admits theoretical analysis. Our theoretical generalization bound for a one layer network lends itself naturally to a heuristic that allows our method to automatically choose per-layer sizes for deep networks. We demonstrate the efficacy of our approach with strong empirical performance on a variety of tasks, models, and datasets $\unicode{x2014}$ from simple one-hidden-layer networks to deep networks on ImageNet. Show Chinese abstract

Abstract: 在训练复杂的深度学习模型后，常见的任务是压缩模型以减少计算和存储需求。 在压缩时，希望保留原始模型的每个示例的决策（例如，超越top-1准确率或保持鲁棒性），保持网络结构，自动确定每层的压缩级别，并消除微调的需要。 现有的压缩方法没有同时满足这些标准$\unicode{x2014}$我们引入了一种有原则的方法，通过利用插值分解来实现。 我们的方法同时选择和消除通道（类似地，神经元），然后构建一个插值矩阵，将校正传播到下一层，从而保持网络结构。 因此，即使不进行微调，我们的方法也能取得良好的性能，并且可以进行理论分析。 我们对单层网络的理论泛化界限自然地引出了一个启发式方法，使我们的方法能够自动为深度网络选择每层的大小。 我们在各种任务、模型和数据集上展示了我们方法的有效性，$\unicode{x2014}$从简单的单隐藏层网络到ImageNet上的深度网络。