标题： 一种结构无关的多目标权重共享压缩方法 显示英文标题

标题： A Novel Structure-Agnostic Multi-Objective Approach for Weight-Sharing Compression in Deep Neural Networks

摘要： 深度神经网络在训练后会在内存中存储数百万甚至数十亿的权重，使得在嵌入式设备上部署内存密集型模型变得具有挑战性。 权重共享技术是一种流行的压缩方法，它使用更少的权重值并在网络中的特定连接之间共享。 在本文中，我们提出了一种基于多目标进化算法（MOEA）的压缩框架，该框架独立于神经网络架构、维度、任务和数据集。 我们使用大小相同的桶将网络权重量化为一个单一的代码本（查找表），以实现高效的权重表示。 使用MOEA，我们通过优化两个目标来搜索帕累托最优的$k$个桶。 然后，我们通过迭代合并技术对非支配帕累托前沿解进行处理，通过组合相邻的桶而不降低性能来减少桶的数量并提高压缩比。 我们的方法与模型和层无关，这意味着权重来自任何层的聚类中混合，本文中使用的统一量化方法具有$O(N)$复杂度，而不是像k-means这样的非统一量化方法，其复杂度为$O(Nkt)$。 此外，我们使用聚类中心作为共享权重值，而不是重新训练共享权重，这在计算上是昂贵的。 使用进化多目标优化的优势在于，它可以相对于性能和共享权重获得非支配的帕累托前沿解。 实验结果表明，我们可以将神经网络内存减少$13.72 \sim14.98 \times$在CIFAR-10上，$11.61 \sim 12.99\times$在CIFAR-100上，以及$7.44 \sim 8.58\times$在ImageNet上，展示了所提出的深度神经网络压缩框架的有效性。 显示英文摘要

摘要： Deep neural networks suffer from storing millions and billions of weights in memory post-training, making challenging memory-intensive models to deploy on embedded devices. The weight-sharing technique is one of the popular compression approaches that use fewer weight values and share across specific connections in the network. In this paper, we propose a multi-objective evolutionary algorithm (MOEA) based compression framework independent of neural network architecture, dimension, task, and dataset. We use uniformly sized bins to quantize network weights into a single codebook (lookup table) for efficient weight representation. Using MOEA, we search for Pareto optimal $k$ bins by optimizing two objectives. Then, we apply the iterative merge technique to non-dominated Pareto frontier solutions by combining neighboring bins without degrading performance to decrease the number of bins and increase the compression ratio. Our approach is model- and layer-independent, meaning the weights are mixed in the clusters from any layer, and the uniform quantization method used in this work has $O(N)$ complexity instead of non-uniform quantization methods such as k-means with $O(Nkt)$ complexity. In addition, we use the center of clusters as the shared weight values instead of retraining shared weights, which is computationally expensive. The advantage of using evolutionary multi-objective optimization is that it can obtain non-dominated Pareto frontier solutions with respect to performance and shared weights. The experimental results show that we can reduce the neural network memory by $13.72 \sim14.98 \times$ on CIFAR-10, $11.61 \sim 12.99\times$ on CIFAR-100, and $7.44 \sim 8.58\times$ on ImageNet showcasing the effectiveness of the proposed deep neural network compression framework.