Title: GANs Can Play Lottery Tickets Too Show Chinese title

Title: GANs 可以玩彩票投注

Abstract: Deep generative adversarial networks (GANs) have gained growing popularity in numerous scenarios, while usually suffer from high parameter complexities for resource-constrained real-world applications. However, the compression of GANs has less been explored. A few works show that heuristically applying compression techniques normally leads to unsatisfactory results, due to the notorious training instability of GANs. In parallel, the lottery ticket hypothesis shows prevailing success on discriminative models, in locating sparse matching subnetworks capable of training in isolation to full model performance. In this work, we for the first time study the existence of such trainable matching subnetworks in deep GANs. For a range of GANs, we certainly find matching subnetworks at 67%-74% sparsity. We observe that with or without pruning discriminator has a minor effect on the existence and quality of matching subnetworks, while the initialization weights used in the discriminator play a significant role. We then show the powerful transferability of these subnetworks to unseen tasks. Furthermore, extensive experimental results demonstrate that our found subnetworks substantially outperform previous state-of-the-art GAN compression approaches in both image generation (e.g. SNGAN) and image-to-image translation GANs (e.g. CycleGAN). Codes available at https://github.com/VITA-Group/GAN-LTH. Show Chinese abstract

Abstract: 深度生成对抗网络（GANs）在许多场景中越来越受欢迎，但通常由于参数复杂度高，难以应用于资源受限的现实世界应用。然而，GANs的压缩研究较少。一些工作表明，启发式地应用压缩技术通常会导致不令人满意的结果，这是由于GANs著名的训练不稳定性。同时，彩票假设在判别模型中表现出色，在定位能够独立训练到完整模型性能的稀疏匹配子网络方面表现突出。在这项工作中，我们首次研究了深度GANs中这种可训练匹配子网络的存在性。对于一系列GANs，我们确实在67%-74%的稀疏度下找到了匹配子网络。我们观察到，无论是否修剪判别器，对匹配子网络的存在性和质量影响较小，而判别器中使用的初始化权重起着重要作用。然后我们展示了这些子网络对未见过的任务的强大可迁移性。此外，广泛的实验结果表明，我们的发现的子网络在图像生成（例如SNGAN）和图像到图像翻译GANs（例如CycleGAN）方面显著优于以前最先进的GAN压缩方法。代码可在https://github.com/VITA-Group/GAN-LTH获取。