标题： 元稀疏性：通过元学习在多任务网络中学习最优稀疏结构 显示英文标题

标题： Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning

摘要： 本文提出了元稀疏性，这是一种学习模型稀疏性的框架，基本上是学习控制稀疏度的参数，使深度神经网络（DNN）在多任务学习（MTL）设置中能够自动生成最优的稀疏共享结构。 该方法允许在各种任务中动态学习稀疏模式，不同于传统稀疏方法严重依赖手动超参数调整。 受模型无关元学习（MAML）的启发，在元训练阶段通过基于惩罚的、通道级别的结构稀疏性来实现多任务场景下共享和最优稀疏参数的学习。 该方法通过移除不必要的参数提高了模型的有效性，并增强了其处理已见和未见过任务的能力。 通过在两个数据集NYU-v2和CelebAMask-HQ上进行的大量实验，严格评估了元稀疏性的有效性，涵盖了从像素级到图像级预测的各种任务。 结果表明，所提出的方法在许多任务中表现良好，表明其作为创建高效和适应性强的稀疏神经网络的多功能工具的潜力。 因此，这项工作提出了一种学习稀疏性的方法，为稀疏神经网络领域的研究做出了贡献，并指出了向简约模型方向研究的新方向。 显示英文摘要

摘要： This paper presents meta-sparsity, a framework for learning model sparsity, basically learning the parameter that controls the degree of sparsity, that allows deep neural networks (DNNs) to inherently generate optimal sparse shared structures in multi-task learning (MTL) setting. This proposed approach enables the dynamic learning of sparsity patterns across a variety of tasks, unlike traditional sparsity methods that rely heavily on manual hyperparameter tuning. Inspired by Model Agnostic Meta-Learning (MAML), the emphasis is on learning shared and optimally sparse parameters in multi-task scenarios by implementing a penalty-based, channel-wise structured sparsity during the meta-training phase. This method improves the model's efficacy by removing unnecessary parameters and enhances its ability to handle both seen and previously unseen tasks. The effectiveness of meta-sparsity is rigorously evaluated by extensive experiments on two datasets, NYU-v2 and CelebAMask-HQ, covering a broad spectrum of tasks ranging from pixel-level to image-level predictions. The results show that the proposed approach performs well across many tasks, indicating its potential as a versatile tool for creating efficient and adaptable sparse neural networks. This work, therefore, presents an approach towards learning sparsity, contributing to the efforts in the field of sparse neural networks and suggesting new directions for research towards parsimonious models.