标题： 通过带有$L_1$先验的正则化Wasserstein邻近算子的稀疏Transformer架构 显示英文标题

标题： Sparse Transformer Architectures via Regularized Wasserstein Proximal Operator with $L_1$ Prior

摘要： 在本工作中，我们提出了一种稀疏变换器架构，该架构将关于底层数据分布的先验信息直接整合到神经网络的变换器结构中。 该模型的设计受到一个特殊的最优传输问题的启发，即正则化的Wasserstein近似算子，该算子具有闭式解，并被证明是变换器架构的一种特殊表示形式。 与经典的基于流的模型相比，所提出的方法改善了优化问题的凸性特性，并促进了生成样本的稀疏性。 通过理论分析和数值实验，包括在生成建模和贝叶斯逆问题中的应用，我们证明了稀疏变换器在达到目标分布方面比经典的神经ODE方法具有更高的准确性和更快的收敛速度。 显示英文摘要

摘要： In this work, we propose a sparse transformer architecture that incorporates prior information about the underlying data distribution directly into the transformer structure of the neural network. The design of the model is motivated by a special optimal transport problem, namely the regularized Wasserstein proximal operator, which admits a closed-form solution and turns out to be a special representation of transformer architectures. Compared with classical flow-based models, the proposed approach improves the convexity properties of the optimization problem and promotes sparsity in the generated samples. Through both theoretical analysis and numerical experiments, including applications in generative modeling and Bayesian inverse problems, we demonstrate that the sparse transformer achieves higher accuracy and faster convergence to the target distribution than classical neural ODE-based methods.