标题： OT-Transformer：具有最优传输正则化的连续时间Transformer架构 显示英文标题

标题： OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization

摘要： Transformer在众多任务中取得了最先进的性能。 在本文中，我们提出了Transformer的连续时间公式。 具体来说，我们考虑一个由Transformer块参数化的动力系统。 我们利用最优传输理论来正则化训练问题，这增强了训练的稳定性并提高了模型的泛化能力。 此外，我们在理论上证明了这种正则化是必要的，因为它促进了解的唯一性和规则性。 我们的模型具有灵活性，几乎任何现有的Transformer架构都可以被采用来构建动力系统，只需对现有代码进行轻微修改。 我们在受自然语言处理、图像分类和点云分类启发的任务上进行了广泛的数值实验。 我们的实验结果表明，所提出的方法改进了其离散对应方法的性能，并优于相关比较模型。 显示英文摘要

摘要： Transformers have achieved state-of-the-art performance in numerous tasks. In this paper, we propose a continuous-time formulation of transformers. Specifically, we consider a dynamical system whose governing equation is parametrized by transformer blocks. We leverage optimal transport theory to regularize the training problem, which enhances stability in training and improves generalization of the resulting model. Moreover, we demonstrate in theory that this regularization is necessary as it promotes uniqueness and regularity of solutions. Our model is flexible in that almost any existing transformer architectures can be adopted to construct the dynamical system with only slight modifications to the existing code. We perform extensive numerical experiments on tasks motivated by natural language processing, image classification, and point cloud classification. Our experimental results show that the proposed method improves the performance of its discrete counterpart and outperforms relevant comparing models.