标题： 理解Transformer在序列建模中的表达能力和机制 显示英文标题

标题： Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling

摘要： 我们对Transformer在长、稀疏和复杂记忆的序列建模中的近似性质进行了系统研究。 我们研究了Transformer的不同组件，如点积自注意力、位置编码和前馈层，是如何影响其表达能力的，并通过建立明确的近似速率来研究它们的综合影响。 我们的研究揭示了Transformer中关键参数的作用，如层数和注意力头的数量，这些见解也为替代架构提供了自然的建议。 显示英文摘要

摘要： We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the dot-product self-attention, positional encoding and feed-forward layer, affect its expressive power, and we study their combined effects through establishing explicit approximation rates. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads, and these insights also provide natural suggestions for alternative architectures.