标题： StagFormer：用于并行运行层的时间错位Transformer解码 显示英文标题

标题： StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel

摘要： 基于Transformer的语言模型中的标准解码过程本质上是顺序的，因为我们必须等待一个标记的嵌入通过网络中的所有层之后，才能开始生成下一个标记。 在本工作中，我们提出了一种新的架构StagFormer（交错Transformer），它沿时间轴交错执行，从而实现了在模型深度上并行化解码过程。 我们通过打破第$l$层中时间步$i$的标记表示对来自第$l-1$层直到时间步$i$的标记表示的依赖性来实现这一点。 相反，我们交错执行，并且只允许对直到时间步$i-1$的标记表示的依赖性。 Transformer的后续部分仍然可以访问前面部分的“丰富”表示，但仅限于那些滞后一个时间步的标记位置。 StagFormer允许模型的不同部分并行执行，在我们的模拟中解码速度潜在提升了33%而质量保持不变。 我们还探索了这种想法的许多自然变体。 我们展示了在内存有限的设置中，不同交错部分之间的权重共享如何更加实用。 我们展示了如何使用这种权重共享在推理过程中近似一个循环模型。 我们探讨了使用有限窗口注意力在不同部分之间传递信息的有效性，这有助于某些应用进一步降低延迟。 我们还探索并展示了这种交错思想在超过2个Transformer部分上的可扩展性。 显示英文摘要

摘要： Standard decoding in a Transformer based language model is inherently sequential as we wait for a token's embedding to pass through all the layers in the network before starting the generation of the next token. In this work, we propose a new architecture StagFormer (Staggered Transformer), which staggered execution along the time axis and thereby enables parallelizing the decoding process along the depth of the model. We achieve this by breaking the dependency of the token representation at time step $i$ in layer $l$ upon the representations of tokens until time step $i$ from layer $l-1$. Instead, we stagger the execution and only allow a dependency on token representations until time step $i-1$. The later sections of the Transformer still get access to the ``rich" representations from the prior section but only from those token positions which are one time step behind. StagFormer allows for different sections of the model to be executed in parallel yielding at potential 33\% speedup in decoding while being quality neutral in our simulations. We also explore many natural variants of this idea. We present how weight-sharing across the different sections being staggered can be more practical in settings with limited memory. We show how one can approximate a recurrent model during inference using such weight-sharing. We explore the efficacy of using a bounded window attention to pass information from one section to another which helps drive further latency gains for some applications. We also explore demonstrate the scalability of the staggering idea over more than 2 sections of the Transformer.