标题： 学习跳过Transformer的中间层 显示英文标题

标题： Learning to Skip the Middle Layers of Transformers

摘要： 条件计算是一种使Transformer更高效的流行策略。 现有方法通常针对单独的模块（例如，专家混合层）或独立地跳过层。 然而，可解释性研究已经表明，Transformer的中间层表现出更大的冗余性，并且早期层将信息聚合到标记位置。 受这些见解的启发，我们提出了一种新的架构，该架构动态地从中间向外跳过可变数量的层。 具体来说，一个学习到的门控机制根据输入决定是否绕过对称的中心块区域，并且一个门控注意力机制防止后续标记访问被跳过的标记位置。 残差范数通过“夹心”或“perilayernorm”方案进行控制，并通过自适应正则化损失控制门控稀疏性。 我们的目标是减少“简单”标记的计算需求，并可能促进出现多级表示层次结构，但就所研究的规模而言，与具有较少层的密集基线相比，我们的方法在验证交叉熵和估计FLOPs之间的权衡上并未取得改进。 我们在https://github.com/tim-lawson/skip-middle发布我们的代码。 显示英文摘要

摘要： Conditional computation is a popular strategy to make Transformers more efficient. Existing methods often target individual modules (e.g., mixture-of-experts layers) or skip layers independently of one another. However, interpretability research has demonstrated that the middle layers of Transformers exhibit greater redundancy, and that early layers aggregate information into token positions. Guided by these insights, we propose a novel architecture that dynamically skips a variable number of layers from the middle outward. In particular, a learned gating mechanism determines whether to bypass a symmetric span of central blocks based on the input, and a gated attention mechanism prevents subsequent tokens from attending to skipped token positions. Residual norms are controlled with a 'sandwich' or 'perilayernorm' scheme and gate sparsity with an adaptive regularization loss. We had aimed to reduce compute requirements for 'simpler' tokens and potentially foster an emergent multi-level representational hierarchy but, at the scales investigated, our approach does not achieve improvements in the trade-off between validation cross-entropy and estimated FLOPs compared to dense baselines with fewer layers. We release our code at https://github.com/tim-lawson/skip-middle.