标题： 螺旋管道：使用注意力并行管道并行化高效训练长序列变压器 显示英文标题

标题： HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism

摘要： 随着Transformer序列长度的增长，现有的管道并行方法由于二次注意力计算和显著的内存开销而表现出次优性能。 为缓解这些挑战，我们提出了HelixPipe，一种用于长序列Transformer训练的新管道并行方法。 首先， HelixPipe引入了注意力并行划分，将在不同微批次上的注意力计算并行地调度到不同的管道阶段，减少管道气泡。 其次，它采用两步先进后出的微批次调度方式，平衡内存使用并实现通信与计算的重叠。 此外，HelixPipe利用无注意力的重新计算和分块MLP来减轻碎片化并支持更长的序列。 实验表明，随着序列长度的增加，HelixPipe的优势逐渐显现，并在不同管道规模、模型规模和集群配置下，在吞吐量和可扩展性方面优于现有方法。 值得注意的是，当在64块H20 GPU上训练一个7B模型且序列长度为128k时，其速度比基线方法提高了26%。 代码可在https://github.com/code-tunnel/Megatron-LM/tree/dev获取。 显示英文摘要

摘要： As transformer sequence lengths grow, existing pipeline parallelisms incur suboptimal performance due to the quadratic attention computation and the substantial memory overhead. To relieve these challenges, we propose HelixPipe, a novel pipeline parallelism for long sequence transformer training. First, HelixPipe introduces attention parallel partition, which schedules attention computations of different micro batches across different pipeline stages in parallel, reducing pipeline bubbles. Second, it employs a two-fold first-in-last-out micro batch schedule to balance memory usage and overlap communication with computation. Additionally, HelixPipe utilizes recomputation without attention and chunked MLP to mitigate fragmentation and enable longer sequences. Experiments demonstrate that HelixPipe gains increasing advantages with longer sequence lengths, and outperforms existing methods in throughput and scalability across varying pipeline sizes, model sizes, and cluster configurations. Notably, it achieves a 26\% speedup over baseline methods when training a 7B model with 128k sequence length on 64 H20 GPUs. Code is available at https://github.com/code-tunnel/Megatron-LM/tree/dev.