Title: FlashNorm: fast normalization for LLMs Show Chinese title

Title: FlashNorm：用于LLM的快速归一化

Abstract: This paper presents FlashNorm, which is an exact but faster implementation of RMSNorm followed by linear layers. RMSNorm is used by many LLMs such as Llama, Mistral, and OpenELM. FlashNorm also speeds up Layer Normalization and its recently proposed replacement Dynamic Tanh (DyT) arXiv:2503.10622. FlashNorm also reduces the number of parameter tensors by simply merging the normalization weights with the weights of the next linear layer. See https://github.com/OpenMachine-ai/transformer-tricks for code and more transformer tricks. Show Chinese abstract

Abstract: 本文介绍了 FlashNorm，它是一种精确但更快的 RMSNorm 实现，随后是线性层。许多大型语言模型（LLMs）如 Llama、Mistral 和 OpenELM 都使用了 RMSNorm。FlashNorm 还加速了层归一化及其最近提出的替代方案动态双曲正切（DyT，arXiv:2503.10622）。FlashNorm 通过简单地将归一化权重与下一个线性层的权重合并，减少了参数张量的数量。更多代码和变换器技巧可参见 https://github.com/OpenMachine-ai/transformer-tricks。