标题： FlashSVD：低秩模型流式传输的内存高效推理 显示英文标题

标题： FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models

摘要： 奇异值分解（SVD）最近作为大规模语言模型（LLMs）压缩的简单而强大的工具引起了广泛关注，越来越多的研究表明，在最小精度损失的情况下可以实现20-80%的参数减少。 基于先前的SVD方法主要集中在减少模型权重的内存占用上，很大程度上忽略了在应用截断因子时通过标准密集CUDA内核进行推理时产生的额外激活内存开销。 我们的实验表明，这种与序列长度和隐藏维度成比例的激活开销，使得当前的SVD压缩技术无法实现峰值推理内存的任何减少，从而限制了它们在实际设备部署中的可行性。 我们引入了FlashSVD，这是一种新颖的端到端秩感知流式推理框架，专门针对SVD压缩的大规模语言模型设计。 FlashSVD可以无缝集成到任何使用基于SVD的方法进行参数减少的模型中。 通过将低秩投影内核直接融合到自注意力和前馈网络（FFN）管道中，FlashSVD避免生成全尺寸的激活缓冲区。 相反，截断因子的小块被加载到片上SRAM中，实时相乘和约简，并立即驱逐，保持高GPU利用率并增加额外延迟。 在标准编码器基准测试（例如，BERT-Base）中，FlashSVD最多可将峰值激活内存减少70.2%，中间瞬态内存减少75%，同时在上游压缩方法中不会造成精度损失，为低秩LLMs的内存受限部署提供了一条实用路径。 显示英文摘要

摘要： Singular Value Decomposition (SVD) has recently seen a surge of interest as a simple yet powerful tool for large language models (LLMs) compression, with a growing number of works demonstrating 20-80% parameter reductions at minimal accuracy loss. Previous SVD-based approaches have focused primarily on reducing the memory footprint of model weights, largely overlooking the additional activation memory overhead incurred during inference when applying truncated factors via standard dense CUDA kernels. Our experiments demonstrate that this activation overhead, scaling with sequence length and hidden dimension, prevents current SVD compression techniques from achieving any reduction in peak inference memory, thereby limiting their viability for real-world, on-device deployments. We introduce FlashSVD, a novel, end-to-end rank-aware streaming inference framework specifically designed for SVD-compressed large language models. FlashSVD can be seamlessly integrated with any model that employs SVD-based methods for parameter reduction. By fusing low-rank projection kernels directly into both the self-attention and feed-forward network (FFN) pipelines, FlashSVD avoid materializing full-size activation buffers. Instead, small tiles of the truncated factors are loaded into on-chip SRAM, multiplied and reduced on the fly, and immediately evicted, preserving high GPU occupancy and adding no extra latency. On standard encoder benchmarks (e.g., BERT-Base), FlashSVD cuts peak activation memory by up to 70.2% and intermediate transient memory by 75%, all while incur no accuracy loss with upstreaming compression methods, offering a practical path toward memory-constrained deployment of low-rank LLMs.