标题： 差分隐私联邦低秩适应超越固定矩阵 显示英文标题

标题： Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix

摘要： 大型语言模型（LLMs）通常需要微调以适应特定领域任务，而LoRA通过训练低秩适配器提供了一种计算高效的解决方案。 LoRA在联邦学习中也具有通信效率，当多个用户协作微调全局LLM模型而不共享其专有原始数据时。 然而，即使在服务器和客户端之间传输本地适配器，也会造成严重的隐私泄露风险。 将差分隐私（DP）应用于联邦LoRA面临两难：对两个适配器添加噪声会放大模型上的合成噪声，而固定一个适配器会损害微调的学习能力。 在本文中，我们提出了FedASK（具有双重草图的差分隐私联邦低秩适配）一种新颖的联邦LoRA框架，以实现对两个低秩适配器的有效更新并具备强大的差分隐私。 受随机SVD的启发，我们的核心思想是一个两阶段的草图管道。 该管道首先聚合经过精心草图处理的、保护隐私的本地更新，然后在服务器上重建全局矩阵，以促进两个适配器的有效更新。 我们理论上证明了FedASK的差分隐私保证及其精确聚合特性。 全面的实验表明，FedASK在各种隐私设置和数据分布下始终优于基线方法。 显示英文摘要

摘要： Large language models (LLMs) typically require fine-tuning for domain-specific tasks, and LoRA offers a computationally efficient approach by training low-rank adapters. LoRA is also communication-efficient for federated LLMs when multiple users collaboratively fine-tune a global LLM model without sharing their proprietary raw data. However, even the transmission of local adapters between a server and clients risks serious privacy leakage. Applying differential privacy (DP) to federated LoRA encounters a dilemma: adding noise to both adapters amplifies synthetic noise on the model, while fixing one adapter impairs the learnability of fine-tuning. In this paper, we propose FedASK (Differentially Private Federated Low Rank Adaptation with Double Sketching) , a novel federated LoRA framework to enable effective updating of both low-rank adapters with robust differential privacy. Inspired by randomized SVD, our key idea is a two-stage sketching pipeline. This pipeline first aggregates carefully sketched, privacy-preserving local updates, and then reconstructs the global matrices on the server to facilitate effective updating of both adapters. We theoretically prove FedASK's differential privacy guarantee and its exact aggregation property. Comprehensive experiments demonstrate that FedASK consistently outperforms baseline methods across a variety of privacy settings and data distributions.