Title: C2A: Client-Customized Adaptation for Parameter-Efficient Federated Learning Show Chinese title

Title: C2A：面向参数高效联邦学习的客户端定制适应

Abstract: Despite the versatility of pre-trained language models (PLMs) across domains, their large memory footprints pose significant challenges in federated learning (FL), where the training model has to be distributed between a server and clients. One potential solution to bypass such constraints might be the use of parameter-efficient fine-tuning (PEFT) in the context of FL. However, we have observed that typical PEFT tends to severely suffer from heterogeneity among clients in FL scenarios, resulting in unstable and slow convergence. In this paper, we propose Client-Customized Adaptation (C2A), a novel hypernetwork-based FL framework that generates client-specific adapters by conditioning the client information. With the effectiveness of the hypernetworks in generating customized weights through learning to adopt the different characteristics of inputs, C2A can maximize the utility of shared model parameters while minimizing the divergence caused by client heterogeneity. To verify the efficacy of C2A, we perform extensive evaluations on FL scenarios involving heterogeneity in label and language distributions. Comprehensive evaluation results clearly support the superiority of C2A in terms of both efficiency and effectiveness in FL scenarios. Show Chinese abstract

Abstract: 尽管预训练语言模型（PLMs）在不同领域具有多功能性，但其较大的内存占用在联邦学习（FL）中带来了重大挑战，因为在FL中，训练模型必须在服务器和客户端之间进行分发。 在FL背景下，使用参数高效的微调（PEFT）可能是绕过这些限制的一种潜在解决方案。 然而，我们观察到典型的PEFT在FL场景中容易受到客户端异质性的严重影响，导致不稳定和缓慢的收敛。 在本文中，我们提出了客户定制适应（C2A），这是一种基于超网络的新型FL框架，通过条件化客户端信息生成特定于客户端的适配器。 由于超网络在通过学习采用不同输入特征来生成定制权重方面的有效性，C2A可以在最大化共享模型参数效用的同时，最小化由客户端异质性引起的差异。 为了验证C2A的有效性，我们在涉及标签和语言分布异质性的FL场景中进行了广泛的评估。 全面的评估结果明确支持C2A在FL场景中的效率和效果方面的优越性。