标题： 分布式动量用于拜占庭容错学习 显示英文标题

标题： Distributed Momentum for Byzantine-resilient Learning

摘要： 动量是梯度下降的一种变体，因其在收敛方面的优势而被提出。 在分布式设置中，动量可以在服务器端或工作器端实现。 当服务器使用的聚合规则是线性的时，加法的交换性使得这两种部署方式等价。 然而，鲁棒性和隐私性是放弃线性聚合规则的动机之一。 在本工作中，我们展示了在工作器端使用动量对鲁棒性的优势。 我们首先证明，在工作器端计算动量可以降低服务器端梯度估计的方差-范数比，从而增强拜占庭容错的聚合规则。 然后，我们提供了对分布式SGD中工作器端动量鲁棒性效果的广泛实验演示。 显示英文摘要

摘要： Momentum is a variant of gradient descent that has been proposed for its benefits on convergence. In a distributed setting, momentum can be implemented either at the server or the worker side. When the aggregation rule used by the server is linear, commutativity with addition makes both deployments equivalent. Robustness and privacy are however among motivations to abandon linear aggregation rules. In this work, we demonstrate the benefits on robustness of using momentum at the worker side. We first prove that computing momentum at the workers reduces the variance-norm ratio of the gradient estimation at the server, strengthening Byzantine resilient aggregation rules. We then provide an extensive experimental demonstration of the robustness effect of worker-side momentum on distributed SGD.