标题： 带有平滑敏感度的差分隐私深度学习 显示英文标题

标题： Differentially Private Deep Learning with Smooth Sensitivity

摘要： 确保用于训练现代机器学习模型的敏感数据的隐私在许多实践领域中至关重要。 研究这些担忧的一种方法是通过差分隐私的视角。 在此框架中，隐私保证通常是通过扰动模型来实现的，使得用于训练模型的数据的具体信息变得模糊。 这种方法的一个具体实例是通过“教师-学生”框架，其中拥有敏感数据的教师向学生提供有用但带有噪声的信息，希望使学生模型能够在不访问敏感数据特定特征的情况下在给定任务上表现良好。 由于更强的隐私保证通常涉及教师进行更显著的扰动，因此部署现有框架本质上涉及学生性能和隐私保证之间的权衡。 先前工作中使用的重要技术之一是教师模型的集成，这些模型根据带有噪声的投票程序向学生提供信息。 在本工作中，我们提出了一种具有平滑敏感度的新投票机制，我们称之为不可变噪声ArgMax，该机制在某些条件下可以承受来自教师的非常大的随机噪声，而不会影响传递给学生的有用信息。 与之前的工作相比，我们的方法在所有指标上都优于最先进的方法，并且在更大任务上表现出更好的性能和更强的隐私（$\epsilon \approx 0$）。 这种新提出的框架可以与任何机器学习模型一起使用，并为需要在大量数据上进行训练的任务提供有吸引力的解决方案。 显示英文摘要

摘要： Ensuring the privacy of sensitive data used to train modern machine learning models is of paramount importance in many areas of practice. One approach to study these concerns is through the lens of differential privacy. In this framework, privacy guarantees are generally obtained by perturbing models in such a way that specifics of data used to train the model are made ambiguous. A particular instance of this approach is through a "teacher-student" framework, wherein the teacher, who owns the sensitive data, provides the student with useful, but noisy, information, hopefully allowing the student model to perform well on a given task without access to particular features of the sensitive data. Because stronger privacy guarantees generally involve more significant perturbation on the part of the teacher, deploying existing frameworks fundamentally involves a trade-off between student's performance and privacy guarantee. One of the most important techniques used in previous works involves an ensemble of teacher models, which return information to a student based on a noisy voting procedure. In this work, we propose a novel voting mechanism with smooth sensitivity, which we call Immutable Noisy ArgMax, that, under certain conditions, can bear very large random noising from the teacher without affecting the useful information transferred to the student. Compared with previous work, our approach improves over the state-of-the-art methods on all measures, and scale to larger tasks with both better performance and stronger privacy ($\epsilon \approx 0$). This new proposed framework can be applied with any machine learning models, and provides an appealing solution for tasks that requires training on a large amount of data.