标题： 通过置信度引导的数据增强改善未知协变量偏移下的知识蒸馏 显示英文标题

标题： Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentation

摘要： 在大规模数据集上训练的大规模基础模型在各个领域都表现出强大的零样本能力。为了在数据和模型规模受限时复制它们的成功，知识蒸馏已成为一种将知识从基础模型转移到小型学生网络的有效工具。然而，蒸馏的效果在很大程度上受到可用训练数据的限制。 本文解决了知识蒸馏中常见的协变量偏移问题，在这种情况下，伪特征会在训练期间出现但在测试时不会。我们提出了一个问题：当这些伪特征未知，但有一个鲁棒的教师可用时，学生是否也能对它们变得鲁棒？我们通过引入一种基于扩散的数据增强策略来解决这个问题，该策略通过最大化教师和学生之间的分歧来生成图像，有效地创建了学生难以处理的具有挑战性的样本。 实验表明，我们的方法显著提高了CelebA和SpuCo Birds上的最差组和平均组准确率，以及在协变量偏移下的伪ImageNet上的伪mAUC，在这些方面优于最先进的基于扩散的数据增强基线。 显示英文摘要

摘要： Large foundation models trained on extensive datasets demonstrate strong zero-shot capabilities in various domains. To replicate their success when data and model size are constrained, knowledge distillation has become an established tool for transferring knowledge from foundation models to small student networks. However, the effectiveness of distillation is critically limited by the available training data. This work addresses the common practical issue of covariate shift in knowledge distillation, where spurious features appear during training but not at test time. We ask the question: when these spurious features are unknown, yet a robust teacher is available, is it possible for a student to also become robust to them? We address this problem by introducing a novel diffusion-based data augmentation strategy that generates images by maximizing the disagreement between the teacher and the student, effectively creating challenging samples that the student struggles with. Experiments demonstrate that our approach significantly improves worst group and mean group accuracy on CelebA and SpuCo Birds as well as the spurious mAUC on spurious ImageNet under covariate shift, outperforming state-of-the-art diffusion-based data augmentation baselines