标题： 线性扩散模型的推广动态 显示英文标题

标题： Generalization Dynamics of Linear Diffusion Models

摘要： 在有限数据集上训练的扩散模型，该数据集包含来自目标分布的 $N$ 样本，表现出从记忆到泛化的转变，在这一过程中，模型从再现训练样本转变为生成反映底层数据分布的新样本。 理解这一转变对于表征生成模型的样本效率和可靠性至关重要，但目前我们对该转变的理论理解尚不完整。 在这里，我们使用线性去噪器研究了一个简单模型中的记忆到泛化转变，这使得可以明确计算测试误差、采样分布以及样本与目标分布之间的 Kullback-Leibler 散度。 利用这些指标，我们预测这一转变大约发生在 $N \asymp d$ （输入维度）时。 当 $N$ 小于输入维度 $d$ 时，即只有部分相关变化方向存在于训练数据中，我们展示了正则化和提前停止如何帮助防止过拟合。 对于 $N > d$，我们发现线性扩散模型的采样分布以 $d/N$ 的线性方式接近其最优值（由 Kullback-Leibler 散度衡量），且与数据分布的具体情况无关。 我们的工作阐明了样本复杂度如何控制基于扩散生成模型的简单模型中的泛化，并提供了对线性去噪器训练动态的见解。 显示英文摘要

摘要： Diffusion models trained on finite datasets with $N$ samples from a target distribution exhibit a transition from memorisation, where the model reproduces training examples, to generalisation, where it produces novel samples that reflect the underlying data distribution. Understanding this transition is key to characterising the sample efficiency and reliability of generative models, but our theoretical understanding of this transition is incomplete. Here, we analytically study the memorisation-to-generalisation transition in a simple model using linear denoisers, which allow explicit computation of test errors, sampling distributions, and Kullback-Leibler divergences between samples and target distribution. Using these measures, we predict that this transition occurs roughly when $N \asymp d$, the dimension of the inputs. When $N$ is smaller than the dimension of the inputs $d$, so that only a fraction of relevant directions of variation are present in the training data, we demonstrate how both regularization and early stopping help to prevent overfitting. For $N > d$, we find that the sampling distributions of linear diffusion models approach their optimum (measured by the Kullback-Leibler divergence) linearly with $d/N$, independent of the specifics of the data distribution. Our work clarifies how sample complexity governs generalisation in a simple model of diffusion-based generative models and provides insight into the training dynamics of linear denoisers.