标题： 基于约束学习的扩散模型的组合与对齐 显示英文标题

标题： Composition and Alignment of Diffusion Models using Constrained Learning

摘要： 扩散模型由于能够从复杂分布中采样，已在生成建模中变得普遍。为了提高生成样本的质量及其与用户需求的符合程度，两种常用的方法是：(i) 对齐，即微调扩散模型以使其与奖励对齐；和 (ii) 组合，即将几个预训练的扩散模型组合在一起，每个模型强调生成输出中的一个期望属性。然而，当优化多个奖励或组合多个模型时，常常会出现权衡，因为它们可能代表竞争的属性。现有方法无法保证生成的模型能忠实生成所有期望属性的样本。为解决这一差距，我们提出了一种约束优化框架，通过强制对齐模型满足奖励约束和/或保持与（可能多个）预训练模型的接近性，从而统一扩散模型的对齐和组合。我们提供了对约束对齐和组合问题解的理论表征，并开发了一种基于拉格朗日的原始-对偶训练算法来近似这些解。在实验中，我们在图像生成中展示了所提方法的有效性和优势，将其应用于对齐和组合，并表明我们的对齐或组合模型能有效满足约束，并优于等权重方法。我们的实现可在 https://github.com/shervinkhalafi/constrained_comp_align 找到。 显示英文摘要

摘要： Diffusion models have become prevalent in generative modeling due to their ability to sample from complex distributions. To improve the quality of generated samples and their compliance with user requirements, two commonly used methods are: (i) Alignment, which involves fine-tuning a diffusion model to align it with a reward; and (ii) Composition, which combines several pre-trained diffusion models, each emphasizing a desirable attribute in the generated outputs. However, trade-offs often arise when optimizing for multiple rewards or combining multiple models, as they can often represent competing properties. Existing methods cannot guarantee that the resulting model faithfully generates samples with all the desired properties. To address this gap, we propose a constrained optimization framework that unifies alignment and composition of diffusion models by enforcing that the aligned model satisfies reward constraints and/or remains close to (potentially multiple) pre-trained models. We provide a theoretical characterization of the solutions to the constrained alignment and composition problems and develop a Lagrangian-based primal-dual training algorithm to approximate these solutions. Empirically, we demonstrate the effectiveness and merits of our proposed approach in image generation, applying it to alignment and composition, and show that our aligned or composed model satisfies constraints effectively, and improves on the equally-weighted approach. Our implementation can be found at https://github.com/shervinkhalafi/constrained_comp_align.