标题： DuetGen：通过分层掩码建模的音乐驱动双人舞蹈生成 显示英文标题

标题： DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modeling

摘要： 我们提出DuetGen，一种从音乐生成互动双人舞蹈的新框架。 这项任务的关键挑战在于双人舞蹈互动的固有复杂性，其中舞伴需要彼此之间以及与音乐同步。 受运动合成最新进展的启发，我们提出了一种两阶段解决方案：将双人动作编码为离散标记，然后从音乐生成这些标记。 为了有效捕捉复杂的互动，我们将两名舞者的动作表示为一个统一的整体，以学习必要的运动标记，并在两个阶段中采用从粗到细的学习策略。 我们的第一阶段使用一个VQ-VAE，它在较粗的时间分辨率上分离高层次语义特征，在更细的时间分辨率上分离低层次细节，从而在不同抽象层次上生成两个离散标记序列。 随后，在第二阶段，两个生成掩码变压器学习将音乐信号映射到这些舞蹈标记：第一个生成高层次语义标记，第二个则在音乐和这些语义标记的条件下生成低层次标记。 我们训练这两个变压器以学习预测序列中的随机掩码标记，使它们能够在推理过程中通过填充一个空标记序列来迭代生成运动标记。 通过分层掩码建模和专门的互动表示，DuetGen实现了在各种流派中同步且互动的双人舞蹈生成。 在基准双人舞蹈数据集上的广泛实验和用户研究证明了DuetGen在运动真实感、音乐-舞蹈对齐和搭档协调方面的最先进性能。 显示英文摘要

摘要： We present DuetGen, a novel framework for generating interactive two-person dances from music. The key challenge of this task lies in the inherent complexities of two-person dance interactions, where the partners need to synchronize both with each other and with the music. Inspired by the recent advances in motion synthesis, we propose a two-stage solution: encoding two-person motions into discrete tokens and then generating these tokens from music. To effectively capture intricate interactions, we represent both dancers' motions as a unified whole to learn the necessary motion tokens, and adopt a coarse-to-fine learning strategy in both the stages. Our first stage utilizes a VQ-VAE that hierarchically separates high-level semantic features at a coarse temporal resolution from low-level details at a finer resolution, producing two discrete token sequences at different abstraction levels. Subsequently, in the second stage, two generative masked transformers learn to map music signals to these dance tokens: the first producing high-level semantic tokens, and the second, conditioned on music and these semantic tokens, producing the low-level tokens. We train both transformers to learn to predict randomly masked tokens within the sequence, enabling them to iteratively generate motion tokens by filling an empty token sequence during inference. Through the hierarchical masked modeling and dedicated interaction representation, DuetGen achieves the generation of synchronized and interactive two-person dances across various genres. Extensive experiments and user studies on a benchmark duet dance dataset demonstrate state-of-the-art performance of DuetGen in motion realism, music-dance alignment, and partner coordination.