Title: Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder Show Chinese title

Title: 探索使用富有表现力的音乐变分自编码器进行古典钢琴演奏生成

Abstract: The creativity of classical music arises not only from composers who craft the musical sheets but also from performers who interpret the static notations with expressive nuances. This paper addresses the challenge of generating classical piano performances from scratch, aiming to emulate the dual roles of composer and pianist in the creative process. We introduce the Expressive Compound Word (ECP) representation, which effectively captures both the metrical structure and expressive nuances of classical performances. Building on this, we propose the Expressive Music Variational AutoEncoder (XMVAE), a model featuring two branches: a Vector Quantized Variational AutoEncoder (VQ-VAE) branch that generates score-related content, representing the Composer, and a vanilla VAE branch that produces expressive details, fulfilling the role of Pianist. These branches are jointly trained with similar Seq2Seq architectures, leveraging a multiscale encoder to capture beat-level contextual information and an orthogonal Transformer decoder for efficient compound tokens decoding. Both objective and subjective evaluations demonstrate that XMVAE generates classical performances with superior musical quality compared to state-of-the-art models. Furthermore, pretraining the Composer branch on extra musical score datasets contribute to a significant performance gain. Show Chinese abstract

Abstract: 古典音乐的创造力不仅来自于创作乐谱的作曲家，也来自于以富有表现力的细微差别诠释静态乐谱的演奏者。 本文解决了从零开始生成古典钢琴演奏的挑战，旨在模仿创作过程中的作曲家和钢琴家的双重角色。 我们引入了表达性复合词（ECP）表示法，该方法有效地捕捉了古典演奏的节奏结构和表达细节。 在此基础上，我们提出了表达性音乐变分自编码器（XMVAE），该模型包含两个分支：一个向量量化变分自编码器（VQ-VAE）分支，用于生成与乐谱相关的内容，代表作曲家；一个普通VAE分支，用于生成表达性细节，履行钢琴家的角色。 这些分支通过类似的Seq2Seq架构进行联合训练，利用多尺度编码器捕捉节拍级别的上下文信息，并使用正交Transformer解码器进行高效的复合标记解码。 客观和主观评估都表明，与最先进的模型相比，XMVAE生成的古典演奏具有更优的音乐质量。 此外，在额外的音乐乐谱数据集上预训练作曲家分支有助于显著提升性能。