标题： 多模态生成：在仿真中使用多模态生成来学习真实世界的多模态策略 显示英文标题

标题： MultiGen: Using Multimodal Generation in Simulation to Learn Multimodal Policies in Real

摘要： 机器人必须整合多种感觉模态，以便在现实世界中有效行动。 然而，在大规模上学习这种多模态策略仍然具有挑战性。 仿真提供了一个可行的解决方案，但尽管视觉受益于高保真模拟器，其他模态（例如声音）却难以模拟。 因此，仿生到现实的迁移主要在基于视觉的任务中取得成功，而多模态迁移仍 largely 未实现。 在这项工作中，我们通过引入 MultiGen 来解决这些挑战，这是一个将大规模生成模型集成到传统物理模拟器中的框架，实现了多感官仿真。 我们在动态任务机器人倒水上展示了我们的框架，该任务本质上依赖于多模态反馈。 通过根据模拟视频合成逼真的音频，我们的方法实现了在丰富视听轨迹上的训练——而无需任何真实机器人数据。 我们展示了在新型容器和液体上的有效零样本迁移，突显了生成建模在模拟难以建模的模态和缩小多模态仿生到现实差距方面的潜力。 显示英文摘要

摘要： Robots must integrate multiple sensory modalities to act effectively in the real world. Yet, learning such multimodal policies at scale remains challenging. Simulation offers a viable solution, but while vision has benefited from high-fidelity simulators, other modalities (e.g. sound) can be notoriously difficult to simulate. As a result, sim-to-real transfer has succeeded primarily in vision-based tasks, with multimodal transfer still largely unrealized. In this work, we tackle these challenges by introducing MultiGen, a framework that integrates large-scale generative models into traditional physics simulators, enabling multisensory simulation. We showcase our framework on the dynamic task of robot pouring, which inherently relies on multimodal feedback. By synthesizing realistic audio conditioned on simulation video, our method enables training on rich audiovisual trajectories -- without any real robot data. We demonstrate effective zero-shot transfer to real-world pouring with novel containers and liquids, highlighting the potential of generative modeling to both simulate hard-to-model modalities and close the multimodal sim-to-real gap.