Title: CBIL: Collective Behavior Imitation Learning for Fish from Real Videos Show Chinese title

Title: 集体行为模仿学习（CBIL）：来自真实视频的鱼类研究

Abstract: Reproducing realistic collective behaviors presents a captivating yet formidable challenge. Traditional rule-based methods rely on hand-crafted principles, limiting motion diversity and realism in generated collective behaviors. Recent imitation learning methods learn from data but often require ground truth motion trajectories and struggle with authenticity, especially in high-density groups with erratic movements. In this paper, we present a scalable approach, Collective Behavior Imitation Learning (CBIL), for learning fish schooling behavior directly from videos, without relying on captured motion trajectories. Our method first leverages Video Representation Learning, where a Masked Video AutoEncoder (MVAE) extracts implicit states from video inputs in a self-supervised manner. The MVAE effectively maps 2D observations to implicit states that are compact and expressive for following the imitation learning stage. Then, we propose a novel adversarial imitation learning method to effectively capture complex movements of the schools of fish, allowing for efficient imitation of the distribution for motion patterns measured in the latent space. It also incorporates bio-inspired rewards alongside priors to regularize and stabilize training. Once trained, CBIL can be used for various animation tasks with the learned collective motion priors. We further show its effectiveness across different species. Finally, we demonstrate the application of our system in detecting abnormal fish behavior from in-the-wild videos. Show Chinese abstract

Abstract: 再现真实的群体行为既引人入胜又极具挑战性。传统的基于规则的方法依赖手工设计的原则，限制了生成的群体行为的运动多样性和真实性。近期的模仿学习方法虽然从数据中学习，但通常需要真实的运动轨迹，并且在高密度且运动无序的群体中往往难以保证真实性。本文提出了一种可扩展的方法——集体行为模仿学习（CBIL），用于直接从视频中学习鱼群行为，而无需依赖捕获的运动轨迹。我们的方法首先利用视频表示学习，其中掩码视频自动编码器（MVAE）以自监督的方式从视频输入中提取隐状态。MVAE有效地将二维观察映射到紧凑且具有表现力的隐状态，以便进入模仿学习阶段。然后，我们提出了一种新的对抗模仿学习方法，以有效捕捉鱼群复杂运动，允许在潜在空间中高效地模仿运动模式分布。该方法还结合了生物启发奖励和先验知识来正则化和稳定训练过程。训练完成后，CBIL可以使用学到的群体运动先验来进行各种动画任务。我们进一步展示了它在不同物种中的有效性。最后，我们演示了该系统在从野外视频中检测异常鱼行为的应用。