标题： 视频大语言模型的强化学习调优：奖励设计与数据效率 显示英文标题

标题： Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency

摘要： 理解具有复杂语义和长时间依赖性的真实世界视频仍然是计算机视觉中的一个基本挑战。 多模态大型语言模型（MLLMs）的最新进展展示了其在视觉-语言任务中的强大能力，而强化学习微调（RLT）进一步提升了它们的推理能力。 在这项工作中，我们探索了RLT作为一种后训练策略，以增强MLLMs的视频特定推理能力。 基于Group Relative Policy Optimization (GRPO)框架，我们提出了一种双奖励公式，通过离散和连续奖励信号监督语义和时间推理。 为了促进有效的基于偏好的优化，我们引入了一种基于重复推理的方差感知数据选择策略，以识别提供信息丰富学习信号的样本。 我们在八个具有代表性的视频理解任务上评估了我们的方法，包括VideoQA、时间视频定位和Grounded VideoQA。 我们的方法始终优于监督微调和现有的RLT基线，在显著减少训练数据的情况下取得了优越的性能。 这些结果强调了奖励设计和数据选择在利用MLLMs推进以推理为中心的视频理解中的重要性。 值得注意的是，两个月前发布的初始代码现已更新，包括优化的奖励机制和额外的数据集。 最新版本可在https://github.com/appletea233/Temporal-R1获取。 显示英文摘要

摘要： Understanding real-world videos with complex semantics and long temporal dependencies remains a fundamental challenge in computer vision. Recent progress in multimodal large language models (MLLMs) has demonstrated strong capabilities in vision-language tasks, while reinforcement learning tuning (RLT) has further improved their reasoning abilities. In this work, we explore RLT as a post-training strategy to enhance the video-specific reasoning capabilities of MLLMs. Built upon the Group Relative Policy Optimization (GRPO) framework, we propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals. To facilitate effective preference-based optimization, we introduce a variance-aware data selection strategy based on repeated inference to identify samples that provide informative learning signals. We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA. Our method consistently outperforms supervised fine-tuning and existing RLT baselines, achieving superior performance with significantly less training data. These results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs. Notably, The initial code release (two months ago) has now been expanded with updates, including optimized reward mechanisms and additional datasets. The latest version is available at https://github.com/appletea233/Temporal-R1 .