Title: Multimodal Large Models Are Effective Action Anticipators Show Chinese title

Title: 多模态大模型是有效的动作预测者

Abstract: The task of long-term action anticipation demands solutions that can effectively model temporal dynamics over extended periods while deeply understanding the inherent semantics of actions. Traditional approaches, which primarily rely on recurrent units or Transformer layers to capture long-term dependencies, often fall short in addressing these challenges. Large Language Models (LLMs), with their robust sequential modeling capabilities and extensive commonsense knowledge, present new opportunities for long-term action anticipation. In this work, we introduce the ActionLLM framework, a novel approach that treats video sequences as successive tokens, leveraging LLMs to anticipate future actions. Our baseline model simplifies the LLM architecture by setting future tokens, incorporating an action tuning module, and reducing the textual decoder layer to a linear layer, enabling straightforward action prediction without the need for complex instructions or redundant descriptions. To further harness the commonsense reasoning of LLMs, we predict action categories for observed frames and use sequential textual clues to guide semantic understanding. In addition, we introduce a Cross-Modality Interaction Block, designed to explore the specificity within each modality and capture interactions between vision and textual modalities, thereby enhancing multimodal tuning. Extensive experiments on benchmark datasets demonstrate the superiority of the proposed ActionLLM framework, encouraging a promising direction to explore LLMs in the context of action anticipation. Code is available at https://github.com/2tianyao1/ActionLLM.git. Show Chinese abstract

Abstract: 长期动作预期的任务需要能够有效建模长时间的时序动态同时深入理解动作的内在语义的解决方案。传统方法主要依赖循环单元或Transformer层来捕捉长期依赖关系，但在应对这些挑战时往往不足。大型语言模型（LLMs）凭借其强大的顺序建模能力和丰富的常识知识，为长期动作预期带来了新的机遇。在本工作中，我们引入了ActionLLM框架，这是一种新方法，将视频序列视为连续的标记，利用LLMs来预测未来动作。我们的基线模型通过设置未来标记简化了LLM架构，结合了一个动作调优模块，并将文本解码器层减少为线性层，从而无需复杂指令或冗余描述即可进行简单的动作预测。为了进一步利用LLMs的常识推理能力，我们预测观察到帧的动作类别，并使用连续的文本线索来引导语义理解。此外，我们引入了一个跨模态交互块，旨在探索每种模态内的特异性并捕捉视觉和文本模态之间的交互，从而增强多模态调优。在基准数据集上的大量实验表明了所提出的ActionLLM框架的优势，鼓励探索LLMs在动作预期背景下的有前途的方向。代码可在https://github.com/2tianyao1/ActionLLM.git获取。