标题： 时空大语言模型：关于环境和行动的推理 显示英文标题

标题： Spatio-Temporal LLM: Reasoning about Environments and Actions

摘要： 尽管多模态大语言模型（MLLMs）最近取得了显著进展，但MLLMs在正确回答需要整体时空理解的提示时仍然存在困难。具体来说，解决同时涉及1）一个代理可以操作的环境整体，以及2）最近发生的、编码在视频片段中的动作的提示是具有挑战性的。然而，这种整体的时空理解对于在现实世界中运行的代理非常重要。为了解决这个问题，我们首先开发了一个框架来收集大规模数据集。使用收集到的“关于环境和动作的推理”（REA）数据集，我们表明近期的方法确实难以正确回答这些提示。为了改进，我们开发了一个“时空大语言模型”（ST-LLM），该模型配备了投影器，以提高对环境的空间理解和对最近观察结果的时间理解。在收集的REA数据上，我们展示了所提出的方法相比之前的工作显著提升了结果。代码和数据可在 https://zoezheng126.github.io/STLLM-website/ 获取。 显示英文摘要

摘要： Despite the significant recent progress of Multimodal Large Language Models (MLLMs), MLLMs still struggle to correctly answer prompts that require a holistic spatio-temporal understanding. Specifically, it is challenging to address prompts that refer to 1) the entirety of an environment that an agent equipped with an MLLM can operate in; and simultaneously also refer to 2) recent actions that just happened and are encoded in a video clip. However, such a holistic spatio-temporal understanding is important for agents operating in the real world. To address this issue, we first develop a framework to collect a large-scale dataset. Using the collected "Reasoning about Environments and Actions" (REA) dataset, we show that recent methods indeed struggle to correctly answer the prompts. To improve, we develop a "spatio-temporal LLM" (ST-LLM), a model equipped with projectors to improve both spatial understanding of an environment and temporal understanding of recent observations. On the collected REA data, we show that the proposed method significantly improves results compared to prior work. Code and data are available at https://zoezheng126.github.io/STLLM-website/.