标题： Zoom-Refine：通过局部缩放和自 refinement 提升高分辨率多模态理解 显示英文标题

标题： Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement

摘要： 多模态大型语言模型（MLLM）通常难以准确解释高分辨率图像，在这些图像中，细微的细节对于复杂的视觉理解至关重要。 我们引入了Zoom-Refine，这是一种新颖的无需训练的方法，旨在提升MLLM的能力以解决此问题。 Zoom-Refine通过\textit{局部缩放}和\textit{自我精炼}的协同过程运作。 在\textit{局部放大}步骤中，Zoom-Refine利用MLLM对输入查询提供初步响应，并通过预测边界框坐标来识别与任务最相关的图像区域。 在\textit{自精化}步骤中，Zoom-Refine将高分辨率裁剪部分（由\textit{局部缩放}确定）的细粒度细节与其初始推理相结合，重新评估并优化其初步响应。 我们的方法利用了MLLM在空间定位、上下文推理和比较分析方面的固有能力，而无需额外的训练或外部专家。 全面的实验表明，Zoom-Refine在两个具有挑战性的高分辨率多模态基准数据集上表现出色。 代码可在\href{https://github.com/xavier-yu114/Zoom-Refine}{\color{magenta}github.com/xavier-yu114/Zoom-Refine}获取。 显示英文摘要

摘要： Multimodal Large Language Models (MLLM) often struggle to interpret high-resolution images accurately, where fine-grained details are crucial for complex visual understanding. We introduce Zoom-Refine, a novel training-free method that enhances MLLM capabilities to address this issue. Zoom-Refine operates through a synergistic process of \textit{Localized Zoom} and \textit{Self-Refinement}. In the \textit{Localized Zoom} step, Zoom-Refine leverages the MLLM to provide a preliminary response to an input query and identifies the most task-relevant image region by predicting its bounding box coordinates. During the \textit{Self-Refinement} step, Zoom-Refine then integrates fine-grained details from the high-resolution crop (identified by \textit{Localized Zoom}) with its initial reasoning to re-evaluate and refine its preliminary response. Our method harnesses the MLLM's inherent capabilities for spatial localization, contextual reasoning and comparative analysis without requiring additional training or external experts. Comprehensive experiments demonstrate the efficacy of Zoom-Refine on two challenging high-resolution multimodal benchmarks. Code is available at \href{https://github.com/xavier-yu114/Zoom-Refine}{\color{magenta}github.com/xavier-yu114/Zoom-Refine}