标题： 大模型的可察觉差异 显示英文标题

标题： Just Noticeable Difference for Large Multimodal Models

摘要： 显著差异（JND），即人类视觉系统（HVS）可以感知的最小变化，已经研究了几十年。 尽管最近的研究将这一领域的研究扩展到了机器视觉，但缺乏系统地探索其在多种任务和刺激类型中的感知边界的研究，尤其是在大型多模态模型（LMMs）迅速发展的当前时代，研究模型的多方面能力已成为主流焦点。 此外，LMMs的感知缺陷尚未得到充分研究，导致潜在的安全问题和响应效率低下。 在本文中，我们进行了初步尝试，并证明当前LMMs中存在显著的视觉盲点。 为了系统地量化这一特性，我们提出一个新的概念，{\bf LMM-JND}，以及其确定流程。 针对揭示与HVS对齐的视觉感知任务中的行为共性，我们深入研究了几种LMM家族，并构建了一个大规模数据集，命名为VPA-JND，其中包含21.5k参考图像，跨越12种失真类型，超过489k个刺激，以促进LMM-JND的研究。 VPA-JND暴露了最先进的LMMs（包括GPT-4o和InternVL2.5系列）在基本比较查询中遇到困难，并且明显低于人类水平的视觉性能。 我们进一步探讨了视觉和语言主干的影响，并发现它们的设计理念之间存在显著相关性，这可能为未来LMMs的视觉敏锐度优化提供指导。 综上所述，我们的研究强调了LMM-JND作为研究LMMs的独特视角的重要性，可预测的LMM-JND对于安全问题至关重要。 这项工作将在https://github.com/zijianchen98/LMM-JND上发布。 显示英文摘要

摘要： Just noticeable difference (JND), the minimum change that the human visual system (HVS) can perceive, has been studied for decades. Although recent work has extended this line of research into machine vision, there has been a scarcity of studies systematically exploring its perceptual boundaries across multiple tasks and stimulus types, particularly in the current era of rapidly advancing large multimodal models (LMMs), where studying the multifaceted capabilities of models has become a mainstream focus. Moreover, the perceptual defects of LMMs are not investigated thoroughly, resulting in potential security issues and suboptimal response efficiency. In this paper, we take an initial attempt and demonstrate that there exist significant visual blind spots in current LMMs. To systemically quantify this characteristic, we propose a new concept, {\bf LMM-JND}, together with its determination pipeline. Targeting uncovering the behavior commonalities in HVS-aligned visual perception tasks, we delve into several LMM families and construct a large-scale dataset, named VPA-JND, which contains 21.5k reference images with over 489k stimuli across 12 distortion types, to facilitate LMM-JND studies. VPA-JND exposes areas where state-of-the-art LMMs, including GPT-4o and the InternVL2.5 series, struggle with basic comparison queries and fall significantly short of human-level visual performance. We further explore the effects of vision and language backbones and find a notable correlation between their design philosophy that may instruct the future refinement of LMMs for their visual acuity. Together, our research underscores the significance of LMM-JND as a unique perspective for studying LMMs, and predictable LMM-JND is crucial for security concerns. This work will be available at https://github.com/zijianchen98/LMM-JND.