标题： MINT：通过标记减少来减轻大型视觉-语言模型中的幻觉 显示英文标题

标题： MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction

摘要： 幻觉一直是大型视觉语言模型（LVLMs）在需要高可靠性的领域应用中长期存在且不可避免的问题。 各种方法依赖于数据标注或训练策略进行改进，但较少关注LLM本身的固有问题。 为了填补这一空白，我们深入研究了LVLM解码过程中的注意力机制。 有趣的是，我们的研究揭示了LVLM层次结构中存在的普遍注意力冗余，表现为深层中过度扩展的图像处理和大量非必要的图像标记。 基于这一观察，我们提出了MINT，一种新颖的无训练解码策略，通过令牌减少来减轻幻觉。 具体来说，我们通过屏蔽LVLM对无关图像标记的注意力，动态增强其局部感知能力。 此外，我们使用对比解码，促使模型更加关注那些关键的图像区域。 我们的完整方法旨在引导模型在生成过程中更加关注关键视觉元素。 在多个流行的公共基准上的广泛实验结果表明，与原始模型相比，我们的方法在减轻由分心感知引起的幻觉方面提高了4%。 同时，尽管我们减少了一套图像标记，我们的方法被证明使模型能够感知到5%更多的视觉点。 显示英文摘要

摘要： Hallucination has been a long-standing and inevitable problem that hinders the application of Large Vision-Language Models (LVLMs) in domains that require high reliability. Various methods focus on improvement depending on data annotations or training strategies, yet place less emphasis on LLM's inherent problems. To fill this gap, we delve into the attention mechanism of the decoding process in the LVLM. Intriguingly, our investigation uncovers the prevalent attention redundancy within the hierarchical architecture of the LVLM, manifesting as overextended image processing in deep layers and an overabundance of non-essential image tokens. Stemming from the observation, we thus propose MINT, a novel training-free decoding strategy, MItigating hallucinations via tokeN reducTion. Specifically, we dynamically intensify the LVLM's local perception capability by masking its attention to irrelevant image tokens. In addition, we use contrastive decoding that pushes the model to focus more on those key image regions. Our full method aims to guide the model in concentrating more on key visual elements during generation. Extensive experimental results on several popular public benchmarks show that our approach achieves a 4% improvement in mitigating hallucinations caused by distracted perception compared to original models. Meanwhile, our approach is demonstrated to make the model perceive 5% more visual points even though we reduce a suite of image tokens.