标题： HSCR：用于对齐医学视觉语言模型的分层自对比奖励 显示英文标题

标题： HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models

摘要： 医学视觉语言模型（Med-VLMs）在各种任务中取得了成功，但大多数现有方法忽略了可能导致临床环境中不可信响应的模态错配问题。 本文提出了一种名为分层自对比奖励（HSCR）的新方法，解决了Med-VLM对齐中的两个关键挑战：1）以成本效益的方式生成高质量的偏好数据；2）捕获细微且上下文感知的偏好以提高对齐效果。 HSCR首先利用Med-VLMs内在的能力，以更高的采样概率生成不受欢迎的响应。 通过分析视觉标记丢弃后的输出对数偏移量，我们识别出导致错配的模态耦合标记，并推导出一个隐式的对齐奖励函数。 该函数在解码过程中引导幻觉标记的替换，生成高质量的不受欢迎的数据。 此外，HSCR引入了多级偏好优化策略，通过结合细微的隐式偏好和利用不受欢迎数据中的相对质量来捕捉微妙的对齐线索，从而超越传统的相邻级别优化策略，实现更精确且上下文感知的优化。 在多个医疗任务上的广泛实验，包括Med-VQA、医疗图像描述生成和指令遵循，表明HSCR不仅提升了零样本性能，而且仅用2,000个训练条目就显著提高了模态对齐和可信度。 显示英文摘要

摘要： Medical Vision-Language Models (Med-VLMs) have achieved success across various tasks, yet most existing methods overlook the modality misalignment issue that can lead to untrustworthy responses in clinical settings. In this paper, we propose Hierarchical Self-Contrastive Rewarding (HSCR), a novel approach that addresses two critical challenges in Med-VLM alignment: 1) Cost-effective generation of high-quality preference data; 2) Capturing nuanced and context-aware preferences for improved alignment. HSCR first leverages the inherent capability of Med-VLMs to generate dispreferred responses with higher sampling probability. By analyzing output logit shifts after visual token dropout, we identify modality-coupled tokens that induce misalignment and derive an implicit alignment reward function. This function guides token replacement with hallucinated ones during decoding, producing high-quality dispreferred data. Furthermore, HSCR introduces a multi-level preference optimization strategy, which extends beyond traditional adjacent-level optimization by incorporating nuanced implicit preferences, leveraging relative quality in dispreferred data to capture subtle alignment cues for more precise and context-aware optimization. Extensive experiments across multiple medical tasks, including Med-VQA, medical image captioning and instruction following, demonstrate that HSCR not only enhances zero-shot performance but also significantly improves modality alignment and trustworthiness with just 2,000 training entries.