标题： 基于关联疾病感知的视觉标记记忆激活用于基于大语言模型的X光报告生成 显示英文标题

标题： Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generation

摘要： 基于X射线图像的医学报告生成在大型语言模型的帮助下近年来取得了显著进展，然而，这些模型尚未充分挖掘视觉图像区域中的有效信息，导致生成的报告在语言上通顺但对关键疾病的描述不足。 在本文中，我们提出了一种新颖的关联记忆增强型X射线报告生成模型，该模型能够有效地模拟专业医生撰写医学报告的过程。 它同时考虑了全局和局部视觉信息的挖掘，并关联历史报告信息以更好地完成当前报告的撰写。 具体来说，给定一张X射线图像，我们首先利用分类模型及其激活图来完成与疾病高度相关的视觉区域的挖掘和疾病查询标记的学习。 然后，我们采用视觉霍普菲尔德网络来建立与疾病相关的标记的记忆关联，并采用报告霍普菲尔德网络来检索报告记忆信息。 这一过程促进了基于大型语言模型的高质量报告生成，并在多个基准数据集上实现了最先进的性能，包括IU X-ray、MIMIC-CXR和Chexpert Plus。 本工作的源代码发布在 \url{https://github.com/Event-AHU/Medical_Image_Analysis}. 显示英文摘要

摘要： X-ray image based medical report generation achieves significant progress in recent years with the help of the large language model, however, these models have not fully exploited the effective information in visual image regions, resulting in reports that are linguistically sound but insufficient in describing key diseases. In this paper, we propose a novel associative memory-enhanced X-ray report generation model that effectively mimics the process of professional doctors writing medical reports. It considers both the mining of global and local visual information and associates historical report information to better complete the writing of the current report. Specifically, given an X-ray image, we first utilize a classification model along with its activation maps to accomplish the mining of visual regions highly associated with diseases and the learning of disease query tokens. Then, we employ a visual Hopfield network to establish memory associations for disease-related tokens, and a report Hopfield network to retrieve report memory information. This process facilitates the generation of high-quality reports based on a large language model and achieves state-of-the-art performance on multiple benchmark datasets, including the IU X-ray, MIMIC-CXR, and Chexpert Plus. The source code of this work is released on \url{https://github.com/Event-AHU/Medical_Image_Analysis}.