标题： 评估使用大型语言模型在放射肿瘤学中自动化计算机断层扫描模拟订单摘要的性能 显示英文标题

标题： Evaluating The Performance of Using Large Language Models to Automate Summarization of CT Simulation Orders in Radiation Oncology

摘要： 目的：本研究旨在使用大型语言模型（LLM）自动化生成来自CT模拟订单的摘要，并评估其性能。 材料和方法：从我们机构的Aria数据库中收集了607份患者的CT模拟订单。 使用本地托管的Llama 3.1 405B模型，通过应用程序编程接口（API）服务访问，用于从CT模拟订单中提取关键词并生成摘要。 下载的CT模拟订单根据治疗方式和疾病部位分为七个组。 对于每组，与治疗师合作开发了定制的指令提示，以指导Llama 3.1 405B模型生成摘要。 相应摘要的基准真实数据是通过仔细审查每份CT模拟订单手动得出的，随后由治疗师进行验证。 治疗师使用经过验证的基准真实数据作为参考，评估LLM生成的摘要的准确性。 结果：大约98%的LLM生成的摘要在准确性方面与手动生成的基准真实数据一致。 我们的评估显示，与相应的治疗师生成的摘要相比，LLM生成的摘要在格式上更加一致，可读性更强。 这种自动化方法在所有组中均表现出一致的性能，无论治疗方式或疾病部位如何。 结论：本研究展示了Llama 3.1 405B模型在提取关键词和总结CT模拟订单方面的高精度和一致性，表明LLM在该任务中具有巨大潜力，可以减轻治疗师的工作负担并提高工作效率。 显示英文摘要

摘要： Purpose: This study aims to use a large language model (LLM) to automate the generation of summaries from the CT simulation orders and evaluate its performance. Materials and Methods: A total of 607 CT simulation orders for patients were collected from the Aria database at our institution. A locally hosted Llama 3.1 405B model, accessed via the Application Programming Interface (API) service, was used to extract keywords from the CT simulation orders and generate summaries. The downloaded CT simulation orders were categorized into seven groups based on treatment modalities and disease sites. For each group, a customized instruction prompt was developed collaboratively with therapists to guide the Llama 3.1 405B model in generating summaries. The ground truth for the corresponding summaries was manually derived by carefully reviewing each CT simulation order and subsequently verified by therapists. The accuracy of the LLM-generated summaries was evaluated by therapists using the verified ground truth as a reference. Results: About 98% of the LLM-generated summaries aligned with the manually generated ground truth in terms of accuracy. Our evaluations showed an improved consistency in format and enhanced readability of the LLM-generated summaries compared to the corresponding therapists-generated summaries. This automated approach demonstrated a consistent performance across all groups, regardless of modality or disease site. Conclusions: This study demonstrated the high precision and consistency of the Llama 3.1 405B model in extracting keywords and summarizing CT simulation orders, suggesting that LLMs have great potential to help with this task, reduce the workload of therapists and improve workflow efficiency.