标题： 评估人工智能生成的临床记录的质量：对大型语言模型记录员的验证评估 显示英文标题

标题： Assessing the Quality of AI-Generated Clinical Notes: A Validated Evaluation of a Large Language Model Scribe

摘要： 在美国内的医疗实践中，医生已经开始采用生成式人工智能（AI）工具来执行记录员的功能，以减轻记录临床会诊的负担。 尽管这些工具被广泛使用，但目前尚无既定的方法来评估AI记录员的质量。 为了解决这一问题，我们开展了一项盲法研究，根据录音的临床会诊，比较大型语言模型（LLM）生成的临床记录与领域专家的相对表现。 来自 Physician Documentation Quality Instrument（PDQI9）的定量指标提供了一个衡量记录质量的框架，我们对此进行了调整，以评估AI生成记录的相对表现。 涵盖5个医学专业领域的临床专家使用PDQI9工具对专家撰写的黄金记录和LLM撰写的Ambient记录进行评估。 每个专业领域的两名评估者对总共97次患者就诊的记录进行了评分。 我们发现，在普通医学、骨科和妇产科中，评估者之间的总体一致性（RWG大于0.7）很高，在儿科和心脏病学中则为中等（RWG 0.5至0.7）到高一致性。 我们发现整体记录质量存在适度但显著的差异，其中黄金记录得分为5分中的4.25分，Ambient记录得分为5分中的4.20分（p = 0.04）。 我们的研究结果支持将PDQI9仪器作为评估LLM撰写记录质量的实用方法，与人工撰写记录相比。 显示英文摘要

摘要： In medical practices across the United States, physicians have begun implementing generative artificial intelligence (AI) tools to perform the function of scribes in order to reduce the burden of documenting clinical encounters. Despite their widespread use, no established methods exist to gauge the quality of AI scribes. To address this gap, we developed a blinded study comparing the relative performance of large language model (LLM) generated clinical notes with those from field experts based on audio-recorded clinical encounters. Quantitative metrics from the Physician Documentation Quality Instrument (PDQI9) provided a framework to measure note quality, which we adapted to assess relative performance of AI generated notes. Clinical experts spanning 5 medical specialties used the PDQI9 tool to evaluate specialist-drafted Gold notes and LLM authored Ambient notes. Two evaluators from each specialty scored notes drafted from a total of 97 patient visits. We found uniformly high inter rater agreement (RWG greater than 0.7) between evaluators in general medicine, orthopedics, and obstetrics and gynecology, and moderate (RWG 0.5 to 0.7) to high inter rater agreement in pediatrics and cardiology. We found a modest yet significant difference in the overall note quality, wherein Gold notes achieved a score of 4.25 out of 5 and Ambient notes scored 4.20 out of 5 (p = 0.04). Our findings support the use of the PDQI9 instrument as a practical method to gauge the quality of LLM authored notes, as compared to human-authored notes.