标题： 用于高效临床信息提取的大语言模型蒸馏 显示英文标题

标题： Distilling Large Language Models for Efficient Clinical Information Extraction

摘要： 大型语言模型（LLMs）在临床信息提取方面表现出色，但其计算需求限制了实际部署。知识蒸馏——从大模型到小模型的知识转移过程——提供了一种潜在的解决方案。我们评估了蒸馏后的 BERT 模型在临床命名实体识别（NER）任务中的表现，这些模型大约比现代 LLM 小 1000 倍。我们利用最先进的 LLM（Gemini 和 OpenAI 模型）以及医学本体（RxNorm 和 SNOMED）作为药物、疾病和症状提取的教师标签器。我们将方法应用于超过 3300 份涵盖五个公开数据集的临床记录，并将蒸馏后的 BERT 模型与它们的教师标签器以及在人工标签上微调的 BERT 模型进行了比较。外部验证使用了 MedAlign 数据集中的临床记录。对于疾病提取，F1 得分分别为 0.82（教师模型）、0.89（在人工标签上训练的 BioBERT）和 0.84（蒸馏的 BioBERT）。对于药物，F1 得分分别为 0.84（教师模型）、0.91（在人工标签上训练的 BioBERT）和 0.87（蒸馏的 BioBERT）。对于症状：教师模型的 F1 得分为 0.73，蒸馏的 BioBERT 为 0.68。蒸馏后的 BERT 模型具有更快的推理速度（分别是 GPT-4o、o1-mini 和 Gemini Flash 的 12 倍、4 倍和 8 倍），并且成本更低（分别是 GPT-4o、o1-mini 和 Gemini Flash 的 85 倍、101 倍和 2 倍）。在外部队列验证数据集中，蒸馏的 BERT 模型实现了 F1 得分 0.883（药物）、0.726（疾病）和 0.699（症状）。蒸馏后的 BERT 模型在 NER 任务上的表现与最先进的 LLM 相当，但成本降低了高达 101 倍，速度提高了 12 倍。蒸馏为临床信息提取提供了计算高效且可扩展的大型 LLM 替代方案。 显示英文摘要

摘要： Large language models (LLMs) excel at clinical information extraction but their computational demands limit practical deployment. Knowledge distillation--the process of transferring knowledge from larger to smaller models--offers a potential solution. We evaluate the performance of distilled BERT models, which are approximately 1,000 times smaller than modern LLMs, for clinical named entity recognition (NER) tasks. We leveraged state-of-the-art LLMs (Gemini and OpenAI models) and medical ontologies (RxNorm and SNOMED) as teacher labelers for medication, disease, and symptom extraction. We applied our approach to over 3,300 clinical notes spanning five publicly available datasets, comparing distilled BERT models against both their teacher labelers and BERT models fine-tuned on human labels. External validation was conducted using clinical notes from the MedAlign dataset. For disease extraction, F1 scores were 0.82 (teacher model), 0.89 (BioBERT trained on human labels), and 0.84 (BioBERT-distilled). For medication, F1 scores were 0.84 (teacher model), 0.91 (BioBERT-human), and 0.87 (BioBERT-distilled). For symptoms: F1 score of 0.73 (teacher model) and 0.68 (BioBERT-distilled). Distilled BERT models had faster inference (12x, 4x, 8x faster than GPT-4o, o1-mini, and Gemini Flash respectively) and lower costs (85x, 101x, 2x cheaper than GPT-4o, o1-mini, and Gemini Flash respectively). On the external validation dataset, the distilled BERT model achieved F1 scores of 0.883 (medication), 0.726 (disease), and 0.699 (symptom). Distilled BERT models were up to 101x cheaper and 12x faster than state-of-the-art LLMs while achieving similar performance on NER tasks. Distillation offers a computationally efficient and scalable alternative to large LLMs for clinical information extraction.