标题： 大型语言模型在消费者健康问题上的实证评估 显示英文标题

标题： An Empirical Evaluation of Large Language Models on Consumer Health Questions

摘要： 本研究评估了多种大型语言模型（LLMs）在MedRedQA上的性能，MedRedQA是一个从AskDocs子版块中提取的基于消费者的医学问题和答案数据集，由经验证的专家提供。 尽管大型语言模型在临床问答（QA）基准测试中表现出色，但它们在现实世界中基于消费者的医学问题上的有效性仍较少被了解。 MedRedQA具有独特的挑战，例如非正式的语言和需要针对非专业查询的精确回答。 为了评估模型性能，使用五种LLMs生成了响应：GPT-4o mini，Llama 3.1:70B，Mistral-123B，Mistral-7B和Gemini-Flash。 采用交叉评估方法，每个模型不仅评估自己的响应，还评估其他模型的响应以减少偏差。 结果表明，根据五位模型评审中的四位，GPT-4o mini与专家响应的对齐度最高，而根据五位模型评审中的三位，Mistral-7B得分最低。 本研究突显了当前LLMs在消费者健康医学问答中的潜力和局限性，指出了进一步发展的方向。 显示英文摘要

摘要： This study evaluates the performance of several Large Language Models (LLMs) on MedRedQA, a dataset of consumer-based medical questions and answers by verified experts extracted from the AskDocs subreddit. While LLMs have shown proficiency in clinical question answering (QA) benchmarks, their effectiveness on real-world, consumer-based, medical questions remains less understood. MedRedQA presents unique challenges, such as informal language and the need for precise responses suited to non-specialist queries. To assess model performance, responses were generated using five LLMs: GPT-4o mini, Llama 3.1: 70B, Mistral-123B, Mistral-7B, and Gemini-Flash. A cross-evaluation method was used, where each model evaluated its responses as well as those of others to minimize bias. The results indicated that GPT-4o mini achieved the highest alignment with expert responses according to four out of the five models' judges, while Mistral-7B scored lowest according to three out of five models' judges. This study highlights the potential and limitations of current LLMs for consumer health medical question answering, indicating avenues for further development.