标题： 谁在询问？ 基于角色的问答模拟用于对话式人工智能评估 显示英文标题

标题： Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluation

摘要： 语言模型用户经常在他们的提问中嵌入个人和社会背景。提问者的角色——隐含在问题的表述方式中——对适当的回应有特定的需求。然而，大多数评估虽然捕捉了模型的回答能力，但往往忽略了提问者是谁。这种差距在像阿片类药物使用障碍（OUD）这样的污名化领域尤其关键，因为在这些领域中，考虑用户背景对于提供易于获取、无污名化的回答至关重要。我们提出了CoRUS（COmmunity-driven Roles for User-centric Question Simulation），一个用于模拟基于角色问题的框架。结合角色理论和一个在线OUD康复社区（r/OpiatesRecovery）的帖子，我们首先构建了一个提问者角色的分类学——患者、护理人员、从业者。接下来，我们利用它来模拟15,321个嵌入每个角色目标、行为和经验的问题。我们的评估显示，这些问题既高度可信，又与现实世界的数据相当。当用于评估五个大型语言模型时，对于相同的问题但不同的角色，我们发现系统性差异：弱势角色，如患者和护理人员，在与从业者相比时，会引发更多支持性的回应（+17%）和减少的知识内容（-19%）。我们的工作展示了隐式地传达用户角色如何影响模型响应，并为对话式人工智能的角色导向评估提供了一种方法。 显示英文摘要

摘要： Language model users often embed personal and social context in their questions. The asker's role -- implicit in how the question is framed -- creates specific needs for an appropriate response. However, most evaluations, while capturing the model's capability to respond, often ignore who is asking. This gap is especially critical in stigmatized domains such as opioid use disorder (OUD), where accounting for users' contexts is essential to provide accessible, stigma-free responses. We propose CoRUS (COmmunity-driven Roles for User-centric Question Simulation), a framework for simulating role-based questions. Drawing on role theory and posts from an online OUD recovery community (r/OpiatesRecovery), we first build a taxonomy of asker roles -- patients, caregivers, practitioners. Next, we use it to simulate 15,321 questions that embed each role's goals, behaviors, and experiences. Our evaluations show that these questions are both highly believable and comparable to real-world data. When used to evaluate five LLMs, for the same question but differing roles, we find systematic differences: vulnerable roles, such as patients and caregivers, elicit more supportive responses (+17%) and reduced knowledge content (-19%) in comparison to practitioners. Our work demonstrates how implicitly signaling a user's role shapes model responses, and provides a methodology for role-informed evaluation of conversational AI.