标题： 针对对立论点的大型语言模型忠实完整性对齐 显示英文标题

标题： Aligning Large Language Models for Faithful Integrity Against Opposing Argument

摘要： 大型语言模型（LLMs）在复杂推理任务中展示了令人印象深刻的能力。 然而，在对话中，它们可能会被不忠实的论点误导，即使它们最初的陈述是正确的。 为此，我们研究了保持LLMs忠实完整性的问题。 这包括确保LLMs在面对相反论点时遵守其忠实的陈述，并且在提出忠实论点时能够纠正其错误的陈述。 在这项工作中，我们提出了一种新的框架，名为基于置信度估计的忠实完整性对齐（AFICE），旨在使LLM的响应与忠实完整性对齐。 具体来说，AFICE首先设计了一种双边置信度估计（BCE）方法，用于估计在特定上下文中LLM生成的每个响应的不确定性，该方法同时根据解码过程中的内部状态估计模型对问题的置信度以及根据累积概率比估计模型对答案的置信度。 通过BCE，我们构建了一个包含上下文、原始陈述和论点的对话偏好数据集，该数据集用于使用直接偏好优化（DPO）对LLM进行忠实完整性对齐。 在广泛基准上的大量实验结果表明，当遇到相反论点时，LLM保持忠实响应的能力有了显著提升，确保了LLM在复杂交互环境中的实际效用和可信度。 代码和数据将通过https://github.com/zhaoy777/AFICE.git发布 显示英文摘要

摘要： Large Language Models (LLMs) have demonstrated impressive capabilities in complex reasoning tasks. However, they can be easily misled by unfaithful arguments during conversations, even when their original statements are correct. To this end, we investigate the problem of maintaining faithful integrity in LLMs. This involves ensuring that LLMs adhere to their faithful statements in the face of opposing arguments and are able to correct their incorrect statements when presented with faithful arguments. In this work, we propose a novel framework, named Alignment for Faithful Integrity with Confidence Estimation (AFICE), which aims to align the LLM responses with faithful integrity. Specifically, AFICE first designs a Bilateral Confidence Estimation (BCE) approach for estimating the uncertainty of each response generated by the LLM given a specific context, which simultaneously estimate the model's confidence to the question based on the internal states during decoding as well as to the answer based on cumulative probability ratios. With the BCE, we construct a conversational preference dataset composed of context, original statement, and argument, which is adopted for aligning the LLM for faithful integrity using Direct Preference Optimization (DPO). Extensive experimental results on a wide range of benchmarks demonstrate significant improvements in the LLM's ability to maintain faithful responses when encountering opposing arguments, ensuring both the practical utility and trustworthiness of LLMs in complex interactive settings. Code and data will be released via https://github.com/zhaoy777/AFICE.git