Title: RooseBERT: A New Deal For Political Language Modelling Show Chinese title

Title: RooseBERT：政治语言建模的新方案

Abstract: The increasing amount of political debates and politics-related discussions calls for the definition of novel computational methods to automatically analyse such content with the final goal of lightening up political deliberation to citizens. However, the specificity of the political language and the argumentative form of these debates (employing hidden communication strategies and leveraging implicit arguments) make this task very challenging, even for current general-purpose pre-trained Language Models. To address this issue, we introduce a novel pre-trained Language Model for political discourse language called RooseBERT. Pre-training a language model on a specialised domain presents different technical and linguistic challenges, requiring extensive computational resources and large-scale data. RooseBERT has been trained on large political debate and speech corpora (8K debates, each composed of several sub-debates on different topics) in English. To evaluate its performances, we fine-tuned it on four downstream tasks related to political debate analysis, i.e., named entity recognition, sentiment analysis, argument component detection and classification, and argument relation prediction and classification. Our results demonstrate significant improvements over general-purpose Language Models on these four tasks, highlighting how domain-specific pre-training enhances performance in political debate analysis. We release the RooseBERT language model for the research community. Show Chinese abstract

Abstract: 随着越来越多的政治辩论和与政治相关的讨论，需要定义新的计算方法来自动分析此类内容，最终目标是为公民提供更清晰的政治讨论。 然而，政治语言的特殊性和这些辩论的论证形式（采用隐藏的沟通策略并利用隐含论点）使得这项任务非常具有挑战性，即使对于当前的通用预训练语言模型也是如此。 为了解决这个问题，我们引入了一个名为RooseBERT的新预训练语言模型，专门用于政治话语语言。 在专业领域上预训练语言模型会带来不同的技术和语言挑战，需要大量的计算资源和大规模数据。 RooseBERT已经在大型政治辩论和演讲语料库（8K场辩论，每场由多个不同主题的子辩论组成）中进行了训练。 为了评估其性能，我们在四个与政治辩论分析相关的下游任务上对其进行了微调，即命名实体识别、情感分析、论点组件检测和分类，以及论点关系预测和分类。 我们的结果表明，在这四个任务上，RooseBERT相比通用语言模型有显著提升，突显了领域特定预训练在政治辩论分析中的性能增强作用。 我们将RooseBERT语言模型发布给研究社区。