标题： 神经掩码生成器：学习为语言模型适应生成自适应词掩码 显示英文标题

标题： Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation

摘要： 我们提出一种方法，以自监督预训练的方式自动生成给定文本的领域和任务自适应掩码，从而使语言模型能够有效适应特定目标任务（例如问答）。 具体而言，我们提出一种基于强化学习的新框架，该框架学习掩码策略，使得使用生成的掩码进行目标语言模型的进一步预训练有助于提高在未见过文本上的任务性能。 我们使用具有熵正则化和经验回放的非策略演员-评论家进行强化学习，并提出一种基于Transformer的策略网络，该网络可以考虑给定文本中单词的相对重要性。 我们在几个问答和文本分类数据集上验证了我们的神经掩码生成器（NMG），使用BERT和DistilBERT作为语言模型，结果表明它优于基于规则的掩码策略，因为它可以自动学习最优的自适应掩码。 显示英文摘要

摘要： We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings.