Title: How Important is Domain Specificity in Language Models and Instruction Finetuning for Biomedical Relation Extraction? Show Chinese title

Title: 语言模型和生物医学关系抽取的指令微调中领域特定性有多重要？

Abstract: Cutting edge techniques developed in the general NLP domain are often subsequently applied to the high-value, data-rich biomedical domain. The past few years have seen generative language models (LMs), instruction finetuning, and few-shot learning become foci of NLP research. As such, generative LMs pretrained on biomedical corpora have proliferated and biomedical instruction finetuning has been attempted as well, all with the hope that domain specificity improves performance on downstream tasks. Given the nontrivial effort in training such models, we investigate what, if any, benefits they have in the key biomedical NLP task of relation extraction. Specifically, we address two questions: (1) Do LMs trained on biomedical corpora outperform those trained on general domain corpora? (2) Do models instruction finetuned on biomedical datasets outperform those finetuned on assorted datasets or those simply pretrained? We tackle these questions using existing LMs, testing across four datasets. In a surprising result, general-domain models typically outperformed biomedical-domain models. However, biomedical instruction finetuning improved performance to a similar degree as general instruction finetuning, despite having orders of magnitude fewer instructions. Our findings suggest it may be more fruitful to focus research effort on larger-scale biomedical instruction finetuning of general LMs over building domain-specific biomedical LMs Show Chinese abstract

Abstract: 在一般自然语言处理（NLP）领域开发的前沿技术通常随后被应用于高价值、数据丰富的生物医学领域。 过去几年中，生成语言模型（LMs）、指令微调和少量样本学习已成为NLP研究的焦点。 因此，基于生物医学语料库预训练的生成LMs已经大量出现，也尝试了生物医学指令微调，所有这些都希望领域特定性能提高下游任务的性能。 鉴于训练此类模型所需的非微不足道的努力，我们研究了它们在关键生物医学NLP任务关系抽取中的优势，如果有的话。 具体来说，我们解决了两个问题：（1）在生物医学语料库上训练的LMs是否优于在通用领域语料库上训练的LMs？（2）在生物医学数据集上指令微调的模型是否优于在各种数据集上微调的模型或仅预训练的模型？ 我们使用现有的LMs，在四个数据集上进行测试。 一个令人惊讶的结果是，通用领域模型通常优于生物医学领域模型。 然而，尽管指令数量少几个数量级，生物医学指令微调仍能将性能提升到与通用指令微调相似的程度。 我们的研究结果表明，将研究重点放在通用LMs的大规模生物医学指令微调上，而不是构建领域特定的生物医学LMs，可能更为有益。