标题： ILID：印度语言的原生脚本语言识别 显示英文标题

标题： ILID: Native Script Language Identification for Indian Languages

摘要： 语言识别任务是自然语言处理中的关键基础步骤。 通常，它作为广泛使用的自然语言处理应用的预处理步骤，例如多语言机器翻译、信息检索、问答和文本摘要。 语言识别的核心挑战在于在嘈杂、简短和代码混合环境中区分语言。 对于具有词汇和语音相似性但存在显著差异的多样化印度语言来说，这变得更加困难。 许多印度语言使用相同的文字，使该任务更加具有挑战性。 考虑到所有这些挑战，我们开发并发布了一个包含250,000个句子的数据集，其中包括23种语言，包括英语和所有22种官方印度语言，并用其语言标识符进行标记，其中大多数语言的数据都是新创建的。 我们还开发并发布了使用机器学习中最先进的方法以及微调预训练转换器模型的基线模型。 我们的模型在语言识别任务中优于最先进的预训练转换器模型。 数据集和代码可在 https://yashingle-ai.github.io/ILID/ 和 Huggingface 开源库中获得。 显示英文摘要

摘要： The language identification task is a crucial fundamental step in NLP. Often it serves as a pre-processing step for widely used NLP applications such as multilingual machine translation, information retrieval, question and answering, and text summarization. The core challenge of language identification lies in distinguishing languages in noisy, short, and code-mixed environments. This becomes even harder in case of diverse Indian languages that exhibit lexical and phonetic similarities, but have distinct differences. Many Indian languages share the same script, making the task even more challenging. Taking all these challenges into account, we develop and release a dataset of 250K sentences consisting of 23 languages including English and all 22 official Indian languages labeled with their language identifiers, where data in most languages are newly created. We also develop and release baseline models using state-of-the-art approaches in machine learning and fine-tuning pre-trained transformer models. Our models outperforms the state-of-the-art pre-trained transformer models for the language identification task. The dataset and the codes are available at https://yashingle-ai.github.io/ILID/ and in Huggingface open source libraries.