标题： ILID：印度语言的原生脚本语言识别 显示英文标题

标题： ILID: Native Script Language Identification for Indian Languages

摘要： 语言识别任务是自然语言处理中的一个关键基础步骤。 通常它作为许多广泛使用的自然语言处理应用的预处理步骤，例如多语言机器翻译、信息检索、问答和文本摘要。 语言识别的核心挑战在于在嘈杂、简短和混合代码的环境中区分语言。 对于表现出词汇和语音相似性但又有明显差异的多种印度语言来说，这变得更加困难。 许多印度语言使用相同的文字系统，使得该任务更加具有挑战性。 在本文中，我们发布了一个包含23万条句子的数据集，其中包括英语和所有22种官方印度语言，并用语言标识符进行了标记，其中大多数语言的数据是新创建的。 我们还开发并发布了使用机器学习和深度学习最新方法的鲁棒基线模型，这些模型可以促进该领域研究。 我们的基线模型在语言识别任务上与最先进的模型相当。 显示英文摘要

摘要： The language identification task is a crucial fundamental step in NLP. Often it serves as a pre-processing step for widely used NLP applications such as multilingual machine translation, information retrieval, question and answering, and text summarization. The core challenge of language identification lies in distinguishing languages in noisy, short, and code-mixed environments. This becomes even harder in case of diverse Indian languages that exhibit lexical and phonetic similarities, but have distinct differences. Many Indian languages share the same script making the task even more challenging. In this paper, we release a dataset of 230K sentences consisting of English and all 22 official Indian languages labeled with their language identifiers where data in most languages are newly created. We also develop and release robust baseline models using state-of-the-art approaches in machine learning and deep learning that can aid the research in this field. Our baseline models are comparable to the state-of-the-art models for the language identification task.