Abstract
Recent NLP advances focus primarily on standardized languages, leaving most low-resource dialects under-served especially in Indian scenarios. In India, the issue is particularly important: despite Hindi being the third most spoken language globally (over 600 million speakers), its numerous dialects remain underrepresented. The situation is similar for Odia, which has around 45 million speakers. While some datasets exist which contain standard Hindi and Odia languages, their regional dialects have almost no web presence. We introduce INDIC-DIALECT, a human-curated parallel corpus of 13k sentence pairs spanning 11 dialects and 2 languages: Hindi and Odia. Using this corpus, we construct a multi-task benchmark with three tasks: dialect classification, multiple-choice question (MCQ) answering, and machine translation (MT). Our experiments show that LLMs like GPT-4o and Gemini 2.5 perform poorly on the classification task. While fine-tuned transformer based models pretrained on Indian languages substantially improve performance e.g., improving F1 from 19.6\% to 89.8\% on dialect classification. For dialect to language translation, we find that hybrid AI model achieves highest BLEU score of 61.32 compared to the baseline score of 23.36. Interestingly, due to complexity in generating dialect sentences, we observe that for language to dialect translation the ``rule-based followed by AI" approach achieves best BLEU score of 48.44 compared to the baseline score of 27.59. INDIC-DIALECT thus is a new benchmark for dialect-aware Indic NLP, and we plan to release it as open source to support further work on low-resource Indian dialects.
Abstract (translated)
最近的自然语言处理（NLP）进展主要集中在标准化语言上，而大多数资源匮乏的语言变体则被忽视，特别是在印度的情况中尤为明显。在印度，这一问题尤为重要：尽管印地语是全球第三大使用最广泛的语言（超过6亿使用者），但其众多方言仍然未得到充分代表。对于奥利亚语来说情况也是如此，它拥有大约4500万的使用者。虽然有一些包含标准印地语和奥利亚语的数据集存在，但是这些语言的地方方言在网上几乎没有任何呈现。 为了解决这个问题，我们引入了INDIC-DIALECT，这是一个由人工编纂的平行语料库，包括13,000对句子跨越11种方言以及两种语言：印地语和奥利亚语。利用这一语料库，我们构建了一个多任务基准测试，包含了三个任务：方言分类、多项选择题（MCQ）回答以及机器翻译（MT）。我们的实验结果显示，像GPT-4o和Gemini 2.5这样的大型语言模型在方言分类任务中表现较差。然而，在印度语系上进行预训练的基于转换器的语言模型经过微调后性能显著提升，例如将F1分数从19.6%提高到了89.8%。 对于方言到标准语言的翻译，我们发现混合AI模型达到了最高的BLEU评分为61.32，而基线评分为23.36。有趣的是，由于生成方言句子复杂性的原因，在标准语言到方言的翻译中，“规则基于后跟AI”的方法达到了最佳的BLEU评分为48.44，而基线评分为27.59。 因此，INDIC-DIALECT成为了评估具有方言意识的印度语NLP的新基准，并计划作为开源资源发布以支持对低资源印度方言进一步的研究工作。
URL
https://arxiv.org/abs/2601.10388
