Abstract
In the Indian subcontinent, Telugu, one of India's six classical languages, is the most widely spoken Dravidian Language. Despite its 96 million speaker base worldwide, Telugu remains underrepresented in the global NLP and Machine Learning landscape, mainly due to lack of high-quality annotated resources. This work introduces TeSent, a comprehensive benchmark dataset for sentiment classification, a key text classification problem, in Telugu. TeSent not only provides ground truth labels for the sentences, but also supplements with provisions for evaluating explainability and fairness, two critical requirements in modern-day machine learning tasks. We scraped Telugu texts covering multiple domains from various social media platforms, news websites and web-blogs to preprocess and generate 26,150 sentences, and developed a custom-built annotation platform and a carefully crafted annotation protocol for collecting the ground truth labels along with their human-annotated rationales. We then fine-tuned several SOTA pre-trained models in two ways: with rationales, and without rationales. Further, we provide a detailed plausibility and faithfulness evaluation suite, which exploits the rationales, for six widely used post-hoc explainers applied on the trained models. Lastly, we curate TeEEC, Equity Evaluation Corpus in Telugu, a corpus to evaluate fairness of Telugu sentiment and emotion related NLP tasks, and provide a fairness evaluation suite for the trained classifier models. Our experimental results suggest that training with rationales may improve model accuracy, reduce bias in models, and make the explainers' output more aligned to human reasoning.
Abstract (translated)
在印度次大陆，泰卢固语（Telugu）是印度六种古典语言之一，也是使用最广泛的德拉威语系语言。尽管其全球使用者数量达到9600万，但由于高质量标注资源的缺乏，泰卢固语在全球自然语言处理和机器学习领域中仍处于相对边缘化的地位。为此，本研究推出了TeSent，一个全面的基准数据集，用于泰卢固语情感分类这一关键文本分类问题的研究。 TeSent不仅提供了句子的真实标签，还补充了评估可解释性和公平性的机制，这两者是现代机器学习任务中的重要需求。我们从各种社交媒体平台、新闻网站和网络博客中抓取了大量的泰卢固语文本，涵盖多个领域，并预处理生成了26,150个句子。同时，我们建立了一个定制的标注平台以及一个精心设计的标注协议来收集这些真实的标签及其人工标注的理由。 接着，我们在两个方面对几个最先进的预训练模型进行了微调：一是带有理由的情况；二是不带理由的情况。此外，我们还提供了一套详细的合理性与忠实性评估工具，这套工具利用了上述理由，并应用于六种广泛使用的后处理解释器，在训练后的模型上进行应用。 最后，为了评估泰卢固语情感和情绪相关的自然语言处理任务的公平性，我们整理出了TeEEC（Telugu Equity Evaluation Corpus），这是一份专门用于评估此类任务中模型公平性的语料库，并且为这些经过训练的分类器提供了一套公平性评估工具。 实验结果表明，使用理由进行训练可以提高模型的准确性，减少模型中的偏见，并使解释器的输出更加符合人类推理。
URL
https://arxiv.org/abs/2508.01486
