Abstract
Code-mixing is the practice of using two or more languages in a single sentence, which often occurs in multilingual communities such as India where people commonly speak multiple languages. Classic NLP tools, trained on monolingual data, face challenges when dealing with code-mixed data. Extracting meaningful information from sentences containing multiple languages becomes difficult, particularly in tasks like hate speech detection, due to linguistic variation, cultural nuances, and data sparsity. To address this, we aim to analyze the significance of code-mixed embeddings and evaluate the performance of BERT and HingBERT models (trained on a Hindi-English corpus) in hate speech detection. Our study demonstrates that HingBERT models, benefiting from training on the extensive Hindi-English dataset L3Cube-HingCorpus, outperform BERT models when tested on hate speech text datasets. We also found that code-mixed Hing-FastText performs better than standard English FastText and vanilla BERT models.
Abstract (translated)
代码混用是指在单个句子中使用两种或更多语言的做法，这通常发生在多语种社区，如印度，在那里人们通常讲多种语言。传统的自然语言处理工具是在单一语言数据上训练的，当面对代码混用的数据时会遇到挑战。从包含多种语言的句子中提取有意义的信息变得困难，特别是在仇恨言论检测等任务中，这是由于语言变异、文化细微差别和数据稀疏性造成的。为了解决这个问题，我们旨在分析代码混用嵌入的重要性，并评估在印地语-英语语料库上训练的BERT和HingBERT模型在仇恨言论检测中的性能表现。我们的研究表明，在测试仇恨言论文本数据集时，得益于在广泛的印地语-英语数据集L3Cube-HingCorpus上的训练，HingBERT模型优于BERT模型。我们还发现代码混用的Hing-FastText比标准英文FastText和普通的BERT模型表现出更好的性能。
URL
https://arxiv.org/abs/2411.18577
