Abstract
In the realm of Natural Language Processing (NLP), common approaches for handling human disagreement consist of aggregating annotators' viewpoints to establish a single ground truth. However, prior studies show that disregarding individual opinions can lead can lead to the side effect of underrepresenting minority perspectives, especially in subjective tasks, where annotators may systematically disagree because of their preferences. Recognizing that labels reflect the diverse backgrounds, life experiences, and values of individuals, this study proposes a new multi-perspective approach using soft labels to encourage the development of the next generation of perspective aware models, more inclusive and pluralistic. We conduct an extensive analysis across diverse subjective text classification tasks, including hate speech, irony, abusive language, and stance detection, to highlight the importance of capturing human disagreements, often overlooked by traditional aggregation methods. Results show that the multi-perspective approach not only better approximates human label distributions, as measured by Jensen-Shannon Divergence (JSD), but also achieves superior classification performance (higher F1 scores), outperforming traditional approaches. However, our approach exhibits lower confidence in tasks like irony and stance detection, likely due to the inherent subjectivity present in the texts. Lastly, leveraging Explainable AI (XAI), we explore model uncertainty and uncover meaningful insights into model predictions.
Abstract (translated)
在自然语言处理（NLP）领域，处理人类意见分歧的常见方法是汇总标注者的观点以确立单一的事实基准。然而，先前的研究表明，忽视个体意见会导致少数派视角代表性不足的问题，尤其是在主观任务中，由于个人偏好，标注者可能系统性地持有不同意见。考虑到标签反映了个人背景、生活经验和价值观的多样性，本研究提出了一种新的多视角方法，使用软标签来促进下一代视角感知模型的发展，使其更加包容和多元化。我们在包括仇恨言论、讽刺、滥用语言以及立场检测在内的多种主观文本分类任务上进行了广泛的分析，以强调捕捉人类意见分歧的重要性，这些分歧往往被传统的汇总方法所忽略。研究结果表明，多视角方法不仅在用杰森-香农散度（JSD）衡量的人类标签分布方面更接近真实情况，而且还在分类性能（更高的F1得分）上表现出色，优于传统方法。然而，在讽刺和立场检测等任务中，我们的方法显示出较低的置信度，这可能是由于文本中存在的固有主观性所致。最后，通过可解释人工智能（XAI），我们探讨了模型不确定性，并揭示了有关模型预测的重要见解。
URL
https://arxiv.org/abs/2506.20209
