Abstract
Pre-trained language models such as BERT have been proved to be powerful in many natural language processing tasks. But in some text classification applications such as emotion recognition and sentiment analysis, BERT may not lead to satisfactory performance. This often happens in applications where keywords play critical roles in the prediction of class labels. Our investigation found that the root cause of the problem is that the context-based BERT embedding of the keywords may not be discriminative enough to produce discriminative text representation for classification. Motivated by this finding, we develop a method to enhance word embeddings using domain-specific lexical knowledge. The knowledge-based embedding enhancement model projects the BERT embedding into a new space where within-class similarity and between-class difference are maximized. To implement the knowledge-based word embedding enhancement model, we also develop a knowledge acquisition algorithm for automatically collecting lexical knowledge from online open sources. Experiment results on three classification tasks, including sentiment analysis, emotion recognition and question answering, have shown the effectiveness of our proposed word embedding enhancing model. The codes and datasets are in this https URL.
Abstract (translated)
预训练语言模型（如BERT）已被证明在许多自然语言处理任务中非常强大。但在某些文本分类应用，例如情感识别和情感分析中，BERT可能无法达到令人满意的效果。这种情况往往发生在关键词在类别预测中起关键作用的应用场景中。我们的研究发现该问题的根本原因是基于上下文的BERT对关键词的嵌入不够具有区分性，从而不能生成用于分类的有效文本表示。 鉴于此发现，我们开发了一种使用特定领域的词汇知识来增强词嵌入的方法。这种基于知识的嵌入增强模型将BERT嵌入投影到一个新的空间，在该空间中同一类别的相似性和不同类别之间的差异被最大化了。为了实现基于知识的单词嵌入增强模型，我们也开发了一个自动从在线公开资源收集词汇知识的知识获取算法。 我们在情感分析、情绪识别和问答三项分类任务上的实验结果表明我们提出的词嵌入增强模型的有效性。代码和数据集可在以下链接中找到：[这里插入URL]。
URL
https://arxiv.org/abs/2506.01621
