Abstract
The representation of events in text plays a significant role in various NLP tasks. Recent research demonstrates that contrastive learning has the ability to improve event comprehension capabilities of Pre-trained Language Models (PLMs) and enhance the performance of event representation learning. However, the efficacy of event representation learning based on contrastive learning and PLMs is limited by the short length of event texts. The length of event texts differs significantly from the text length used in the pre-training of PLMs. As a result, there is inconsistency in the distribution of text length between pre-training and event representation learning, which may undermine the learning process of event representation based on PLMs. In this study, we present PromptCL, a novel framework for event representation learning that effectively elicits the capabilities of PLMs to comprehensively capture the semantics of short event texts. PromptCL utilizes a Prompt template borrowed from prompt learning to expand the input text during Contrastive Learning. This helps in enhancing the event representation learning by providing a structured outline of the event components. Moreover, we propose Subject-Predicate-Object (SPO) word order and Event-oriented Masked Language Modeling (EventMLM) to train PLMs to understand the relationships between event components. Our experimental results demonstrate that PromptCL outperforms state-of-the-art baselines on event related tasks. Additionally, we conduct a thorough analysis and demonstrate that using a prompt results in improved generalization capabilities for event representations. Our code will be available at this https URL.
Abstract (translated)
文本中事件表示在各种自然语言处理任务中扮演着重要角色。最近的研究表明，对比学习有能力提高预训练语言模型（PLMs）的事件理解能力，并增强事件表示学习的效果。然而，基于对比学习和PLMs的事件表示学习的效果受到短事件文本长度的影响。事件文本的长度与PLMs预训练时使用的文本长度显著不同。因此，在预训练和事件表示学习之间文本长度的分布存在不稳定性，这可能削弱基于PLMs的事件表示学习过程。在本研究中，我们提出了PromptCL，一种新颖的事件表示学习框架，有效地激发了PLMs的全面理解短事件文本的能力。PromptCL利用从提示学习中借用的提示模板在对比学习期间扩展输入文本。这有助于增强事件表示学习，通过提供事件组件的有序结构。此外，我们提出了Subject-Predicate-Object（SPO）词序和Event-oriented Masked Language Modeling（EventMLM）来训练PLMs理解事件组件之间的关系。我们的实验结果表明，PromptCL在事件相关任务上优于最先进的基线。此外，我们进行了详细的分析和演示，使用提示会导致事件表示的泛化能力得到改善。我们的代码将在此处https:// URL上提供。
URL
https://arxiv.org/abs/2404.17877
