Abstract
Unsupervised pre-trained word embeddings are used effectively for many tasks in natural language processing to leverage unlabeled textual data. Often these embeddings are either used as initializations or as fixed word representations for task-specific classification models. In this work, we extend our classification model's task loss with an unsupervised auxiliary loss on the word-embedding level of the model. This is to ensure that the learned word representations contain both task-specific features, learned from the supervised loss component, and more general features learned from the unsupervised loss component. We evaluate our approach on the task of temporal relation extraction, in particular, narrative containment relation extraction from clinical records, and show that continued training of the embeddings on the unsupervised objective together with the task objective gives better task-specific embeddings, and results in an improvement over the state of the art on the THYME dataset, using only a general-domain part-of-speech tagger as linguistic resource.
Abstract (translated)
无监督的预训练词嵌入有效地用于自然语言处理中的许多任务，以利用未标记的文本数据。这些嵌入通常用作初始化或用作特定任务分类模型的固定字表示。在这项工作中，我们扩展了分类模型的任务损失，在模型的字嵌入级别上使用无监督的辅助损失。这是为了确保学习的单词表示包含从受监督的损失组件中学习的任务特定的特征，以及从无监督的损失组件中学习的更一般的特征。我们评估了我们关于时间关系提取任务的方法，特别是从临床记录中提取叙事遏制关系，并表明继续训练嵌入在无监督目标上与任务目标一起提供了更好的任务特定嵌入，并导致对THYME数据集的最新技术水平的改进，仅使用通用域词性标注器作为语言资源。
URL
https://arxiv.org/abs/1808.02374
