Abstract
Current state-of-the-art relation extraction methods typically rely on a set of lexical, syntactic, and semantic features, explicitly computed in a pre-processing step. Training feature extraction models requires additional annotated language resources, which severely restricts the applicability and portability of relation extraction to novel languages. Similarly, pre-processing introduces an additional source of error. To address these limitations, we introduce TRE, a Transformer for Relation Extraction, extending the OpenAI Generative Pre-trained Transformer [Radford et al., 2018]. Unlike previous relation extraction models, TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions. TRE allows us to learn implicit linguistic features solely from plain text corpora by unsupervised pre-training, before fine-tuning the learned language representations on the relation extraction task. TRE obtains a new state-of-the-art result on the TACRED and SemEval 2010 Task 8 datasets, achieving a test F1 of 67.4 and 87.1, respectively. Furthermore, we observe a significant increase in sample efficiency. With only 20% of the training examples, TRE matches the performance of our baselines and our model trained from scratch on 100% of the TACRED dataset. We open-source our trained models, experiments, and source code.
Abstract (translated)
当前最先进的关系提取方法通常依赖于一组在预处理步骤中显式计算的词汇、句法和语义特征。训练特征提取模型需要额外的注释语言资源，这严重制约了关系提取对新语言的适用性和可移植性。同样，预处理引入了一个额外的错误源。为了解决这些局限性，我们引入了用于关系提取的变压器tre，扩展了openai生成式预培训变压器【Radford等人，2018年】。与以前的关系提取模型不同，Tre使用预先训练的深层语言表示，而不是明确的语言特征来通知关系分类，并将其与自我关注的转换结构相结合，以有效地建模实体提及之间的长期依赖关系。tre允许我们在对关系提取任务中学习的语言表示进行微调之前，通过无监督的预训练，仅从纯文本语料库中学习隐含的语言特征。Tre在Tacred和Semeval 2010任务8数据集上获得了一个新的最先进的结果，分别实现了67.4和87.1的测试f1。此外，我们观察到样品效率显著提高。只有20%的训练示例，tre匹配基线的性能，我们的模型在100%的隐性数据集上从头开始训练。我们开放源码我们训练的模型、实验和源代码。
URL
https://arxiv.org/abs/1906.03088
