Abstract
Recurrent Neural Networks (RNNs) are widely used in the field of natural language processing (NLP), ranging from text categorization to question answering and machine translation. However, RNNs generally read the whole text from beginning to end or vice versa sometimes, which makes it inefficient to process long texts. When reading a long document for a categorization task, such as topic categorization, large quantities of words are irrelevant and can be skipped. To this end, we propose Leap-LSTM, an LSTM-enhanced model which dynamically leaps between words while reading texts. At each step, we utilize several feature encoders to extract messages from preceding texts, following texts and the current word, and then determine whether to skip the current word. We evaluate Leap-LSTM on several text categorization tasks: sentiment analysis, news categorization, ontology classification and topic classification, with five benchmark data sets. The experimental results show that our model reads faster and predicts better than standard LSTM. Compared to previous models which can also skip words, our model achieves better trade-offs between performance and efficiency.
Abstract (translated)
递归神经网络(RNN)广泛应用于自然语言处理(NLP)领域,从文本分类到问答和机器翻译。然而,RNN通常从头到尾阅读整个文本,有时反之亦然,这使得处理长文本效率低下。当阅读分类任务(如主题分类)的长文档时,大量的单词是不相关的,可以跳过。为此,我们提出了LEAP LSTM,这是一个LSTM增强的模型,它在阅读文本时在单词之间动态跳跃。在每一步中,我们使用几个特征编码器从前面的文本、后面的文本和当前单词中提取消息,然后确定是否跳过当前单词。我们使用五个基准数据集对LeapLSTM进行了评估,包括情感分析、新闻分类、本体分类和主题分类。实验结果表明,该模型比标准LSTM具有更快的读取速度和更好的预测能力。与以前可以跳过单词的模型相比,我们的模型在性能和效率之间实现了更好的权衡。
URL
https://arxiv.org/abs/1905.11558