Abstract
In the natural language processing literature, neural networks are becoming increasingly deeper and complex. The recent poster child of this trend is the deep language representation model, which includes BERT, ELMo, and GPT. These developments have led to the conviction that previous-generation, shallower neural networks for language understanding are obsolete. In this paper, however, we demonstrate that rudimentary, lightweight neural networks can still be made competitive without architecture changes, external training data, or additional input features. We propose to distill knowledge from BERT, a state-of-the-art language representation model, into a single-layer BiLSTM, as well as its siamese counterpart for sentence-pair tasks. Across multiple datasets in paraphrasing, natural language inference, and sentiment classification, we achieve comparable results with ELMo, while using roughly 100 times fewer parameters and 15 times less inference time.
Abstract (translated)
在自然语言处理文献中，神经网络越来越深入和复杂。这一趋势的最新支柱是深层次语言表达模型，其中包括伯特、埃尔莫和GPT。这些发展使人们相信，上一代的、用于语言理解的较浅的神经网络已经过时了。然而，在本文中，我们证明了在不改变体系结构、外部训练数据或附加输入特性的情况下，基本的、轻量级的神经网络仍然具有竞争力。我们建议将伯特的知识，一种最先进的语言表达模型，提炼成一个单层的bilstm，以及它的暹罗语对应的句子对任务。在释义、自然语言推理和情感分类的多个数据集中，我们使用大约100倍的参数和15倍的推理时间，获得了与ELMO类似的结果。
URL
https://arxiv.org/abs/1903.12136
