Abstract
Text classification approaches have usually required task-specific model architectures and huge labeled datasets. Recently, thanks to the rise of text-based transfer learning techniques, it is possible to pre-train a language model in an unsupervised manner and leverage them to perform effective on downstream tasks. In this work we focus on Japanese and show the potential use of transfer learning techniques in text classification. Specifically, we perform binary and multi-class sentiment classification on the Rakuten product review and Yahoo movie review datasets. We show that transfer learning-based approaches perform better than task-specific models trained on 3 times as much data. Furthermore, these approaches perform just as well for language modeling pre-trained on only 1/30 of the data. We release our pre-trained models and code as open source.
Abstract (translated)
文本分类方法通常需要特定于任务的模型架构和巨大的标记数据集。最近，由于基于文本的转移学习技术的兴起，可以无监督地预先训练语言模型，并利用它们有效地执行下游任务。在这项工作中，我们主要关注日语，并展示了转移学习技术在文本分类中的潜在应用。具体来说，我们对Rakuten产品评论和雅虎电影评论数据集进行了二元和多类情感分类。我们表明，基于转移学习的方法比基于3倍数据的任务特定模型的性能更好。此外，这些方法在语言建模方面也表现得很好，只对1/30的数据进行了预先培训。我们将预先培训的模型和代码作为开放源码发布。
URL
https://arxiv.org/abs/1905.09642
