Abstract
In this paper, we investigate a novel deep-model reusing task. Our goal is to train a lightweight and versatile student model, without human-labelled annotations, that amalgamates the knowledge and masters the expertise of two pretrained teacher models working on heterogeneous problems, one on scene parsing and the other on depth estimation. To this end, we propose an innovative training strategy that learns the parameters of the student intertwined with the teachers, achieved by 'projecting' its amalgamated features onto each teacher's domain and computing the loss. We also introduce two options to generalize the proposed training strategy to handle three or more tasks simultaneously. The proposed scheme yields very encouraging results. As demonstrated on several benchmarks, the trained student model achieves results even superior to those of the teachers in their own expertise domains and on par with the state-of-the-art fully supervised models relying on human-labelled annotations.
Abstract (translated)
本文研究了一种新的深度模型重用任务。我们的目标是培养一个轻量级和多功能的学生模型，不需要人工标记的注释，它融合了知识，掌握了两个预先培训的教师模型在处理异构问题时的专业知识，一个在现场分析，另一个在深度估计。为此，我们提出了一种创新的培训策略，通过将学生的综合特征“投射”到每个教师的领域并计算损失，来学习与教师交织在一起的参数。我们还介绍了两种方法来概括所提出的同时处理三个或更多任务的训练策略。提出的方案产生了非常令人鼓舞的结果。正如在多个基准上所证明的那样，经过培训的学生模型在各自的专业领域取得的成果甚至优于教师的成果，并且与最先进的完全受监督的模型相当，这种模型依赖于人类标记的注释。
URL
https://arxiv.org/abs/1904.10167
