Abstract
The lightweight semi-supervised learning (LSL) strategy provides an effective approach of conserving labeled samples and minimizing model inference costs. Prior research has effectively applied knowledge transfer learning and co-training regularization from large to small models in LSL. However, such training strategies are computationally intensive and prone to local optima, thereby increasing the difficulty of finding the optimal solution. This has prompted us to investigate the feasibility of integrating three low-cost scenarios for text mining tasks: limited labeled supervision, lightweight fine-tuning, and rapid-inference small models. We propose NanoNet, a novel framework for lightweight text mining that implements parameter-efficient learning with limited supervision. It employs online knowledge distillation to generate multiple small models and enhances their performance through mutual learning regularization. The entire process leverages parameter-efficient learning, reducing training costs and minimizing supervision requirements, ultimately yielding a lightweight model for downstream inference.
Abstract (translated)
轻量级半监督学习（LSL）策略提供了一种有效的方法，用于节省标注样本并最小化模型推理成本。先前的研究已经成功地将知识迁移学习和协作训练正则化从大型模型应用到小型模型中以实现LSL的目标。然而，这类训练策略计算负担较重且容易陷入局部最优解，从而增加了找到最佳解决方案的难度。这促使我们研究在文本挖掘任务中整合三种低成本场景的可能性：有限标注监督、轻量级微调和快速推理的小型模型。为此，我们提出了一种名为NanoNet的新框架，用于轻量级文本挖掘，该框架实现参数高效的有限监督学习。它使用在线知识蒸馏来生成多个小型模型，并通过相互学习正则化提高它们的性能。整个过程利用了参数高效的学习方法，从而减少了训练成本并最小化了监督需求，最终为下游推理提供了一个轻量级模型。
URL
https://arxiv.org/abs/2602.06093
