Abstract
Deploying natural language processing (NLP) models on mobile platforms requires models that can adapt across diverse applications while remaining efficient in memory and computation. We investigate pre-finetuning strategies to enhance the adaptability of lightweight BERT-like encoders for two fundamental NLP task families: named entity recognition (NER) and text classification. While pre-finetuning improves downstream performance for each task family individually, we find that naïve multi-task pre-finetuning introduces conflicting optimization signals that degrade overall performance. To address this, we propose a simple yet effective multi-task pre-finetuning framework based on task-primary LoRA modules, which enables a single shared encoder backbone with modular adapters. Our approach achieves performance comparable to individual pre-finetuning while meeting practical deployment constraint. Experiments on 21 downstream tasks show average improvements of +0.8% for NER and +8.8% for text classification, demonstrating the effectiveness of our method for versatile mobile NLP applications.
Abstract (translated)
在移动平台上部署自然语言处理（NLP）模型需要能够在多种应用场景中适应并保持内存和计算效率的模型。我们研究了预微调策略，以增强轻量级BERT类似编码器对两种基本NLP任务类别：命名实体识别（NER）和文本分类的适用性。虽然单独的任务预微调可以改善每个任务类别的下游性能，但我们发现单纯的多任务预微调会引入相互冲突的优化信号，从而降低整体表现。 为了解决这个问题，我们提出了一种基于主要任务LoRA模块的简单而有效的多任务预微调框架，这使得单一共享编码器主干能够通过模块化适配器实现。我们的方法在满足实际部署约束的同时，实现了与单独预微调相当的表现水平。在21个下游任务上的实验表明，对于NER任务改进了+0.8%，对于文本分类改进了+8.8%，展示了我们方法在移动NLP应用中的通用性及有效性。
URL
https://arxiv.org/abs/2510.07566
