Abstract
How can we teach large multimodal models (LMMs) new skills without erasing prior abilities? We study sequential fine-tuning on five target skills while monitoring general ability on eight held-out benchmarks across three model families. We observe that apparent "forgetting" on held-out tasks after narrow fine-tuning can partly recover at later stages. We trace this behavior to a measurable shift in the output token distribution, manifested through a simple counting-bias probe that co-varies with forgetting. Guided by this picture, we identify two simple, robust tuning recipes that learn strongly while limiting drift: (i) updating only the self-attention projection layers, and (ii) updating only the MLP Gate&Up while freezing the Down projection. Across models and tasks, these choices deliver strong target gains while largely preserving held-out performance. Code is available at this https URL
Abstract (translated)
如何在不抹去先前能力的情况下教会大规模多模态模型（LMMs）新技能？我们研究了在三个不同的模型家族中，在五个目标技能上进行顺序微调的同时，监测八个保留基准上的通用能力。我们观察到，在特定任务的微调后，似乎会出现对保留任务的“遗忘”，但这种遗忘可以在后续阶段部分恢复。我们将这一行为追溯到输出标记分布的可测量变化，并通过一个简单的计数偏差探测器来体现，该探测器与遗忘呈协变关系。 基于此图景，我们确定了两种简单而稳健的微调方案，在这些方案中，模型能够强烈学习新技能的同时限制漂移：（i）仅更新自注意力投影层；（ii）仅更新MLP门控和上层同时冻结下层。无论是在不同模型还是任务上，这些选择都能在很大程度上保留未见数据的表现，同时实现显著的目标改进。 代码可在以下链接获取：[此URL]（请将此处的"this https URL"替换为实际提供的具体网址）。
URL
https://arxiv.org/abs/2510.08564
