Abstract
Continual post-training (CPT) is a popular and effective technique for adapting foundation models like multimodal large language models to specific and ever-evolving downstream tasks. While existing research has primarily concentrated on methods like data replay, model expansion, or parameter regularization, the fundamental role of the learning paradigm within CPT remains largely unexplored. This paper presents a comparative analysis of two core post-training paradigms: supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT), investigating their respective impacts on knowledge retention during CPT. Our experiments are conducted on a benchmark comprising seven diverse multimodal tasks, utilizing Qwen2.5-VL-7B-Instruct as the base model for continual post-training. The investigation yields two significant findings: (1) When continuously learning on downstream tasks, SFT leads to catastrophic forgetting of previously learned tasks. In contrast, RFT inherently preserves prior knowledge and achieve performance comparable to multi-task training. (2) RFT successfully protects and even enhances the model's general knowledge on standard benchmarks (e.g., MMMU and MMLU-Pro). Conversely, SFT degrades general model capabilities severely. Further analysis shows that explicit mechanisms, such as KL penalty and chain-of-thought reasoning, are not the primary factors. Instead, we find that the implicit regularization inherent to RFT is a key factor in mitigating forgetting. Finally, we propose a rollout-based instance filtering algorithm to improve the stability and efficiency of RFT. Our comprehensive study demonstrates the superiority of RFT as a robust paradigm for continual post-training.
Abstract (translated)
持续后训练（CPT）是将基础模型如多模态大型语言模型适应特定且不断发展的下游任务的一种流行且有效的方法。尽管现有研究主要集中在数据重播、模型扩展或参数正则化等方法上，但CPT中学习范式的基本作用仍鲜有探讨。本文对比分析了两种核心后训练范式：监督微调（SFT）和强化微调（RFT），探究它们在持续后训练过程中对知识保留的不同影响。我们的实验是在包含七个多样化多模态任务的基准上进行，以Qwen2.5-VL-7B-Instruct作为基础模型用于持续后训练。 研究结果得出两个重要的发现： 1. 在不断学习下游任务时，SFT会导致对先前所学任务的灾难性遗忘。相反，RFT能够固有地保留之前的知识，并实现与多任务训练相当的表现。 2. RFT成功保护并甚至增强了模型在标准基准测试（如MMMU和MMLU-Pro）上的通用知识。而SFT则严重削弱了模型的一般能力。 进一步分析表明，明确机制，例如KL惩罚和链式思维推理，并非是主要因素。相反，我们发现RFT中的隐含正则化是减少遗忘的关键因素。最后，我们提出了一种基于rollout的实例过滤算法，以提高RFT的稳定性和效率。我们的全面研究表明，RFT作为持续后训练的一种稳健范式具有明显优势。
URL
https://arxiv.org/abs/2507.05386
