Abstract
Traditional black-box distillation for Large Vision-Language Models (LVLMs) typically relies on a single teacher response per input, which often yields high-variance responses and format inconsistencies in multimodal or temporal scenarios. To mitigate this unreliable supervision, we propose R-MSD (Reliable Multi-Sample Distillation), a framework that explicitly models teacher sampling variance to enhance distillation stability. Rather than relying on a single teacher response, our approach leverages a task-adaptive teacher pool to provide robust supervision tailored to both closed-ended and open-ended reasoning. By integrating quality-aware signal matching with an adversarial distillation objective, our approach effectively filters teacher noise while maximizing knowledge transfer. Extensive evaluations across comprehensive video understanding benchmarks demonstrate that R-MSD consistently outperforms single sample distillation methods. We additionally include an original SFT+RL 4B baseline under the same training budget, which shows only marginal gains, while our method achieves significant improvements. With a 4B student model, our approach delivers gains on VideoMME (+1.5%), Video-MMMU (+3.2%), and MathVerse (+3.6%).
Abstract (translated)
传统黑盒蒸馏（Black-box distillation）对于大型视觉-语言模型（Large Vision-Language Models，LVLMs）通常依赖于每个输入只有一个教师响应的方法，这在多模态或多时间场景中往往会导致高方差的响应和格式不一致的问题。为了解决这种不可靠的监督问题，我们提出了R-MSD（Reliable Multi-Sample Distillation），这是一个通过显式建模教师样本变异性来增强蒸馏稳定性的框架。 与依赖单一教师响应不同，我们的方法利用了一个任务适应性教师池，以提供针对封闭和开放类型推理都能有效支持的稳健监督。通过整合质量感知信号匹配以及对抗性蒸馏目标，我们的方法能够有效地过滤掉教师噪声并最大化知识传递效率。 广泛的评估显示，在全面覆盖视频理解基准测试时，R-MSD始终优于单一样本蒸馏方法。我们还提供了一个原始SFT+RL 4B基线，该基线在相同训练预算下仅显示出边际改善，而我们的方法实现了显著的改进。使用一个40亿参数的学生模型，我们的方法分别在VideoMME（+1.5%）、Video-MMMU（+3.2%）和MathVerse（+3.6%）上取得了提升效果。 总结来说，R-MSD通过有效管理教师样本变异性，并结合质量感知信号匹配与对抗性蒸馏目标来增强知识的稳定传递，在大规模多模态任务中展现出优越的表现。
URL
https://arxiv.org/abs/2603.11423
