Abstract
Vision-Language Models (VLMs) have achieved remarkable progress, yet their large scale often renders them impractical for resource-constrained environments. This paper introduces Unified Reinforcement and Imitation Learning (RIL), a novel and efficient training algorithm designed to create powerful, lightweight VLMs. RIL distinctively combines the strengths of reinforcement learning with adversarial imitation learning. This enables smaller student VLMs not only to mimic the sophisticated text generation of large teacher models but also to systematically improve their generative capabilities through reinforcement signals. Key to our imitation framework is an LLM-based discriminator that adeptly distinguishes between student and teacher outputs, complemented by guidance from multiple large teacher VLMs to ensure diverse learning. This unified learning strategy, leveraging both reinforcement and imitation, empowers student models to achieve significant performance gains, making them competitive with leading closed-source VLMs. Extensive experiments on diverse vision-language benchmarks demonstrate that RIL significantly narrows the performance gap with state-of-the-art open- and closed-source VLMs and, in several instances, surpasses them.
Abstract (translated)
视觉语言模型（VLMs）已经取得了显著的进步，但其庞大的规模往往使其在资源受限的环境中难以应用。本文介绍了统一强化与模仿学习（RIL），这是一种新颖且高效的训练算法，旨在创建强大而轻量级的VLM。RIL独特地结合了强化学习和对抗性模仿学习的优势。这使得较小的学生VLM不仅能够模仿大型教师模型复杂的文本生成能力，还能通过强化信号系统性地提高其生成能力。 我们所提出的模仿框架的关键是一个基于大规模语言模型（LLM）的判别器，它能巧妙地区分学生模型与教师模型的输出，并辅以多个大型教师VLM的指导，确保多样化的学习。这种统一的学习策略，同时利用了强化和模仿的优势，使学生模型能够实现显著的性能提升，使其在与领先闭源VLM的竞争中具备竞争力。 广泛的实验证明，在多种视觉语言基准上，RIL大大缩小了与最先进的开源和闭源VLM之间的性能差距，并且在某些情况下超越了它们。
URL
https://arxiv.org/abs/2510.19307
