Abstract
Reinforcement learning for agentic multimodal models often suffers from interaction collapse, where models learn to reduce tool usage and multi-turn reasoning, limiting the benefits of agentic behavior. We introduce PyVision-RL, a reinforcement learning framework for open-weight multimodal models that stabilizes training and sustains interaction. Our approach combines an oversampling-filtering-ranking rollout strategy with an accumulative tool reward to prevent collapse and encourage multi-turn tool use. Using a unified training pipeline, we develop PyVision-Image and PyVision-Video for image and video understanding. For video reasoning, PyVision-Video employs on-demand context construction, selectively sampling task-relevant frames during reasoning to significantly reduce visual token usage. Experiments show strong performance and improved efficiency, demonstrating that sustained interaction and on-demand visual processing are critical for scalable multimodal agents.
Abstract (translated)
基于代理的多模态模型在强化学习过程中常常会遇到交互崩溃的问题，即模型学会减少工具使用和多轮推理，从而限制了代理行为的好处。为此，我们引入了一个名为PyVision-RL的框架，它是一个面向开放权重多模态模型的强化学习框架，能够稳定训练并维持互动过程。 我们的方法结合了一种过采样-过滤-排序展开策略（oversampling-filtering-ranking rollout strategy）和累积工具奖励机制，以防止交互崩溃，并鼓励使用多轮工具。通过统一的训练流水线，我们开发了PyVision-Image和PyVision-Video，分别用于图像理解和视频理解。 对于视频推理，PyVision-Video采用按需构建上下文的方法，在推理过程中选择性地采样与任务相关的帧，从而大幅减少视觉标记使用量。实验结果表明，我们的方法在性能上表现出色，并提高了效率。这证明了可持续互动和按需视觉处理对可扩展多模态代理来说至关重要。
URL
https://arxiv.org/abs/2602.20739
