Abstract
While current Multimodal Large Language Models (MLLMs) have demonstrated proficiency in reasoning tasks such as mathematics and logic, their capacity for long-chain reflective reasoning, a prerequisite for solving complex real-world problems, remains largely underexplored. In this work, we first conduct an extensive empirical investigation to evaluate this capability. Leveraging a carefully designed data synthesis engine, we construct MM-HELIX, a multimodal benchmark consisting 1,260 samples of 42 challenging synthetic tasks that require iterative thinking and backtracking. Empirical results on this benchmark reveal that existing MLLMs exhibit significant performance deficits in long-chain reflective reasoning. To address this limitation, we generate post-training data and further explore learning paradigms for exploiting such data. We first develop the Step-Elicited Response Generation pipeline to create MM-HELIX-100K, a large-scale dataset of 100k high-quality, reflective reasoning traces for instruction-tuning stage. Given that standard Reinforcement Learning fails on complex tasks due to sparse reward signals and catastrophic forgetting after Supervised Fine-Tuning, we propose Adaptive Hybrid Policy Optimization (AHPO), a novel training strategy that dynamically unifies offline supervision and online optimization into a single stage. This strategy enables the model to learn from expert data when rewards are sparse and conduct independent exploration once proficient. When applied to the Qwen2.5-VL-7B baseline, our method achieves a +18.6\% accuracy improvement on MM-HELIX benchmark and demonstrates strong generalization with a +5.7\% average performance gain on general mathematic and logic tasks. Our work demonstrate that reflective reasoning in MLLMs can be effectively learned and generalized, paving the way for developing more capable MLLMs.
Abstract (translated)
尽管当前的多模态大型语言模型（MLLMs）在数学和逻辑等推理任务中表现出色，但它们对于长链反思性推理的能力，即解决复杂现实世界问题的前提条件，依然鲜有研究。在这项工作中，我们首先进行了一次详尽的经验调查来评估这种能力。通过精心设计的数据合成引擎，我们构建了MM-HELIX，这是一个包含1,260个需要迭代思维和回溯的42个具有挑战性的合成任务的多模态基准。在这一基准上的实证结果揭示了现有MLLMs在长链反思性推理中的表现存在显著不足。 为了解决这一局限，我们生成了后训练数据，并进一步探索利用这些数据的学习范式。首先开发了Step-Elicited Response Generation（步骤诱发响应生成）管道来创建MM-HELIX-100K，这是一个包含10万条高质量反思性推理轨迹的大型数据集，用于指令微调阶段。 鉴于标准强化学习在复杂任务上由于稀疏奖励信号和监督微调后的灾难性遗忘而无法取得成功，我们提出了自适应混合策略优化（AHPO），这是一种新的训练策略，能够动态地将离线监督与在线优化结合在一个阶段内。这一策略使模型能够在奖励稀缺时从专家数据中学习，并在熟练后进行独立探索。 当应用于Qwen2.5-VL-7B基准时，我们的方法在MM-HELIX基准上实现了18.6%的准确率提升，并且在一般的数学和逻辑任务上表现出显著的泛化能力，平均性能提高了5.7%。本工作证明了MLLMs中的反思性推理可以有效地被学习并推广，为开发更强大的MLLMs铺平了道路。
URL
https://arxiv.org/abs/2510.08540
