Abstract
Empowered by large-scale training, vision-language models (VLMs) achieve strong image and video understanding, yet their ability to perform spatial reasoning in both static scenes and dynamic videos remains limited. Recent advances try to handle this limitation by injecting geometry tokens from pretrained 3D foundation models into VLMs. Nevertheless, we observe that naive token fusion followed by standard fine-tuning in this line of work often leaves such geometric cues underutilized for spatial reasoning, as VLMs tend to rely heavily on 2D visual cues. In this paper, we propose GeoSR, a framework designed to make geometry matter by encouraging VLMs to actively reason with geometry tokens. GeoSR introduces two key components: (1) Geometry-Unleashing Masking, which strategically masks portions of 2D vision tokens during training to weaken non-geometric shortcuts and force the model to consult geometry tokens for spatial reasoning; and (2) Geometry-Guided Fusion, a gated routing mechanism that adaptively amplifies geometry token contributions in regions where geometric evidence is critical. Together, these designs unleash the potential of geometry tokens for spatial reasoning tasks. Extensive experiments on both static and dynamic spatial reasoning benchmarks demonstrate that GeoSR consistently outperforms prior methods and establishes new state-of-the-art performance by effectively leveraging geometric information. The project page is available at this https URL.
Abstract (translated)
在大规模训练的赋能下，视觉语言模型（VLMs）已具备强大的图像与视频理解能力，但其在静态场景与动态视频中的空间推理能力仍存在局限。近期研究尝试通过将预训练三维基础模型中的几何令牌注入VLMs来应对这一挑战。然而我们观察到，此类工作中简单的令牌融合配合标准微调往往导致几何线索在空间推理中利用不足，因为VLMs倾向于过度依赖二维视觉线索。本文提出GeoSR框架，旨在通过促使VLMs主动运用几何令牌来提升几何信息的重要性。GeoSR引入两个关键组件：（1）几何释放掩码（Geometry-Unleashing Masking），在训练过程中策略性掩蔽部分二维视觉令牌，以削弱非几何捷径并迫使模型在空间推理中 consult 几何令牌；（2）几何引导融合（Geometry-Guided Fusion），一种自适应增强几何令牌在关键几何证据区域贡献的门控路由机制。这些设计共同释放了几何令牌在空间推理任务中的潜力。在静态与动态空间推理基准测试上的大量实验表明，GeoSR通过有效利用几何信息持续优于先前方法，并建立了新的最先进性能。项目页面可通过此链接访问。
URL
https://arxiv.org/abs/2603.26639
