Abstract
We present MetaSpatial, the first reinforcement learning (RL)-based framework designed to enhance 3D spatial reasoning in vision-language models (VLMs), enabling real-time 3D scene generation without the need for hard-coded optimizations. MetaSpatial addresses two core challenges: (i) the lack of internalized 3D spatial reasoning in VLMs, which limits their ability to generate realistic layouts, and (ii) the inefficiency of traditional supervised fine-tuning (SFT) for layout generation tasks, as perfect ground truth annotations are unavailable. Our key innovation is a multi-turn RL-based optimization mechanism that integrates physics-aware constraints and rendered image evaluations, ensuring generated 3D layouts are coherent, physically plausible, and aesthetically consistent. Methodologically, MetaSpatial introduces an adaptive, iterative reasoning process, where the VLM refines spatial arrangements over multiple turns by analyzing rendered outputs, improving scene coherence progressively. Empirical evaluations demonstrate that MetaSpatial significantly enhances the spatial consistency and formatting stability of various scale models. Post-training, object placements are more realistic, aligned, and functionally coherent, validating the effectiveness of RL for 3D spatial reasoning in metaverse, AR/VR, digital twins, and game development applications. Our code, data, and training pipeline are publicly available at this https URL.
Abstract (translated)
我们介绍了MetaSpatial，这是一个基于强化学习（RL）的框架，旨在增强视觉语言模型（VLMs）中的3D空间推理能力，使其能够在无需硬编码优化的情况下实现实时3D场景生成。MetaSpatial解决了两个核心挑战：(i) VLM中缺乏内在化的3D空间推理能力，这限制了它们生成逼真布局的能力；(ii) 传统监督微调（SFT）方法在布局生成任务中的低效性，因为无法获得完美的地面真实标注。 我们的关键创新是一个基于多轮RL的优化机制，该机制集成了物理感知约束和渲染图像评估，确保生成的3D布局具有连贯性、物理合理性及美学一致性。从方法论的角度来看，MetaSpatial引入了一个自适应迭代推理过程，在这个过程中，VLM通过分析渲染输出来逐步改进空间布置，从而提高场景的一致性。 实证研究表明，MetaSpatial显著增强了各种规模模型的空间一致性和格式稳定性。经过训练后，物体放置变得更加逼真、对齐且功能上连贯，这验证了RL在元宇宙、AR/VR、数字孪生和游戏开发应用中的3D空间推理能力的有效性。我们的代码、数据及培训流程可在以下网址公开获取：[此处插入URL]。
URL
https://arxiv.org/abs/2503.18470
