Abstract
While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in image and video understanding, their ability to comprehend the physical world has become an increasingly important research focus. Despite their improvements, current MLLMs struggle significantly with high-level physics reasoning. In this work, we investigate the first step of physical reasoning, i.e., intuitive physics understanding, revealing substantial limitations in understanding the dynamics of continuum objects. To isolate and evaluate this specific capability, we introduce two fundamental benchmark tasks: Next Frame Selection (NFS) and Temporal Coherence Verification (TCV). Our experiments demonstrate that even state-of-the-art MLLMs perform poorly on these foundational tasks. To address this limitation, we propose Scene Dynamic Field (SDF), a concise approach that leverages physics simulators within a multi-task fine-tuning framework. SDF substantially improves performance, achieving up to 20.7% gains on fluid tasks while showing strong generalization to unseen physical domains. This work not only highlights a critical gap in current MLLMs but also presents a promising cost-efficient approach for developing more physically grounded MLLMs. Our code and data are available at this https URL.
Abstract (translated)
尽管多模态大语言模型（MLLMs）在图像与视频理解方面已展现出卓越能力，但其对物理世界的理解正日益成为关键研究焦点。尽管性能持续提升，当前MLLMs在高阶物理推理方面仍存在显著不足。本研究聚焦物理推理的首个环节——直觉物理理解，揭示了模型在连续体物体动态认知上的重大局限。为独立评估该能力，我们引入两项基础基准任务：下一帧选择（NFS）与时序一致性验证（TCV）。实验表明，即便是最先进的MLLMs在这些基础任务上表现仍欠佳。针对此局限，我们提出场景动态场（SDF）方法——一种基于多任务微调框架、融合物理模拟器的简洁方案。SDF显著提升性能，在流体任务上最高实现20.7%的增益，并对未见物理领域展现出强泛化能力。本研究不仅揭示了当前MLLMs的关键缺陷，更为开发更具物理基础性的模型提供了高性价比的可行路径。代码与数据已公开于此链接。
URL
https://arxiv.org/abs/2604.03302
