Abstract
A critical gap exists between the general-purpose visual understanding of state-of-the-art physical AI models and the specialized perceptual demands of structured real-world deployment environments. We present PRISM, a 270K-sample multi-view video supervised fine-tuning (SFT) corpus for embodied vision-language-models (VLMs) in real-world retail environments. PRISM is motivated by a simple observation - physical AI systems fail not because of poor visual recognition, but because they do not understand space, physical dynamics and embodied action well enough to operate reliably in the world. To this end, PRISM is grounded in a novel three-dimensional knowledge ontology that spans spatial knowledge, temporal and physical knowledge, and embodied action knowledge. It covers 20+ capability probes across four evaluation dimensions - Embodied Reasoning (ER), Common Sense (CS), Spatial Perception (SP), and Intuitive Physics (IP), and to our knowledge, PRISM is the first dataset to instantiate all three knowledge dimensions within a single real-world deployment domain. The corpus captures data from egocentric, exocentric and 360° viewpoints across five supermarket locations and includes open-ended, chain-of-thought, and multiple-choice supervision. At 4 fps, PRISM spans approximately 11.8M video frames and approximately 730M tokens, placing it among the largest domain-specific video SFT corpora. Fine-tuning on PRISM reduces the error rate across all 20+ probes by 66.6% over the pre-trained baseline, with significant gains in embodied action understanding where the accuracy improves by 36.4%. Our results suggest that ontology-structured, domain specific SFT can meaningfully strengthen embodied VLMs for real-world settings. The PRISM dataset and more details are available at this https URL
Abstract (translated)
当前最先进的物理AI模型在通用视觉理解能力与结构化真实部署环境的专业化感知需求之间存在关键差距。我们提出了PRISM——一个面向真实零售环境中具身视觉语言模型（VLMs）的27万样本多视角视频监督微调（SFT）语料库。PRISM的提出源于一个简单观察：物理AI系统的失败并非由于视觉识别能力不足，而是因为它们对空间、物理动力学及具身动作的理解不足以支撑在现实世界中的可靠运作。为此，PRISM基于一个新颖的三维知识本体论构建，该本体论涵盖空间知识、时序与物理知识以及具身动作知识。它在四个评估维度——具身推理（ER）、常识（CS）、空间感知（SP）和直觉物理（IP）——中覆盖了20余项能力探测。据我们所知，PRISM是首个在单一真实部署领域内实现全部三个知识维度的数据集。该语料库采集自五个超市场景中的第一人称、第三人称及360°视角数据，并包含开放式、思维链及多项选择等多种监督形式。以4帧/秒的帧率计算，PRISM涵盖约1180万视频帧及约7.3亿token，使其成为规模最大的领域专用视频SFT语料库之一。在PRISM上进行微调后，所有20余项探测任务的错误率相比预训练基线降低66.6%，其中具身动作理解领域的准确率提升尤为显著，达到36.4%。我们的结果表明，基于本体论结构化、领域专用的SFT能够有效增强具身VLMs在真实场景中的性能。PRISM数据集及更多详情可通过此https URL获取。
URL
https://arxiv.org/abs/2603.29281
