Abstract
Constructing an accurate simulation model of real-world environments requires reliable estimation of physical parameters such as mass, geometry, friction, and contact surfaces. Traditional real-to-simulation (Real2Sim) pipelines rely on manual measurements or fixed, pre-programmed exploration routines, which limit their adaptability to varying tasks and user intents. This paper presents a Real2Sim framework that autonomously generates and executes Behavior Trees for task-specific physical interactions to acquire only the parameters required for a given simulation objective, without relying on pre-defined task templates or expert-designed exploration routines. Given a high-level user request, an incomplete simulation description, and an RGB observation of the scene, a vision-language model performs multi-modal reasoning to identify relevant objects, infer required physical parameters, and generate a structured Behavior Tree composed of elementary robotic actions. The resulting behavior is executed on a torque-controlled Franka Emika Panda, enabling compliant, contact-rich interactions for parameter estimation. The acquired measurements are used to automatically construct a physics-aware simulation. Experimental results on the real manipulator demonstrate estimation of object mass, surface height, and friction-related quantities across multiple scenarios, including occluded objects and incomplete prior models. The proposed approach enables interpretable, intent-driven, and autonomously Real2Sim pipelines, bridging high-level reasoning with physically-grounded robotic interaction.
Abstract (translated)
构建一个真实世界环境的精确仿真模型，需要可靠地估算物理参数，如质量、几何形状、摩擦力和接触面等。传统的真实到模拟（Real-to-Simulation，简称Real2Sim）管道依赖于手动测量或固定预编程的探索程序，这限制了它们适应各种任务和用户意图的能力。本文提出了一种Real2Sim框架，该框架能够自主生成并执行行为树以进行特定任务所需的物理交互，并获取给定仿真目标所需的具体参数，而无需依赖预先定义的任务模板或专家设计的探索程序。 当接收到高级别用户的请求、不完整的模拟描述以及场景的RGB观察数据时，视觉语言模型会进行跨模态推理来识别相关对象，推断所需的物理参数，并生成一个由基本机器人动作组成的结构化行为树。由此产生的行为将被执行在具有扭矩控制功能的Franka Emika Panda机械臂上，以实现用于参数估计的顺应性和接触丰富的交互。 获取到的测量数据被用来自动构建一个基于物理学原理的仿真模型。实验结果表明，在真实操纵器上的多种场景下（包括遮挡物体和不完整的先前模型），本方法能够估算出物体的质量、表面高度以及与摩擦相关的量。该提出的方案使得实现解释性、意图驱动且自主运行的Real2Sim管道成为可能，从而在高层次推理和基于物理原理的机器人交互之间建立了桥梁。
URL
https://arxiv.org/abs/2601.08454
