Abstract
Accurately analyzing the motion parts and their motion attributes in dynamic environments is crucial for advancing key areas such as embodied intelligence. Addressing the limitations of existing methods that rely on dense multi-view images or detailed part-level annotations, we propose an innovative framework that can analyze 3D mobility from monocular videos in a zero-shot manner. This framework can precisely parse motion parts and motion attributes only using a monocular video, completely eliminating the need for annotated training data. Specifically, our method first constructs the scene geometry and roughly analyzes the motion parts and their initial motion attributes combining depth estimation, optical flow analysis and point cloud registration method, then employs 2D Gaussian splatting for scene representation. Building on this, we introduce an end-to-end dynamic scene optimization algorithm specifically designed for articulated objects, refining the initial analysis results to ensure the system can handle 'rotation', 'translation', and even complex movements ('rotation+translation'), demonstrating high flexibility and versatility. To validate the robustness and wide applicability of our method, we created a comprehensive dataset comprising both simulated and real-world scenarios. Experimental results show that our framework can effectively analyze articulated object motions in an annotation-free manner, showcasing its significant potential in future embodied intelligence applications.
Abstract (translated)
在动态环境中精确分析运动部件及其运动属性对于推进具身智能等关键领域至关重要。为了解决现有方法依赖密集多视角图像或详细部件级标注的限制，我们提出了一种创新框架，可以从单目视频中零样本地分析3D运动性。该框架仅使用单目视频就能精准解析运动部件和运动属性，并且完全不需要注释训练数据。 具体来说，我们的方法首先通过深度估计、光流分析和点云配准技术构建场景几何结构并粗略地分析运动部件及其初始运动属性，然后采用二维高斯喷涂法进行场景表示。在此基础上，我们引入了一种专为刚体对象设计的端到端动态场景优化算法，对初步分析结果进行细化处理，以确保系统能够应对“旋转”、“平移”，甚至复杂的组合动作（如“旋转+平移”），展示了极高的灵活性和实用性。 为了验证方法的鲁棒性和广泛适用性，我们创建了一个包含仿真和现实世界场景的综合数据集。实验结果显示，我们的框架能够在无需标注的情况下有效分析刚体对象的动作，展现出在未来具身智能应用中的巨大潜力。
URL
https://arxiv.org/abs/2505.11868
