Abstract
Learning human-object manipulation presents significant challenges due to its fine-grained and contact-rich nature of the motions involved. Traditional physics-based animation requires extensive modeling and manual setup, and more importantly, it neither generalizes well across diverse object morphologies nor scales effectively to real-world environment. To address these limitations, we introduce LOME, an egocentric world model that can generate realistic human-object interactions as videos conditioned on an input image, a text prompt, and per-frame human actions, including both body poses and hand gestures. LOME injects strong and precise action guidance into object manipulation by jointly estimating spatial human actions and the environment contexts during training. After finetuning a pretrained video generative model on videos of diverse egocentric human-object interactions, LOME demonstrates not only high action-following accuracy and strong generalization to unseen scenarios, but also realistic physical consequences of hand-object interactions, e.g., liquid flowing from a bottle into a mug after executing a ``pouring'' action. Extensive experiments demonstrate that our video-based framework significantly outperforms state-of-the-art image based and video-based action-conditioned methods and Image/Text-to-Video (I/T2V) generative model in terms of both temporal consistency and motion control. LOME paves the way for photorealistic AR/VR experiences and scalable robotic training, without being limited to simulated environments or relying on explicit 3D/4D modeling.
Abstract (translated)
学习人-物操纵因其动作的精细性和高接触性而面临重大挑战。传统基于物理的动画需要大量建模和手动设置，更重要的是，它既不能很好地泛化到不同形态的物体，也无法有效扩展到真实环境。为应对这些局限，我们提出了LOME——一个以自我为中心的世界模型，能够根据输入图像、文本提示以及每帧的人体动作（包括身体姿态和手势）生成逼真的人-物交互视频。LOME通过在训练中联合估计人体空间动作和环境上下文，将强大而精确的动作引导注入物体操纵中。在对预训练视频生成模型进行多样化第一人称人-物交互视频微调后，LOME不仅展现出高动作跟随精度和对未知场景的强泛化能力，还能呈现手-物交互的真实物理后果，例如执行“倾倒”动作后液体从瓶中流入杯子的效果。大量实验表明，我们的视频框架在时间一致性和运动控制方面，均显著优于最先进的图像/视频动作条件化方法以及图像/文本到视频（I/T2V）生成模型。LOME为照片般真实的AR/VR体验和可扩展的机器人训练铺平了道路，且不受模拟环境限制或依赖显式3D/4D建模。
URL
https://arxiv.org/abs/2603.27449
