Abstract
Egocentric interactive world models are essential for augmented reality and embodied AI, where visual generation must respond to user input with low latency, geometric consistency, and long-term stability. We study egocentric interaction generation from a single scene image under free-space hand gestures, aiming to synthesize photorealistic videos in which hands enter the scene, interact with objects, and induce plausible world dynamics under head motion. This setting introduces fundamental challenges, including distribution shift between free-space gestures and contact-heavy training data, ambiguity between hand motion and camera motion in monocular views, and the need for arbitrary-length video generation. We present Hand2World, a unified autoregressive framework that addresses these challenges through occlusion-invariant hand conditioning based on projected 3D hand meshes, allowing visibility and occlusion to be inferred from scene context rather than encoded in the control signal. To stabilize egocentric viewpoint changes, we inject explicit camera geometry via per-pixel Plücker-ray embeddings, disentangling camera motion from hand motion and preventing background drift. We further develop a fully automated monocular annotation pipeline and distill a bidirectional diffusion model into a causal generator, enabling arbitrary-length synthesis. Experiments on three egocentric interaction benchmarks show substantial improvements in perceptual quality and 3D consistency while supporting camera control and long-horizon interactive generation.
Abstract (translated)
自体中心互动世界模型对于增强现实和具身人工智能至关重要，在这些领域中，视觉生成必须能够快速响应用户输入，并且保持几何一致性和长期稳定性。本文研究了仅基于单一场景图像以及自由空间手势的情况下进行自体中心交互生成的方法，旨在合成这样的视频：在其中手进入场景、与物体互动并根据头部运动产生合理的世界动态变化。这种设置引入了许多基本挑战，包括自由空间手势和接触密集型训练数据之间的分布差异、单目视角下手部动作和相机移动的模糊性以及任意长度视频生成的需求。 我们提出了Hand2World框架，这是一个统一的自回归架构，通过基于投影3D手模型的手部条件处理解决了这些问题。这种处理方式使得可视性和遮挡可以从场景上下文中推断出来而不是直接编码到控制信号中。为了稳定自体视角的变化，我们在每像素Plücker射线嵌入的基础上注入了显式的相机几何信息，从而将相机移动与手部动作区分开来，并防止背景漂移现象的发生。 我们还开发了一条全自动单目注释流水线并从双向扩散模型中提取出了因果生成器，这使得任意长度的合成成为了可能。我们在三个自体中心交互基准上进行了实验，结果显示在感知质量、3D一致性以及支持相机控制和长期互动生成方面取得了显著改进。 简单来说，Hand2World框架通过处理手势与场景之间的关系，并结合先进的模型简化技术，能够生成更加真实且稳定的交互式视频内容，在增强现实等领域具有广泛的应用前景。
URL
https://arxiv.org/abs/2602.09600
