Abstract
Egocentric vision is essential for both human and machine visual understanding, particularly in capturing the detailed hand-object interactions needed for manipulation tasks. Translating third-person views into first-person views significantly benefits augmented reality (AR), virtual reality (VR) and robotics applications. However, current exocentric-to-egocentric translation methods are limited by their dependence on 2D cues, synchronized multi-view settings, and unrealistic assumptions such as necessity of initial egocentric frame and relative camera poses during inference. To overcome these challenges, we introduce EgoWorld, a novel two-stage framework that reconstructs an egocentric view from rich exocentric observations, including projected point clouds, 3D hand poses, and textual descriptions. Our approach reconstructs a point cloud from estimated exocentric depth maps, reprojects it into the egocentric perspective, and then applies diffusion-based inpainting to produce dense, semantically coherent egocentric images. Evaluated on the H2O and TACO datasets, EgoWorld achieves state-of-the-art performance and demonstrates robust generalization to new objects, actions, scenes, and subjects. Moreover, EgoWorld shows promising results even on unlabeled real-world examples.
Abstract (translated)
自视视觉对于人类和机器的视觉理解都至关重要，尤其是在捕捉操作任务所需的详细手部与物体互动方面。将第三人称视角转换为第一人称视角可以显著提升增强现实（AR）、虚拟现实（VR）以及机器人技术的应用效果。然而，当前从外向内（exocentric到egocentric）视图转化的方法受到依赖二维线索、同步多视角设置以及诸如需要初始自视帧和推断过程中的相对相机姿态等不切实际假设的限制。 为了解决这些挑战，我们引入了EgoWorld，这是一种新颖的两阶段框架，可以从丰富的外向观察（包括投影点云、三维手部姿势和文本描述）中重构出第一人称视角。我们的方法首先根据估计的第三人称深度图重建点云，然后将其重新投影到自视视角，并通过扩散基础插值生成稠密且语义一致的第一人称图像。 在H2O和TACO数据集上的评估显示，EgoWorld达到了最先进的性能水平，并展示了对新对象、动作、场景及主体的稳健泛化能力。此外，即使在未标记的真实世界示例上，EgoWorld也表现出令人鼓舞的结果。
URL
https://arxiv.org/abs/2506.17896
