Abstract
Egocentric "walking tour" videos provide a rich source of image data to develop rich and diverse visual models of environments around the world. However, the significant presence of humans in frames of these videos due to crowds and eye-level camera perspectives mitigates their usefulness in environment modeling applications. We focus on addressing this challenge by developing a generative algorithm that can realistically remove (i.e., inpaint) humans and their associated shadow effects from walking tour videos. Key to our approach is the construction of a rich semi-synthetic dataset of video clip pairs to train this generative model. Each pair in the dataset consists of an environment-only background clip, and a composite clip of walking humans with simulated shadows overlaid on the background. We randomly sourced both foreground and background components from real egocentric walking tour videos around the world to maintain visual diversity. We then used this dataset to fine-tune the state-of-the-art Casper video diffusion model for object and effects inpainting, and demonstrate that the resulting model performs far better than Casper both qualitatively and quantitatively at removing humans from walking tour clips with significant human presence and complex backgrounds. Finally, we show that the resulting generated clips can be used to build successful 3D/4D models of urban locations.
Abstract (translated)
以自我视角拍摄的"行走游览"视频为构建全球丰富多样的环境视觉模型提供了充足的图像数据。然而，由于人群聚集和摄像机处于人眼高度，这些视频画面中频繁出现的人物显著降低了其在环境建模应用中的效用。本研究聚焦于解决这一挑战，开发了一种能够逼真地从行走游览视频中移除人物及其伴随阴影效果的生成算法。该方法的关键在于构建了一个丰富的半合成视频片段配对数据集用于训练生成模型。数据集中的每对样本均包含一个仅含环境的背景片段，以及一个将行走人物与模拟阴影叠加在背景上的合成片段。我们随机从全球真实的第一人称行走游览视频中抽取前景和背景组件以保持视觉多样性。随后利用该数据集对最先进的Casper视频扩散模型进行微调，使其具备对象与效果重绘能力。实验表明，经微调的模型在移除高人物密度、复杂背景的行走游览片段时，其定性与定量表现均显著优于原始Casper模型。最后，我们证明所生成的视频片段可成功用于构建城市地点的三维/四维模型。
URL
https://arxiv.org/abs/2603.29036
