Abstract
Inpainting for real-world human and pedestrian removal in high-resolution video clips presents significant challenges, particularly in achieving high-quality outcomes, ensuring temporal consistency, and managing complex object interactions that involve humans, their belongings, and their shadows. In this paper, we introduce VIP (Video Inpainting Pipeline), a novel promptless video inpainting framework for real-world human removal applications. VIP enhances a state-of-the-art text-to-video model with a motion module and employs a Variational Autoencoder (VAE) for progressive denoising in the latent space. Additionally, we implement an efficient human-and-belongings segmentation for precise mask generation. Sufficient experimental results demonstrate that VIP achieves superior temporal consistency and visual fidelity across diverse real-world scenarios, surpassing state-of-the-art methods on challenging datasets. Our key contributions include the development of the VIP pipeline, a reference frame integration technique, and the Dual-Fusion Latent Segment Refinement method, all of which address the complexities of inpainting in long, high-resolution video sequences.
Abstract (translated)
在高分辨率视频片段中去除真实世界中的行人和人类，以实现高质量的结果、确保时间一致性以及处理涉及人类及其随身物品和阴影的复杂对象交互，面临着重大挑战。本文介绍了VIP（Video Inpainting Pipeline），这是一个新颖的无提示视频修复框架，专门用于现实世界的行人移除应用。VIP通过引入一个运动模块增强了最先进的文本到视频模型，并利用变分自编码器（VAE）在潜在空间中进行渐进式去噪处理。此外，我们还实施了一个高效的分割方法来生成精确的人及其随身物品的掩码。充分的实验结果表明，VIP在各种现实世界场景中实现了卓越的时间一致性和视觉保真度，在具有挑战性的数据集上超越了现有最先进的方法。我们的主要贡献包括开发VIP管道、参考帧集成技术以及双融合潜在段细化方法，这些都解决了长时间高分辨率视频序列中的修复复杂性问题。
URL
https://arxiv.org/abs/2504.03041
