Abstract
Many existing video inpainting algorithms utilize optical flows to construct the corresponding maps and then propagate pixels from adjacent frames to missing areas by mapping. Despite the effectiveness of the propagation mechanism, they might encounter blurry and inconsistencies when dealing with inaccurate optical flows or large masks. Recently, Diffusion Transformer (DiT) has emerged as a revolutionary technique for video generation tasks. However, pretrained DiT models for video generation all contain a large amount of parameters, which makes it very time consuming to apply to video inpainting tasks. In this paper, we present DiTPainter, an end-to-end video inpainting model based on Diffusion Transformer (DiT). DiTPainter uses an efficient transformer network designed for video inpainting, which is trained from scratch instead of initializing from any large pretrained models. DiTPainter can address videos with arbitrary lengths and can be applied to video decaptioning and video completion tasks with an acceptable time cost. Experiments show that DiTPainter outperforms existing video inpainting algorithms with higher quality and better spatial-temporal consistency.
Abstract (translated)
许多现有的视频修复算法利用光流来构建相应的映射，然后通过映射将相邻帧的像素传播到缺失区域。虽然这种传播机制非常有效，但在处理不准确的光流或大面积遮罩时可能会导致模糊和不一致的问题。最近，扩散变换器（Diffusion Transformer, DiT）作为视频生成任务中的革命性技术崭露头角。然而，用于视频生成的预训练DiT模型参数量巨大，这使得它们在应用于视频修复任务时非常耗时。 在这篇论文中，我们介绍了DiTPainter，这是一个基于扩散变换器（Diffusion Transformer, DiT）的端到端视频修复模型。DiTPainter使用了一种专门设计用于视频修复的有效Transformer网络，并且是从头开始训练的，而不是从任何大型预训练模型初始化的。DiTPainter可以处理任意长度的视频，并能以可接受的时间成本应用于视频去遮挡和视频完成任务。实验表明，DiTPainter在质量和时空一致性方面优于现有的视频修复算法。
URL
https://arxiv.org/abs/2504.15661
