Abstract
Omnimatte aims to decompose a given video into semantically meaningful layers, including the background and individual objects along with their associated effects, such as shadows and reflections. Existing methods often require extensive training or costly self-supervised optimization. In this paper, we present OmnimatteZero, a training-free approach that leverages off-the-shelf pre-trained video diffusion models for omnimatte. It can remove objects from videos, extract individual object layers along with their effects, and composite those objects onto new videos. We accomplish this by adapting zero-shot image inpainting techniques for video object removal, a task they fail to handle effectively out-of-the-box. We then show that self-attention maps capture information about the object and its footprints and use them to inpaint the object's effects, leaving a clean background. Additionally, through simple latent arithmetic, object layers can be isolated and recombined seamlessly with new video layers to produce new videos. Evaluations show that OmnimatteZero not only achieves superior performance in terms of background reconstruction but also sets a new record for the fastest Omnimatte approach, achieving real-time performance with minimal frame runtime.
Abstract (translated)
Omnimatte旨在将给定的视频分解为语义上有意义的图层，包括背景和各个对象及其相关效果（如阴影和反射）。现有的方法通常需要大量的训练或昂贵的自监督优化。在本文中，我们提出了一种无需训练的方法——OmnimatteZero，该方法利用现成的预训练视频扩散模型进行全场景材质分解。它可以移除视频中的物体，提取单独的对象图层及其相关效果，并将这些对象合并到新的视频上。通过将零样本图像修复技术适应于视频对象去除任务（这是它们在没有修改的情况下无法有效处理的任务），我们实现了这一目标。此外，我们展示了自注意力映射可以捕获有关对象及其足迹的信息，并利用这些信息来修复对象的效果，从而留下干净的背景。通过简单的潜在算术运算，还可以将物体图层隔离并无缝地与新的视频图层重新组合以生成新视频。评估结果表明，OmnimatteZero不仅在背景重建方面表现出色，而且还创下了最快全场景材质分解方法的新纪录，在实现近乎实时性能的同时保证了最小的帧处理时间。
URL
https://arxiv.org/abs/2503.18033
