Abstract
Existing video omnimatte methods typically rely on slow, multi-stage, or inference-time optimization pipelines that fail to fully exploit powerful generative priors, producing suboptimal decompositions. Our key insight is that, if a video inpainting model can be finetuned to remove the foreground-associated effects, then it must be inherently capable of perceiving these effects, and hence can also be finetuned for the complementary task: foreground layer decomposition with associated effects. However, although naïvely finetuning the inpainting model with LoRA applied to all blocks can produce high-quality alpha mattes, it fails to capture associated effects. Our systematic analysis reveals this arises because effect-related cues are primarily encoded in specific DiT blocks and become suppressed when LoRA is applied across all blocks. To address this, we introduce EasyOmnimatte, the first unified, end-to-end video omnimatte method. Concretely, we finetune a pretrained video inpainting diffusion model to learn dual complementary experts while keeping its original weights intact: an Effect Expert, where LoRA is applied only to effect-sensitive DiT blocks to capture the coarse structure of the foreground and associated effects, and a fully LoRA-finetuned Quality Expert learns to refine the alpha matte. During sampling, Effect Expert is used for denoising at early, high-noise steps, while Quality Expert takes over at later, low-noise steps. This design eliminates the need for two full diffusion passes, significantly reducing computational cost without compromising output quality. Ablation studies validate the effectiveness of this Dual-Expert strategy. Experiments demonstrate that EasyOmnimatte sets a new state-of-the-art for video omnimatte and enables various downstream tasks, significantly outperforming baselines in both quality and efficiency.
Abstract (translated)
现有的视频全遮罩（omnimatte）方法通常依赖于慢速、多阶段或推理时间优化的管道，这些方法未能充分利用强大的生成先验知识，导致分解效果不佳。我们的关键洞察是：如果一个视频修复模型可以通过微调来移除与前景相关的效应，则该模型必然具有感知这些效应的能力，并且也可以被微调用于互补任务——即带有相关效应的前景层分解。 然而，尽管直接使用LoRA（低秩适应）对所有块进行微调可以生成高质量的alpha遮罩，但它无法捕捉到相关的效果。我们系统性的分析发现这主要是因为与效果相关的线索主要编码在特定的DiT（差异变换）模块中，在整个模型的所有块上应用LoRA时会抑制这些效应。 为了解决这个问题，我们引入了EasyOmnimatte——首个统一、端到端的视频全遮罩方法。具体而言，我们微调一个预训练的视频修复扩散模型以学习两个互补专家，同时保持其原始权重不变：一个是效果专家（Effect Expert），仅在敏感于效应的DiT模块上应用LoRA来捕捉前景及其相关效应的粗略结构；另一个是完全使用LoRA进行微调的质量专家（Quality Expert），用于精细化alpha遮罩。在采样过程中，效果专家会在早期、高噪声步骤中去噪，而在后期、低噪声阶段则由质量专家接管。 这种设计消除了需要两次完整的扩散过程的必要性，大幅降低了计算成本而不牺牲输出质量。消融研究验证了双专家策略的有效性。实验表明，EasyOmnimatte在视频全遮罩任务上达到了新的最先进水平，并且支持多种下游任务，在质量和效率方面均显著优于基线方法。
URL
https://arxiv.org/abs/2512.21865
