Abstract
Video object removal has achieved advanced performance due to the recent success of video generative models. However, when addressing the side effects of objects, e.g., their shadows and reflections, existing works struggle to eliminate these effects for the scarcity of paired video data as supervision. This paper presents ROSE, termed Remove Objects with Side Effects, a framework that systematically studies the object's effects on environment, which can be categorized into five common cases: shadows, reflections, light, translucency and mirror. Given the challenges of curating paired videos exhibiting the aforementioned effects, we leverage a 3D rendering engine for synthetic data generation. We carefully construct a fully-automatic pipeline for data preparation, which simulates a large-scale paired dataset with diverse scenes, objects, shooting angles, and camera trajectories. ROSE is implemented as an video inpainting model built on diffusion transformer. To localize all object-correlated areas, the entire video is fed into the model for reference-based erasing. Moreover, additional supervision is introduced to explicitly predict the areas affected by side effects, which can be revealed through the differential mask between the paired videos. To fully investigate the model performance on various side effect removal, we presents a new benchmark, dubbed ROSE-Bench, incorporating both common scenarios and the five special side effects for comprehensive evaluation. Experimental results demonstrate that ROSE achieves superior performance compared to existing video object erasing models and generalizes well to real-world video scenarios. The project page is this https URL.
Abstract (translated)
视频对象移除技术由于最近视频生成模型的成功而取得了先进的性能。然而，当处理物体的副作用（如阴影和反射）时，现有方法因缺乏成对视频数据作为监督信号而难以消除这些效果。本文介绍了ROSE（Remove Objects with Side Effects），这是一个旨在系统研究环境中物体会影响的框架，并将这些影响分为五种常见情况：阴影、反射、光线变化、透明度以及镜子中的映像。 鉴于收集展示上述效应的成对视频面临的挑战，我们利用3D渲染引擎生成合成数据。我们精心构建了一个全自动的数据准备管道，该管道模拟了涵盖各种场景、物体、拍摄角度和摄像机轨迹的大规模成对数据集。ROSE作为一个基于扩散变换器的视频修复模型来实现，在这种框架下，整个视频被输入到模型中进行参考基线擦除操作，以便定位所有与对象相关的区域。此外，引入了额外的监督机制以明确预测受副作用影响的区域，这些区域可以通过成对视频之间的差异掩码揭示出来。 为了全面评估模型在各种副作用移除方面的性能，我们提出了一项新的基准测试，名为ROSE-Bench，该基准集成了常见场景以及五种特殊副作用进行综合评价。实验结果表明，ROSE与现有的视频对象擦除模型相比，在效果上更胜一筹，并且能很好地应用于现实世界的视频情况中。 项目页面在此链接：https://这个URL应该被替换为实际的网址或者具体描述，因为你提供的信息没有包含具体的网址或项目的公开访问方式。
URL
https://arxiv.org/abs/2508.18633
