Abstract
Object-level manipulation, relocating or reorienting objects in images or videos while preserving scene realism, is central to film post-production, AR, and creative editing. Yet existing methods struggle to jointly achieve three core goals: background preservation, geometric consistency under viewpoint shifts, and user-controllable transformations. Geometry-based approaches offer precise control but require explicit 3D reconstruction and generalize poorly; diffusion-based methods generalize better but lack fine-grained geometric control. We present Ctrl&Shift, an end-to-end diffusion framework to achieve geometry-consistent object manipulation without explicit 3D representations. Our key insight is to decompose manipulation into two stages, object removal and reference-guided inpainting under explicit camera pose control, and encode both within a unified diffusion process. To enable precise, disentangled control, we design a multi-task, multi-stage training strategy that separates background, identity, and pose signals across tasks. To improve generalization, we introduce a scalable real-world dataset construction pipeline that generates paired image and video samples with estimated relative camera poses. Extensive experiments demonstrate that Ctrl&Shift achieves state-of-the-art results in fidelity, viewpoint consistency, and controllability. To our knowledge, this is the first framework to unify fine-grained geometric control and real-world generalization for object manipulation, without relying on any explicit 3D modeling.
Abstract (translated)
对象级操作，即在图像或视频中重新定位或重新定向物体以保持场景真实感，是电影后期制作、增强现实（AR）和创意编辑的核心。然而，现有方法难以同时实现三个核心目标：背景保留、视角变化下的几何一致性以及用户可控的变换。基于几何的方法提供了精确控制但需要显式的三维重建且泛化能力差；扩散基础的方法具有更好的泛化能力，但在细粒度几何控制方面表现不佳。 我们提出了Ctrl&Shift框架，这是一个端到端的扩散框架，可以在不使用明确三维表示的情况下实现几何一致的对象操作。我们的关键见解是将对象处理分解为两个阶段：对象去除和基于参考引导的修复（inpainting），同时在显式的相机姿态控制下进行。我们将这两个过程编码在一个统一的扩散过程中。 为了实现精确、解耦的控制，我们设计了一个多任务、多阶段训练策略，在不同任务中分离背景信号、身份信号和姿势信号。为了提高泛化能力，我们引入了一种可扩展的真实世界数据集构建流水线，该流程生成具有估计相对相机姿态配对图像和视频样本。 广泛的实验表明，Ctrl&Shift在保真度、视角一致性以及可控性方面均达到了最先进的成果。据我们所知，这是第一个框架，在不依赖任何显式三维建模的情况下统一了细粒度几何控制与现实世界的泛化能力，实现了对象操作的突破。
URL
https://arxiv.org/abs/2602.11440
