Abstract
Recent video editing methods achieve attractive results in style transfer or appearance modification. However, editing the structural content of 3D scenes in videos remains challenging, particularly when dealing with significant viewpoint changes, such as large camera rotations or zooms. Key challenges include generating novel view content that remains consistent with the original video, preserving unedited regions, and translating sparse 2D inputs into realistic 3D video outputs. To address these issues, we propose Sketch3DVE, a sketch-based 3D-aware video editing method to enable detailed local manipulation of videos with significant viewpoint changes. To solve the challenge posed by sparse inputs, we employ image editing methods to generate edited results for the first frame, which are then propagated to the remaining frames of the video. We utilize sketching as an interaction tool for precise geometry control, while other mask-based image editing methods are also supported. To handle viewpoint changes, we perform a detailed analysis and manipulation of the 3D information in the video. Specifically, we utilize a dense stereo method to estimate a point cloud and the camera parameters of the input video. We then propose a point cloud editing approach that uses depth maps to represent the 3D geometry of newly edited components, aligning them effectively with the original 3D scene. To seamlessly merge the newly edited content with the original video while preserving the features of unedited regions, we introduce a 3D-aware mask propagation strategy and employ a video diffusion model to produce realistic edited videos. Extensive experiments demonstrate the superiority of Sketch3DVE in video editing. Homepage and code: http://http://geometrylearning.com/Sketch3DVE/
Abstract (translated)
最近的视频编辑方法在风格转换或外观修改方面取得了引人注目的成果。然而，对具有显著视角变化（如大角度旋转或变焦）的三维场景结构内容进行编辑仍然存在挑战性。关键问题包括生成与原始视频一致的新视图内容、保持未编辑区域不变以及将稀疏的2D输入转换为真实的3D视频输出。为了应对这些问题，我们提出了Sketch3DVE，这是一种基于草图的3D感知视频编辑方法，旨在实现具有显著视角变化的视频的详细局部操作。 针对由稀疏输入带来的挑战，我们利用图像编辑技术生成第一个帧的编辑结果，并将其传播到视频中的其余帧中。我们使用绘图作为精确几何控制的交互工具，同时支持其他基于蒙版的图像编辑方法。为了处理视角的变化，我们在视频中对3D信息进行了详细的分析和操作。具体而言，我们采用了一种密集立体法来估计输入视频的点云及其摄像机参数。接着，我们提出了一种点云编辑的方法，利用深度图表示新编辑组件的3D几何结构，并将其有效地与原始场景对齐。 为了无缝融合新的编辑内容并保持未修改区域的特征，我们引入了基于3D感知的蒙版传播策略，并使用视频扩散模型生成逼真的编辑视频。大量实验表明Sketch3DVE在视频编辑方面的优越性。官方网站和代码：http://geometrylearning.com/Sketch3DVE/
URL
https://arxiv.org/abs/2508.13797
