Abstract
The landscape of AI video generation is undergoing a pivotal shift: moving beyond general generation - which relies on exhaustive prompt-engineering and "cherry-picking" - towards fine-grained, controllable generation and high-fidelity post-processing. In professional AI-assisted filmmaking, it is crucial to perform precise, targeted modifications. A cornerstone of this transition is video instance insertion, which requires inserting a specific instance into existing footage while maintaining scene integrity. Unlike traditional video editing, this task demands several requirements: precise spatial-temporal placement, physically consistent scene interaction, and the faithful preservation of original dynamics - all achieved under minimal user effort. In this paper, we propose PISCO, a video diffusion model for precise video instance insertion with arbitrary sparse keyframe control. PISCO allows users to specify a single keyframe, start-and-end keyframes, or sparse keyframes at arbitrary timestamps, and automatically propagates object appearance, motion, and interaction. To address the severe distribution shift induced by sparse conditioning in pretrained video diffusion models, we introduce Variable-Information Guidance for robust conditioning and Distribution-Preserving Temporal Masking to stabilize temporal generation, together with geometry-aware conditioning for realistic scene adaptation. We further construct PISCO-Bench, a benchmark with verified instance annotations and paired clean background videos, and evaluate performance using both reference-based and reference-free perceptual metrics. Experiments demonstrate that PISCO consistently outperforms strong inpainting and video editing baselines under sparse control, and exhibits clear, monotonic performance improvements as additional control signals are provided. Project page: this http URL.
Abstract (translated)
人工智能视频生成领域的格局正经历着关键的转变：从依赖详尽提示工程和“挑选”的通用生成，转向精细、可控的生成以及高保真的后期处理。在专业的人工智能辅助电影制作中，进行精确、有针对性的修改至关重要。这一转型的核心是视频实例插入技术，该技术要求将特定实例精确定位到现有镜头中，并保持场景完整性。与传统的视频编辑不同，这项任务需要满足多种需求：包括精确的时间和空间定位、物理上一致的场景交互以及真实动态的忠实保存——所有这些都应在用户最小化努力的情况下实现。 本文提出了一种名为PISCO（Precise Instance Insertion with Sparse Keyframe Control）的视频扩散模型，用于在任意稀疏关键帧控制下进行准确的视频实例插入。PISCO允许用户指定单一关键帧、开始和结束的关键帧或任意时间戳上的稀疏关键帧，并自动传播对象外观、运动及交互情况。 为了应对预训练视频扩散模型中由稀疏条件引起的严重分布偏移问题，我们引入了变量信息引导以实现稳健的条件设置以及保持分布一致的时间掩码技术来稳定时序生成。此外还结合几何感知条件以实现实景场景适应性。 为评估性能，我们构建了一个基准测试PISCO-Bench，该基准包含有验证过的实例注释和配对干净背景视频，并使用参考依赖性和无参考的感知度量进行评价。实验结果表明，在稀疏控制下，PISCO始终超越强大的图像修复及视频编辑基线模型，随着提供更多的控制信号，其性能表现出明显的单调改善趋势。 项目页面：[此HTTP URL](http://this%20URL)（请将“this http URL”替换为实际的链接）。
URL
https://arxiv.org/abs/2602.08277
