Abstract
Recent advances in camera-controlled video diffusion models have significantly improved video-camera alignment. However, the camera controllability still remains limited. In this work, we build upon Reward Feedback Learning and aim to further improve camera controllability. However, directly borrowing existing ReFL approaches faces several challenges. First, current reward models lack the capacity to assess video-camera alignment. Second, decoding latent into RGB videos for reward computation introduces substantial computational overhead. Third, 3D geometric information is typically neglected during video decoding. To address these limitations, we introduce an efficient camera-aware 3D decoder that decodes video latent into 3D representations for reward quantization. Specifically, video latent along with the camera pose are decoded into 3D Gaussians. In this process, the camera pose not only acts as input, but also serves as a projection parameter. Misalignment between the video latent and camera pose will cause geometric distortions in the 3D structure, resulting in blurry renderings. Based on this property, we explicitly optimize pixel-level consistency between the rendered novel views and ground-truth ones as reward. To accommodate the stochastic nature, we further introduce a visibility term that selectively supervises only deterministic regions derived via geometric warping. Extensive experiments conducted on RealEstate10K and WorldScore benchmarks demonstrate the effectiveness of our proposed method. Project page: \href{this https URL}{CamPilot Page}.
Abstract (translated)
近期，在相机控制视频扩散模型方面的进展显著提高了视频与摄像机之间的对齐精度。然而，摄像机的可控性仍然有限。在这项工作中，我们基于奖励反馈学习（Reward Feedback Learning）方法，并致力于进一步提升摄像机的可控性。不过，直接借用现有的ReFL方法会遇到几个挑战：首先，当前的奖励模型缺乏评估视频与摄像机对齐能力的能力；其次，在计算奖励时将潜在变量解码为RGB视频带来了大量的计算开销；第三，在视频解码过程中通常忽略了3D几何信息。 为了应对这些局限性，我们引入了一个高效的感知相机的3D解码器，该解码器能够将视频潜变量解码成用于奖励量化的3D表示。具体而言，视频潜在编码与摄像机姿态一起被解码为3D高斯分布，在这一过程中，摄像机姿态不仅作为输入，还充当投影参数的角色。如果视频潜在变量和摄像机姿态之间存在对齐误差，则会导致3D结构的几何失真，并进而导致渲染模糊。 基于该特性，我们明确地优化了合成视角与真实视图之间的像素级一致性作为奖励计算的基础。考虑到这一随机性质，我们进一步引入了一个可见性项，仅针对通过几何变形导出的确定区域进行监督。在RealEstate10K和WorldScore基准上的广泛实验验证了所提出方法的有效性。 项目页面：\[链接\]（请将“this https URL”替换为实际链接）。
URL
https://arxiv.org/abs/2601.16214
