Abstract
We present IDC-Net (Image-Depth Consistency Network), a novel framework designed to generate RGB-D video sequences under explicit camera trajectory control. Unlike approaches that treat RGB and depth generation separately, IDC-Net jointly synthesizes both RGB images and corresponding depth maps within a unified geometry-aware diffusion model. The joint learning framework strengthens spatial and geometric alignment across frames, enabling more precise camera control in the generated sequences. To support the training of this camera-conditioned model and ensure high geometric fidelity, we construct a camera-image-depth consistent dataset with metric-aligned RGB videos, depth maps, and accurate camera poses, which provides precise geometric supervision with notably improved inter-frame geometric consistency. Moreover, we introduce a geometry-aware transformer block that enables fine-grained camera control, enhancing control over the generated sequences. Extensive experiments show that IDC-Net achieves improvements over state-of-the-art approaches in both visual quality and geometric consistency of generated scene sequences. Notably, the generated RGB-D sequences can be directly feed for downstream 3D Scene reconstruction tasks without extra post-processing steps, showcasing the practical benefits of our joint learning framework. See more at this https URL.
Abstract (translated)
我们提出了IDC-Net（图像深度一致性网络），这是一种新颖的框架，旨在根据明确的相机轨迹控制生成RGB-D视频序列。与将RGB和深度生成分开处理的方法不同，IDC-Net在一个统一的几何感知扩散模型中联合合成RGB图像及其对应的深度图，从而加强了帧间的空间和几何对齐，使得在生成序列中可以实现更精确的相机控制。 为了支持该条件化相机模型的训练，并确保高几何保真度，我们构建了一个包含度量一致的RGB视频、深度图及准确相机姿态的相机-图像-深度一致性数据集。该数据集提供了精准的几何监督，并且显著提高了帧间几何一致性。此外，我们还引入了一种具有几何感知能力的变压器模块，使细粒度的相机控制成为可能，从而增强对生成序列的控制。 大量的实验表明，在视觉质量和生成场景序列中的几何一致性的方面，IDC-Net相比最先进的方法实现了改进。值得注意的是，生成的RGB-D序列可以直接用于下游3D场景重建任务而无需额外的后处理步骤，展示了我们联合学习框架的实际益处。有关更多信息，请访问此链接：[https URL]（请注意这里的URL应替换为实际提供的链接）。
URL
https://arxiv.org/abs/2508.04147
