Abstract
The scene perception, understanding, and simulation are fundamental techniques for embodied-AI agents, while existing solutions are still prone to segmentation deficiency, dynamic objects' interference, sensor data sparsity, and view-limitation problems. This paper proposes a novel framework, named SPORTS, for holistic scene understanding via tightly integrating Video Panoptic Segmentation (VPS), Visual Odometry (VO), and Scene Rendering (SR) tasks into an iterative and unified perspective. Firstly, VPS designs an adaptive attention-based geometric fusion mechanism to align cross-frame features via enrolling the pose, depth, and optical flow modality, which automatically adjust feature maps for different decoding stages. And a post-matching strategy is integrated to improve identities tracking. In VO, panoptic segmentation results from VPS are combined with the optical flow map to improve the confidence estimation of dynamic objects, which enhances the accuracy of the camera pose estimation and completeness of the depth map generation via the learning-based paradigm. Furthermore, the point-based rendering of SR is beneficial from VO, transforming sparse point clouds into neural fields to synthesize high-fidelity RGB views and twin panoptic views. Extensive experiments on three public datasets demonstrate that our attention-based feature fusion outperforms most existing state-of-the-art methods on the odometry, tracking, segmentation, and novel view synthesis tasks.
Abstract (translated)
场景感知、理解和模拟是具身人工智能代理的基本技术，而现有解决方案仍然容易出现分割不足、动态物体干扰、传感器数据稀疏和视点限制等问题。本文提出了一种新颖的框架——SPORTS（通过紧密整合视频全景分割(VPS)、视觉测距(VO) 和场景渲染(SR) 任务，在迭代和统一的角度下进行整体场景理解）。首先，VPS 设计了一个基于自适应注意力的几何融合机制，该机制通过引入姿态、深度和光流模态来对齐跨帧特征，并自动调整不同解码阶段的特征图。此外，还集成了一个后匹配策略以改进身份追踪。 在VO任务中，将从VPS得到的全景分割结果与光流图结合使用，提高了动态物体置信度估计，增强了基于学习范式的相机姿态估计精度和深度图生成的完整性。另外，SR中的点云渲染得益于VO模块，通过转换稀疏点云为神经场来合成高保真RGB视图和孪生全景视图。 在三个公开数据集上的广泛实验表明，我们提出的注意力特征融合方法在测距、追踪、分割以及新颖视图合成任务上均优于大多数现有的最先进的方法。
URL
https://arxiv.org/abs/2510.12749
