Abstract
Semantic and panoptic occupancy prediction for road scene analysis provides a dense 3D representation of the ego vehicle's surroundings. Current camera-only approaches typically rely on costly dense 3D supervision or require training models on data from the target domain, limiting deployment in unseen environments. We propose FreeOcc, a training-free pipeline that leverages pretrained foundation models to recover both semantics and geometry from multi-view images. FreeOcc extracts per-view panoptic priors with a promptable foundation segmentation model and prompt-to-taxonomy rules, and reconstructs metric 3D points with a reconstruction foundation model. Depth- and confidence- aware filtering lifts reliable labels into 3D, which are fused over time and voxelized with a deterministic refinement stack. For panoptic occupancy, instances are recovered by fitting and merging robust current-view 3D box candidates, enabling instance-aware occupancy without any learned 3D model. On Occ3D-nuScenes, FreeOcc achieves 16.9 mIoU and 16.5 RayIoU train-free, on par with state-of-the-art weakly supervised methods. When employed as a pseudo-label generation pipeline for training downstream models, it achieves 21.1 RayIoU, surpassing the previous state-of-the-art weakly supervised baseline. Furthermore, FreeOcc sets new baselines for both train-free and weakly supervised panoptic occupancy prediction, achieving 3.1 RayPQ and 3.9 RayPQ, respectively. These results highlight foundation-model-driven perception as a practical route to training-free 3D scene understanding.
Abstract (translated)
语义和全景占用预测对于道路场景分析提供了一个密集的3D环境表示，特别是针对自动驾驶车辆周围的环境。目前仅使用相机的方法通常依赖于昂贵的密集3D监督或需要在目标领域数据上训练模型，这限制了其在未知环境中部署的能力。我们提出了FreeOcc，这是一种无需训练的流水线方法，该方法利用预训练的基础模型从多视角图像中恢复语义和几何信息。FreeOcc通过使用可提示的基础分割模型和规则到分类法的映射来提取每视图的全景先验，并且通过重建基础模型来重构度量3D点。深度感知和置信度感知过滤将可靠的标签提升至三维空间，随后这些标签被时间融合并用确定性细化堆栈进行体素化处理。对于全景占用预测，FreeOcc通过拟合和合并稳健的当前视图3D边界框候选来恢复实例信息，从而在不依赖任何学习到的3D模型的情况下实现基于实例的占用预测。 在Occ3D-nuScenes数据集上，FreeOcc实现了16.9 mIoU（平均交并比）和16.5 RayIoU（光线交并比），这与现有的弱监督方法的表现相当。当作为下游模型训练中的伪标签生成流程使用时，它可以达到21.1 RayIoU的性能水平，超过了之前最先进的弱监督基线。此外，FreeOcc为无训练和弱监督全景占用预测分别设定了新的基准，分别是3.1 RayPQ（光线交并质量）和3.9 RayPQ。 这些结果突显了基于基础模型驱动感知作为实现无需训练的3D场景理解的一种实用途径的重要性。
URL
https://arxiv.org/abs/2603.06166
