Abstract
Recent vision foundation models (VFMs), typically based on Vision Transformer (ViT), have significantly advanced numerous computer vision tasks. Despite their success in tasks focused solely on RGB images, the potential of VFMs in RGB-depth driving scene parsing remains largely under-explored. In this article, we take one step toward this emerging research area by investigating a feasible technique to fully exploit VFMs for generalizable RGB-depth driving scene parsing. Specifically, we explore the inherent characteristics of RGB and depth data, thereby presenting a Heterogeneous Feature Integration Transformer (HFIT). This network enables the efficient extraction and integration of comprehensive heterogeneous features without re-training ViTs. Relative depth prediction results from VFMs, used as inputs to the HFIT side adapter, overcome the limitations of the dependence on depth maps. Our proposed HFIT demonstrates superior performance compared to all other traditional single-modal and data-fusion scene parsing networks, pre-trained VFMs, and ViT adapters on the Cityscapes and KITTI Semantics datasets. We believe this novel strategy paves the way for future innovations in VFM-based data-fusion techniques for driving scene parsing. Our source code is publicly available at this https URL.
Abstract (translated)
最近的视觉基础模型（VFMs），通常基于Vision Transformer (ViT)，已经在众多计算机视觉任务中取得了显著的进步。尽管它们在仅关注RGB图像的任务上表现出色，但在RGB-深度驾驶场景解析中的潜力却尚未被充分探索。本文朝着这一新兴研究领域迈出了一步，通过调查一种可行的技术来充分利用VFMs进行通用的RGB-深度驾驶场景解析。具体而言，我们探讨了RGB和深度数据的基本特性，并提出了异构特征集成变换器（HFIT）。该网络能够高效地提取并整合全面的异构特征，而无需重新训练ViT模型。来自VFMs的相关深度预测结果被用作HFIT侧适配器的输入，从而克服了对深度图依赖的限制。我们提出的HFIT在Cityscapes和KITTI Semantics数据集上与所有传统的单模态和数据融合场景解析网络、预训练的VFMs以及ViT适配器相比，表现出了更优越的性能。我们认为这一新颖策略为基于VFM的数据融合技术在未来驾驶场景解析中的创新铺平了道路。我们的源代码公开可获取于[提供的URL]。
URL
https://arxiv.org/abs/2502.06219
