Abstract
Existing image foundation models are not optimized for spherical images having been trained primarily on perspective images. PanoSAMic integrates the pre-trained Segment Anything (SAM) encoder to make use of its extensive training and integrate it into a semantic segmentation model for panoramic images using multiple modalities. We modify the SAM encoder to output multi-stage features and introduce a novel spatio-modal fusion module that allows the model to select the relevant modalities and best features from each modality for different areas of the input. Furthermore, our semantic decoder uses spherical attention and dual view fusion to overcome the distortions and edge discontinuity often associated with panoramic images. PanoSAMic achieves state-of-the-art (SotA) results on Stanford2D3DS for RGB, RGB-D, and RGB-D-N modalities and on Matterport3D for RGB and RGB-D modalities. this https URL
Abstract (translated)
现有的图像基础模型主要是基于透视图的图像进行训练，并未针对球形图像进行优化。PanoSAMic 将预先训练好的 Segment Anything (SAM) 编码器集成到语义分割全景图像的多模态模型中，以利用其广泛的预训练成果。我们对 SAM 编码器进行了修改，使其能够输出多阶段特征，并引入了一个新颖的空间-模式融合模块，使模型能够在不同输入区域选择相关模式和每个模式的最佳特征。 此外，我们的语义解码器使用球形注意力机制以及双视图融合来克服全景图像常有的扭曲和边缘不连续问题。PanoSAMic 在 Stanford2D3DS 数据集上针对 RGB、RGB-D 和 RGB-D-N 模态以及在 Matterport3D 数据集上的 RGB 和 RGB-D 模态实现了当前最佳（SotA）的结果。 原文链接：[这里插入 URL]
URL
https://arxiv.org/abs/2601.07447
