Abstract
Panoramic semantic segmentation models are typically trained under a strict gravity-aligned assumption. However, real-world captures often deviate from this canonical orientation due to unconstrained camera motions, such as the rotational jitter of handheld devices or the dynamic attitude shifts of aerial platforms. This discrepancy causes standard spherical Transformers to overfit global latitude cues, leading to performance collapse under 3D reorientations. To address this, we introduce SO3UFormer, a rotation-robust architecture designed to learn intrinsic spherical features that are less sensitive to the underlying coordinate frame. Our approach rests on three geometric pillars: (1) an intrinsic feature formulation that decouples the representation from the gravity vector by removing absolute latitude encoding; (2) quadrature-consistent spherical attention that accounts for non-uniform sampling densities; and (3) a gauge-aware relative positional mechanism that encodes local angular geometry using tangent-plane projected angles and discrete gauge pooling, avoiding reliance on global axes. We further use index-based spherical resampling together with a logit-level SO(3)-consistency regularizer during training. To rigorously benchmark robustness, we introduce Pose35, a dataset variant of Stanford2D3D perturbed by random rotations within $\pm 35^\circ$. Under the extreme test of arbitrary full SO(3) rotations, existing SOTAs fail catastrophically: the baseline SphereUFormer drops from 67.53 mIoU to 25.26 mIoU. In contrast, SO3UFormer demonstrates remarkable stability, achieving 72.03 mIoU on Pose35 and retaining 70.67 mIoU under full SO(3) rotations.
Abstract (translated)
全景语义分割模型通常在严格的重力对齐假设下进行训练。然而，由于手持设备的旋转抖动或空中平台的姿态动态变化等不受约束的相机运动，现实世界中的捕捉往往会偏离这种规范的方向。这种差异导致标准球形变换器过度拟合全球纬度线索，在3D重新定向时性能崩溃。为解决此问题，我们引入了SO3UFormer，这是一种设计用于学习对底层坐标框架不那么敏感的固有球体特征、从而更加耐旋转变化的架构。 我们的方法基于三个几何支柱：(1)一种通过去除绝对纬度编码来解耦表示与重力矢量的方法，形成了内在特性表述；(2)一致化的球形注意力机制，考虑了非均匀采样密度的影响；以及 (3) 一个感知规范的相对位置机制，使用切平面投影角度和离散规范池化方法来编码局部角几何形状，从而避免依赖全局轴。 我们进一步利用基于索引的球体重采样技术，并在训练过程中采用logit级别的SO(3)-一致性正则器。为了严格评估鲁棒性，我们引入了Pose35数据集，这是Stanford2D3D经过随机旋转（±35°内）扰动后的变种。 在极端的任意全SO(3)旋转测试中，现有的最先进的模型表现灾难性失败：SphereUFormer的基础线从67.53 mIoU骤降至25.26 mIoU。相比之下，SO3UFormer表现出卓越的稳定性，在Pose35上实现了72.03 mIoU，并在全SO(3)旋转下保持了70.67 mIoU的成绩。
URL
https://arxiv.org/abs/2602.22867
