Abstract
Panoramic distortion poses a significant challenge in 360 depth estimation, particularly pronounced at the north and south poles. Existing methods either adopt a bi-projection fusion strategy to remove distortions or model long-range dependencies to capture global structures, which can result in either unclear structure or insufficient local perception. In this paper, we propose a spherical geometry transformer, named SGFormer, to address the above issues, with an innovative step to integrate spherical geometric priors into vision transformers. To this end, we retarget the transformer decoder to a spherical prior decoder (termed SPDecoder), which endeavors to uphold the integrity of spherical structures during decoding. Concretely, we leverage bipolar re-projection, circular rotation, and curve local embedding to preserve the spherical characteristics of equidistortion, continuity, and surface distance, respectively. Furthermore, we present a query-based global conditional position embedding to compensate for spatial structure at varying resolutions. It not only boosts the global perception of spatial position but also sharpens the depth structure across different patches. Finally, we conduct extensive experiments on popular benchmarks, demonstrating our superiority over state-of-the-art solutions.
Abstract (translated)
全景失真在360度深度估计中是一个重大的挑战，特别是在北极和南极。现有的方法要么采用双投影融合策略来消除失真，要么通过建模长距离依赖来捕捉全局结构，从而可能导致清晰的结构或不足的局部感知。在本文中，我们提出了一种球形几何变换器，名为SGFormer，来解决上述问题，并在视觉变换器中创新地将球形几何先验集成到其中。为此，我们将变换器解码器重定向为球形先验解码器（称为SPDecoder），它努力保持解码过程中的球形结构的完整性。具体来说，我们利用极性投影、圆周旋转和曲线局部嵌入来保留等距离失真、连续性和表面距离的球形特征。此外，我们还提出了一个基于查询的全局条件位置嵌入，以补偿不同分辨率下的空间结构。它不仅提高了全局空间位置的感知，还 sharpens the depth structure across different patches。最后，我们在多个流行基准上进行了广泛的实验，证明了我们在现有解决方案中的优越性。
URL
https://arxiv.org/abs/2404.14979
