Abstract
Standard Vision Transformers flatten 2D images into 1D sequences, disrupting the natural spatial topology. While Rotary Positional Embedding (RoPE) excels in 1D, it inherits this limitation, often treating spatially distant patches (e.g., at row edges) as sequence neighbors. Existing 2D approaches typically treat spatial axes independently, failing to decouple this false sequential proximity from true spatial distance. To restore the 2D spatial manifold, we introduce Geometric Positional Embedding (GeoPE), a framework that extends rotations to 3D Euclidean space using quaternions. To overcome non-commutativity and ensure symmetry, GeoPE constructs a unified rotational operator by computing the geometric mean in the Lie algebra. This creates a geometrically coupled encoding that effectively separates spatial dimensions. Extensive experiments on image classification, object detection, and 3D semantic segmentation demonstrate that GeoPE consistently outperforms existing 2D RoPE variants and significantly enhances shape bias, confirming its ability to capture true geometric structure.
Abstract (translated)
标准的视觉变换模型将二维图像展平为一维序列，这破坏了自然的空间拓扑结构。虽然旋转位置编码（RoPE）在处理一维数据时表现优异，但它继承了这一限制，常常将空间上距离较远的补丁（例如位于行边缘的补丁）视为序列中的邻居。现有的二维方法通常独立地处理空间轴，未能解开这种错误的顺序接近与真实的空间距离之间的关联。 为了恢复二维的空间流形结构，我们引入了几何位置编码（GeoPE），这是一个通过四元数将旋转扩展到三维欧几里得空间的框架。为克服非交换性并确保对称性，GeoPE通过在李代数中计算几何平均值来构建一个统一的旋转变换算符。这创建了一个几何耦合编码，有效地分离了空间维度。 大量的实验结果表明，在图像分类、目标检测和三维语义分割任务上，GeoPE持续优于现有的二维RoPE变体，并显著增强了形状偏向性，证明其能够捕捉真实的几何结构。
URL
https://arxiv.org/abs/2512.04963
