Abstract
While Rotary Position Embedding (RoPE) and its variants are widely adopted for their long-context capabilities, the extension of the 1D RoPE to video, with its complex spatio-temporal structure, remains an open challenge. This work first introduces a comprehensive analysis that identifies four key characteristics essential for the effective adaptation of RoPE to video, which have not been fully considered in prior work. As part of our analysis, we introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors) task, which adds periodic distractors into V-NIAH. The V-NIAH-D task demonstrates that previous RoPE variants, lacking appropriate temporal dimension allocation, are easily misled by distractors. Based on our analysis, we introduce \textbf{VideoRoPE}, with a \textit{3D structure} designed to preserve spatio-temporal relationships. VideoRoPE features \textit{low-frequency temporal allocation} to mitigate periodic oscillations, a \textit{diagonal layout} to maintain spatial symmetry, and \textit{adjustable temporal spacing} to decouple temporal and spatial indexing. VideoRoPE consistently surpasses previous RoPE variants, across diverse downstream tasks such as long video retrieval, video understanding, and video hallucination. Our code will be available at \href{this https URL}{this https URL}.
Abstract (translated)
尽管旋转位置嵌入（RoPE）及其变体因其长上下文处理能力而被广泛采用，但将其从一维扩展到视频领域的复杂时空结构中仍然是一项开放性挑战。这项工作首先引入了一项全面分析，识别出四种关键特性对于有效将 RoPE 适应视频来说是必不可少的，这些特性在以往的工作中未得到充分考虑。作为我们分析的一部分，我们介绍了具有挑战性的 V-NIAH-D（带有干扰物的视觉针锋相对）任务，在该任务中，我们将周期性干扰物添加到了 V-NIAH 中。V-NIAH-D 任务表明，以前的 RoPE 变体由于缺乏适当的时序维度分配，容易被干扰物误导。基于我们的分析，我们引入了 **VideoRoPE**，它具有一个设计用于保持时空关系的 **3D 结构**。VideoRoPE 特征包括用于减轻周期性振荡的 **低频时域分配**、用于维持空间对称性的 **对角布局** 以及用于解耦时间和空间索引的 **可调时间间隔**。在长视频检索、视频理解及视频幻觉等多种下游任务中，VideoRoPE 持续超越了之前的 RoPE 变体。我们的代码将在 [此链接](this https URL) 提供。
URL
https://arxiv.org/abs/2502.05173
