Abstract
Adapting image-pretrained backbones to video typically relies on time-domain adapters tuned to a single temporal scale. Our experiments show that these modules pick up static image cues and very fast flicker changes, while overlooking medium-speed motion. Capturing dynamics across multiple time-scales is, however, crucial for fine-grained temporal analysis (i.e., opening vs. closing bottle). To address this, we introduce Frame2Freq -- a family of frequency-aware adapters that perform spectral encoding during image-to-video adaptation of pretrained Vision Foundation Models (VFMs), improving fine-grained action recognition. Frame2Freq uses Fast Fourier Transform (FFT) along time and learns frequency-band specific embeddings that adaptively highlight the most discriminative frequency ranges. Across five fine-grained activity recognition datasets, Frame2Freq outperforms prior PEFT methods and even surpasses fully fine-tuned models on four of them. These results provide encouraging evidence that frequency analysis methods are a powerful tool for modeling temporal dynamics in image-to-video transfer. Code is available at this https URL.
Abstract (translated)
将图像预训练骨干网络适应到视频中通常依赖于针对单一时间尺度调优的时间域适配器。我们的实验表明，这些模块倾向于捕捉静态图像线索和非常快速的闪烁变化，而忽略了中速运动。然而，跨多个时间尺度捕获动态对于细粒度的时间分析（例如，开瓶与关瓶）至关重要。为了解决这一问题，我们引入了Frame2Freq——一种频率感知适配器家族，在将预训练的视觉基础模型（VFMs）从图像转换到视频时进行频谱编码，从而改进细粒度的动作识别。 Frame2Freq使用快速傅里叶变换（FFT）沿时间轴执行，并学习特定于频率带的嵌入以自适应地突出最具判别性的频率范围。在五个细粒度活动识别数据集中，与以前的PEFT方法相比，Frame2Freq表现出色，在其中四个上甚至超过了完全微调的模型。这些结果提供了令人鼓舞的证据，表明频谱分析方法是建模图像到视频转换中的时间动态的强大工具。 代码可在以下链接获取：[此 URL]（请将此处的URL替换为实际提供的GitHub或其他存储库链接）。
URL
https://arxiv.org/abs/2602.18977
