Abstract
Recently, there has been a surge of interest in applying deep learning techniques to animal behavior recognition, particularly leveraging pre-trained visual language models, such as CLIP, due to their remarkable generalization capacity across various downstream tasks. However, adapting these models to the specific domain of animal behavior recognition presents two significant challenges: integrating motion information and devising an effective temporal modeling scheme. In this paper, we propose AnimalMotionCLIP to address these challenges by interleaving video frames and optical flow information in the CLIP framework. Additionally, several temporal modeling schemes using an aggregation of classifiers are proposed and compared: dense, semi dense, and sparse. As a result, fine temporal actions can be correctly recognized, which is of vital importance in animal behavior analysis. Experiments on the Animal Kingdom dataset demonstrate that AnimalMotionCLIP achieves superior performance compared to state-of-the-art approaches.
Abstract (translated)
近期，人们对将深度学习技术应用于动物行为识别的兴趣大增，尤其是在利用像 CLIP 这样的预训练视觉语言模型方面。这些模型因其在各种下游任务中的卓越泛化能力而备受青睐。然而，要将这类模型适配到特定的动物行为识别领域，则面临着两个主要挑战：一是如何整合运动信息，二是如何设计有效的时序建模方案。 为此，在本文中我们提出了一种名为 AnimalMotionCLIP 的方法来解决这些挑战。该方法通过在 CLIP 模型框架内交错插入视频帧和光流信息来实现这一目标。此外，还提出了并比较了几种使用分类器聚合进行时序建模的方法：密集（dense）、半密集（semi dense）和稀疏（sparse）。结果表明，这些方案能够准确识别出精细的时序动作，在动物行为分析中具有重要意义。 通过在“动物王国”数据集上的实验验证，AnimalMotionCLIP 方法相较于现有最先进的方法取得了更优的表现。
URL
https://arxiv.org/abs/2505.00569
