Abstract
We address the task of zero-shot fine-grained video classification, where no video examples or temporal annotations are available for unseen action classes. While contrastive vision-language models such as SigLIP demonstrate strong open-set recognition via mean-pooled image-text similarity, they fail to capture the temporal structure critical for distinguishing fine-grained activities. We introduce ActAlign, a zero-shot framework that formulates video classification as sequence alignment. For each class, a large language model generates an ordered sub-action sequence, which is aligned with video frames using Dynamic Time Warping (DTW) in a shared embedding space. Without any video-text supervision or fine-tuning, ActAlign achieves 30.5% accuracy on the extremely challenging ActionAtlas benchmark, where human accuracy is only 61.6%. ActAlign outperforms billion-parameter video-language models while using approximately 8x less parameters. These results demonstrate that structured language priors, combined with classical alignment techniques, offer a scalable and general approach to unlocking the open-set recognition potential of vision-language models for fine-grained video understanding.
Abstract (translated)
我们解决了零样本细粒度视频分类任务，该任务在未知动作类别上没有提供任何视频示例或时间标注。虽然对比视觉-语言模型（如SigLIP）通过均值池化的图像-文本相似性展示了强大的开放集识别能力，但它们未能捕捉到区分精细动作所需的时序结构。我们引入了ActAlign，这是一个零样本框架，它将视频分类定义为序列对齐问题。对于每个类别，大型语言模型生成一个有序的子行动序列，并通过在共享嵌入空间中使用动态时间规整（DTW）与视频帧进行对齐。无需任何视频-文本监督或微调的情况下，ActAlign在极具挑战性的ActionAtlas基准测试上达到了30.5%的准确率，而人类的表现仅为61.6%。ActAlign在参数量仅约为数十亿参数视觉语言模型八分之一的情况下超越了这些模型。这一结果表明，结合结构化的语言先验知识和经典的对齐技术可以提供一种可扩展且通用的方法来释放视觉-语言模型的开放集识别潜力，从而实现细粒度视频理解。
URL
https://arxiv.org/abs/2506.22967
