Abstract
In this work, we focus on scaling open-vocabulary action detection. Existing approaches for action detection are predominantly limited to closed-set scenarios and rely on complex, parameter-heavy architectures. Extending these models to the open-vocabulary setting poses two key challenges: (1) the lack of large-scale datasets with many action classes for robust training, and (2) parameter-heavy adaptations to a pretrained vision-language contrastive model to convert it for detection, risking overfitting the additional non-pretrained parameters to base action classes. Firstly, we introduce an encoder-only multimodal model for video action detection, reducing the reliance on parameter-heavy additions for video action detection. Secondly, we introduce a simple weakly supervised training strategy to exploit an existing closed-set action detection dataset for pretraining. Finally, we depart from the ill-posed base-to-novel benchmark used by prior works in open-vocabulary action detection and devise a new benchmark to evaluate on existing closed-set action detection datasets without ever using them for training, showing novel results to serve as baselines for future work.
Abstract (translated)
在这项工作中，我们专注于开放词汇动作检测的扩展。现有的动作检测方法主要局限于封闭集场景，并依赖于复杂、参数繁重的架构。将这些模型扩展到开放词汇设置面临两个关键挑战：（1）缺乏具有大量动作类别的大规模数据集以进行稳健训练；（2）对预训练的视觉-语言对比模型进行参数繁重的调整，将其转换为检测模型，这会增加额外未预训练参数过度拟合基础动作类别的风险。首先，我们引入了一个用于视频动作检测的编码器仅模式多模态模型，减少了依赖于视频动作检测中参数繁重的添加项。其次，我们提出了一种简单的弱监督训练策略，利用现有的封闭集动作检测数据集进行预训练。最后，我们偏离了之前工作在开放词汇动作检测中使用的病态基础到新颖基准，并设计了一个新的评估标准，在从未使用过的现有封闭集动作检测数据集上进行测试，展示了新颖的结果作为未来研究的基线。
URL
https://arxiv.org/abs/2504.03096
