Abstract
Neural models such as YOLO and HuBERT can be used to detect local properties such as objects ("car") and emotions ("angry") in individual frames of videos and audio clips respectively. The likelihood of these detections is indicated by scores in [0, 1]. Lifting these scores to temporal properties over sequences can be useful for several downstream applications such as query matching (e.g., "does the speaker eventually sound happy in this audio clip?"), and ranked retrieval (e.g., "retrieve top 5 videos with a 10 second scene where a car is detected until a pedestrian is detected"). In this work, we formalize this problem of assigning Scores for TempOral Properties (STOPs) over sequences, given potentially noisy score predictors for local properties. We then propose a scoring function called LogSTOP that can efficiently compute these scores for temporal properties represented in Linear Temporal Logic. Empirically, LogSTOP, with YOLO and HuBERT, outperforms Large Vision / Audio Language Models and other Temporal Logic-based baselines by at least 16% on query matching with temporal properties over objects-in-videos and emotions-in-speech respectively. Similarly, on ranked retrieval with temporal properties over objects and actions in videos, LogSTOP with Grounding DINO and SlowR50 reports at least a 19% and 16% increase in mean average precision and recall over zero-shot text-to-video retrieval baselines respectively.
Abstract (translated)
神经模型如YOLO和HuBERT可以用于检测视频帧中的对象（例如“汽车”）和音频片段中的情感（例如“愤怒”）。这些检测的可能性由[0, 1]范围内的分数表示。将这些分数提升为时间序列上的时序属性对于诸如查询匹配（例如，“在这个音频片段中，说话者最终听起来是否开心？”）和排名检索（例如，“检索前5个在10秒场景内从检测到汽车直到检测到行人为止的视频”）等下游应用非常有用。在这项工作中，我们形式化了给定潜在噪声局部属性评分器的情况下，在序列上为时序属性分配分数的问题，并将其命名为Scores for TempOral Properties（STOPs）。然后，我们提出了一种名为LogSTOP的评分函数，该函数可以有效地计算用线性时间逻辑表示的时序属性得分。实证研究表明，使用YOLO和HuBERT时，LogSTOP在对象视频和语音情感的时间属性查询匹配方面至少比大型视觉/音频语言模型和其他基于时间逻辑的方法高出16%。同样，在使用Grounding DINO和SlowR50进行视频中对象和动作的时间属性排名检索任务上，LogSTOP分别报告了平均精度和召回率至少提高了19%和16%，相比零样本文本到视频检索基线方法而言。
URL
https://arxiv.org/abs/2510.06512
