Abstract
Modern video understanding systems excel at tasks such as scene classification, object detection, and short video retrieval. However, as video analysis becomes increasingly central to real-world applications, there is a growing need for proactive video agents for the systems that not only interpret video streams but also reason about events and take informed actions. A key obstacle in this direction is temporal reasoning: while deep learning models have made remarkable progress in recognizing patterns within individual frames or short clips, they struggle to understand the sequencing and dependencies of events over time, which is critical for action-driven decision-making. Addressing this limitation demands moving beyond conventional deep learning approaches. We posit that tackling this challenge requires a neuro-symbolic perspective, where video queries are decomposed into atomic events, structured into coherent sequences, and validated against temporal constraints. Such an approach can enhance interpretability, enable structured reasoning, and provide stronger guarantees on system behavior, all key properties for advancing trustworthy video agents. To this end, we present a grand challenge to the research community: developing the next generation of intelligent video agents that integrate three core capabilities: (1) autonomous video search and analysis, (2) seamless real-world interaction, and (3) advanced content generation. By addressing these pillars, we can transition from passive perception to intelligent video agents that reason, predict, and act, pushing the boundaries of video understanding.
Abstract (translated)
现代视频理解系统在场景分类、目标检测和短片检索等任务上表现出色。然而，随着视频分析在现实世界应用中的重要性日益增加，对于能够不仅解读视频流还能够对事件进行推理并采取明智行动的主动型视频代理的需求也在增长。这一方向上的一个关键障碍是时间推理：虽然深度学习模型在识别单帧或短片段内的模式方面取得了显著进展，但它们难以理解事件随时间排列和依赖关系，这对于基于行为的决策至关重要。解决这个限制需要超越传统的深度学习方法。我们认为，应对这一挑战需要采用神经符号学视角，即将视频查询分解为原子事件、结构化成连贯序列，并根据时间约束进行验证。这种做法可以增强可解释性，支持结构化的推理过程，并提供更有力的行为保证，这些都是构建值得信赖的视频代理的关键属性。 为此，我们向研究界提出了一个重大挑战：开发下一代智能视频代理，整合三个核心能力：(1) 自主视频搜索和分析；(2) 无缝的现实世界交互；以及 (3) 高级内容生成。通过解决这些支柱问题，我们可以从被动感知过渡到能够推理、预测并采取行动的智能视频代理，推动视频理解技术的发展边界。
URL
https://arxiv.org/abs/2505.13851
