Abstract
Current video retrieval systems, especially those used in competitions, primarily focus on querying individual keyframes or images rather than encoding an entire clip or video segment. However, queries often describe an action or event over a series of frames, not a specific image. This results in insufficient information when analyzing a single frame, leading to less accurate query results. Moreover, extracting embeddings solely from images (keyframes) does not provide enough information for models to encode higher-level, more abstract insights inferred from the video. These models tend to only describe the objects present in the frame, lacking a deeper understanding. In this work, we propose a system that integrates the latest methodologies, introducing a novel pipeline that extracts multimodal data, and incorporate information from multiple frames within a video, enabling the model to abstract higher-level information that captures latent meanings, focusing on what can be inferred from the video clip, rather than just focusing on object detection in one single image.
Abstract (translated)
当前的视频检索系统，特别是在竞赛中使用的那些，主要侧重于查询单个关键帧或图像，而不是对整个片段或视频段进行编码。然而，查询通常描述的是跨越一系列帧的动作或事件，而不仅仅是特定图像。这导致在分析单一帧时信息不足，从而产生不准确的查询结果。此外，仅从图像（关键帧）中提取嵌入信息不足以提供足够的信息，使模型能够编码更高层次、更抽象的理解，这些理解是从视频中推断出来的。这些模型倾向于仅仅描述帧中存在的对象，缺乏深层次的理解。 在本研究中，我们提出了一种整合最新方法的系统，并引入了一个新颖的管道，该管道提取多模态数据并结合视频中的多个帧的信息，使模型能够提炼出更高层次的信息，捕捉潜在的意义。我们的重点在于从视频片段中推断出来的信息，而不仅仅是关注单个图像中的目标检测。
URL
https://arxiv.org/abs/2412.07584
