Video_Retrieval
-
Prompt-aware of Frame Sampling for Efficient Text-Video Retrieval
2025-07-21 10:46:49Deyu Zhang, Tingting Long, Jinrui Zhang, Ligeng Chen, Ju Ren, Yaoxue ZhangAbstract
Enabling efficient text-video retrieval on edge-end devices is critical for real-world applications. Yet, existing methods face a critical challenge in balancing accuracy and computational efficiency: uniform frame sampling methods ensure content coverage but incur prohibitive computational costs, while salient-frame sampling methods reduce overhead but suffer from query-agnostic frame selection that biases retrieval results. To address this, we propose ProCLIP, a user-centric framework that achieves state-of-the-art accuracy with significantly improved efficiency. We design a prompt-aware frame sampling strategy that dynamically guides lightweight feature extractors using textual prompts to select semantically relevant frames, overcoming the limitations of existing salient-frame sampling methods which rely on static, query-agnostic selection criteria. Moreover, we adopt a two-stage candidate pruning strategy that combines rapid coarse filtering via a lightweight module with CLIP-powered fine-grained re-ranking, enhancing retrieval efficiency while preserving accuracy. Experiments across benchmarks show ProCLIP achieves 75.3% latency reduction versus baselines while maintaining competitive accuracy, i.e., R@1=49.0 in MSR-VTT dataset. Code is available at this https URL.
Abstract (translated)
在边缘设备上实现高效的文字-视频检索对于实际应用至关重要。然而，现有的方法面临着如何平衡准确性和计算效率的关键挑战：统一的帧采样方法虽然确保了内容覆盖范围，但带来了高昂的计算成本；而显著帧采样方法则通过减少开销来缓解这一问题，但由于缺乏基于查询的相关性选择标准，它们可能导致检索结果偏向。为解决这些问题，我们提出了ProCLIP，这是一种以用户为中心的框架，能够在保持极高准确性的前提下大幅提升效率。 我们的框架设计了一个提示感知的帧采样策略，该策略动态地使用文本提示引导轻量级特征提取器选取语义相关的帧，从而克服了现有显著帧采样方法依赖于静态且与查询无关的选择标准这一局限性。此外，我们还采用了一种两阶段候选筛选策略，结合轻量化模块的快速粗略过滤和基于CLIP（Contrastive Language-Image Pre-training）技术的精细重排名，以此提高检索效率并保持准确性。 实验结果表明，在多个基准测试中，ProCLIP相较于基线方法减少了75.3%的延迟时间，并在MSR-VTT数据集上依然保持了49.0%的准确率（以R@1度量）。该框架的代码可在提供的链接获取。
URL
https://arxiv.org/abs/2507.15491
PDF
-
U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs
2025-07-20 10:27:34Xiaojie Li, Chu Li, Shi-Zhe Chen, Xi ChenAbstract
Universal multimodal retrieval (UMR), which aims to address complex retrieval tasks where both queries and candidates span diverse modalities, has been significantly advanced by the emergence of MLLMs. While state-of-the-art MLLM-based methods in the literature predominantly adopt contrastive learning principles, they often differ in their specific training recipes. Despite their success, the mechanisms underlying their retrieval capabilities remain largely unexplored, potentially resulting in suboptimal performance and limited generalization ability. To address these issues, we present a comprehensive study aimed at uncovering the key factors that drive effective embedding learning for UMR using MLLMs. We begin by implementing a general MLLM-based embedding learning pipeline, and systematically analyze the primary contributors to high-performing universal retrieval systems. Based on this, we explore various aspects of the details in embedding generation and training strategies, including progressive transition, hard negative mining and re-ranker distillation. Notably, our findings reveal that often-overlooked factors can have a substantial impact on model performance. Building on these discoveries, we introduce a unified framework termed U-MARVEL (\textbf{U}niversal \textbf{M}ultimod\textbf{A}l \textbf{R}etrie\textbf{V}al via \textbf{E}mbedding \textbf{L}earning), which outperforms state-of-the-art competitors on the M-BEIR benchmark by a large margin in supervised settings, and also exihibits strong zero-shot performance on several tasks such as composed image retrieval and text-to-video retrieval. These results underscore the generalization potential of our framework across various embedding-based retrieval tasks. Code is available at this https URL
Abstract (translated)
通用多模态检索（UMR）旨在解决复杂检索任务，其中查询和候选对象跨越多种不同的模式。近年来，随着多模态大规模语言模型（MLLMs）的出现，这一领域取得了显著进展。尽管文献中基于MLLM的方法在顶尖水平上广泛采用对比学习原理，但它们的具体训练方法却各不相同。虽然这些方法已经取得了一定的成功，但其检索能力背后的机制仍很大程度上未被探索，这可能导致了次优性能和有限的泛化能力。 为了解决这些问题，我们提出一项全面研究，旨在揭示使用MLLM进行UMR有效嵌入学习的关键因素。首先，我们将实现一个通用的基于MLLM的嵌入学习流水线，并系统地分析促进高性能通用检索系统的主因。在此基础上，我们会探索嵌入生成和训练策略各个方面的细节，包括逐步过渡、难负样本挖掘以及重排序器蒸馏等。 值得注意的是，我们的研究发现一些经常被忽视的因素对模型性能有着显著影响。基于这些发现，我们提出了一种统一的框架——U-MARVEL（通用多模态检索嵌入学习），该框架在监督设置下于M-BEIR基准测试中大幅超越现有最佳竞争者，并且还在诸如组合图像检索和文本到视频检索等任务上表现出强大的零样本性能。这些结果凸显了我们的框架跨各种基于嵌入的检索任务上的泛化潜力。 相关代码可在提供的链接处获取。
URL
https://arxiv.org/abs/2507.14902
PDF
-
Smart Routing for Multimodal Video Retrieval: When to Search What
2025-07-12 15:45:03Kevin Dela RosaAbstract
We introduce ModaRoute, an LLM-based intelligent routing system that dynamically selects optimal modalities for multimodal video retrieval. While dense text captions can achieve 75.9% Recall@5, they require expensive offline processing and miss critical visual information present in 34% of clips with scene text not captured by ASR. By analyzing query intent and predicting information needs, ModaRoute reduces computational overhead by 41% while achieving 60.9% Recall@5. Our approach uses GPT-4.1 to route queries across ASR (speech), OCR (text), and visual indices, averaging 1.78 modalities per query versus exhaustive 3.0 modality search. Evaluation on 1.8M video clips demonstrates that intelligent routing provides a practical solution for scaling multimodal retrieval systems, reducing infrastructure costs while maintaining competitive effectiveness for real-world deployment.
Abstract (translated)
我们介绍了一种基于大型语言模型（LLM）的智能路由系统——ModaRoute，该系统能够动态选择最佳模式来进行多模态视频检索。尽管密集的文字字幕可以达到75.9%的召回率@5指标，但它们需要昂贵的离线处理，并且会错过34%包含场景文本的片段中的关键视觉信息（这些信息未被自动语音识别（ASR）捕捉到）。通过分析查询意图并预测信息需求，ModaRoute在减少计算开销41%的同时实现了60.9%的召回率@5。我们的方法使用GPT-4.1将查询路由到ASR（语音）、OCR（文本）和视觉索引之间，在每个查询中平均使用了1.78种模式，而全面搜索则需要3.0种模式。在对180万视频片段进行的评估显示，智能路由为扩展多模态检索系统提供了实际解决方案，能够降低基础设施成本并保持现实世界部署中的竞争力效果。
URL
https://arxiv.org/abs/2507.13374
PDF
-
VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
2025-07-07 00:51:57Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, Semih YavuzAbstract
Multimodal embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering over different modalities. However, existing multimodal embeddings like VLM2Vec, E5-V, GME are predominantly focused on natural images, with limited support for other visual forms such as videos and visual documents. This restricts their applicability in real-world scenarios, including AI agents, multi-modal search and recommendation, and retrieval-augmented generation (RAG). To close this gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark that extends MMEB with five new task types: visual document retrieval, video retrieval, temporal grounding, video classification and video question answering - spanning text, image, video, and visual document inputs. Next, we train VLM2Vec-V2, a general-purpose embedding model that supports text, image, video, and visual document inputs. Extensive experiments show that VLM2Vec-V2 achieves strong performance not only on the newly introduced video and document retrieval tasks, but also improves over prior baselines on the original image benchmarks. Through extensive evaluation, our study offers insights into the generalizability of various multimodal embedding models and highlights effective strategies for unified embedding learning, laying the groundwork for more scalable and adaptable representation learning in both research and real-world settings.
Abstract (translated)
多模态嵌入模型在各种下游任务中发挥了关键作用，如语义相似性、信息检索和跨不同模式的聚类。然而，现有的多模态嵌入方法（例如VLM2Vec、E5-V、GME）主要关注自然图像，对视频和其他视觉形式的支持有限，这限制了它们在现实场景中的应用范围，包括AI代理、多模态搜索和推荐以及检索增强生成(RAG)。为填补这一空白，我们提出了VLM2Vec-V2，这是一个用于跨多种视觉形式学习嵌入的统一框架。 首先，我们介绍MMEB-V2，这是一个综合基准测试库，它在原有的MMEB基础上新增了五种任务类型：视觉文档检索、视频检索、时间定位（temporal grounding）、视频分类和视频问答——这些任务涵盖了文本、图像、视频和可视文档输入。接下来，我们训练了一个通用的嵌入模型VLM2Vec-V2，该模型支持文本、图像、视频和可视文档输入。 经过广泛的实验验证，结果显示VLM2Vec-V2不仅在新引入的视频和文档检索任务中表现出色，在原有的图像基准测试上也超越了先前的方法。通过全面评估，我们的研究为各种多模态嵌入模型的泛化能力提供了见解，并突出了统一嵌入学习的有效策略，从而为科研界和实际应用中的可扩展性和适应性表示学习奠定了基础。
URL
https://arxiv.org/abs/2507.04590
PDF
-
Are Synthetic Videos Useful? A Benchmark for Retrieval-Centric Evaluation of Synthetic Videos
2025-07-03 05:01:46Zecheng Zhao, Selena Song, Tong Chen, Zhi Chen, Shazia Sadiq, Yadan LuoAbstract
Text-to-video (T2V) synthesis has advanced rapidly, yet current evaluation metrics primarily capture visual quality and temporal consistency, offering limited insight into how synthetic videos perform in downstream tasks such as text-to-video retrieval (TVR). In this work, we introduce SynTVA, a new dataset and benchmark designed to evaluate the utility of synthetic videos for building retrieval models. Based on 800 diverse user queries derived from MSRVTT training split, we generate synthetic videos using state-of-the-art T2V models and annotate each video-text pair along four key semantic alignment dimensions: Object \& Scene, Action, Attribute, and Prompt Fidelity. Our evaluation framework correlates general video quality assessment (VQA) metrics with these alignment scores, and examines their predictive power for downstream TVR performance. To explore pathways of scaling up, we further develop an Auto-Evaluator to estimate alignment quality from existing metrics. Beyond benchmarking, our results show that SynTVA is a valuable asset for dataset augmentation, enabling the selection of high-utility synthetic samples that measurably improve TVR outcomes. Project page and dataset can be found at this https URL.
Abstract (translated)
文本到视频（T2V）合成技术取得了迅速进展，然而当前的评估指标主要关注于视觉质量和时间一致性，对于下游任务如文本到视频检索（TVR）中合成视频的表现提供了有限的信息。在此项工作中，我们介绍了SynTVA，这是一个全新的数据集和基准测试工具，旨在评估合成视频在构建检索模型方面的实用性。基于从MSRVTT训练集中衍生出的800个多样化用户查询，我们使用最先进的T2V模型生成了合成视频，并对每个视频-文本配对进行了四项关键语义对齐维度上的标注：对象与场景、动作、属性以及提示忠实度。我们的评估框架将通用视频质量评估（VQA）指标与这些对齐分数相关联，并考察它们对于下游TVR性能的预测能力。为了探索扩展路径，我们进一步开发了一个自动评价器来从现有指标中估算对齐质量。除了基准测试之外，我们的结果表明SynTVA是一个有价值的资产，可用于数据集增强，能够选择那些能显著提升TVR成果的高实用性的合成样本。项目页面和数据集可以在此URL找到：[此链接位置应填写具体网址]。
URL
https://arxiv.org/abs/2507.02316
PDF
-
Can Sound Replace Vision in LLaVA With Token Substitution?
2025-06-12 07:14:23Ali Vosoughi, Jing Bi, Pinxin Liu, Yunlong Tang, Chenliang XuAbstract
While multimodal systems have achieved impressive advances, they typically rely on text-aligned representations rather than directly integrating audio and visual inputs. This reliance can limit the use of acoustic information in tasks requiring nuanced audio understanding. In response, SoundCLIP explores direct audio-visual integration within multimodal large language models (MLLMs) by substituting CLIP's visual tokens with audio representations and selecting sound-relevant patch tokens in models such as LLaVA. We investigate two configurations: (1) projecting audio features into CLIP's visual manifold via a multilayer perceptron trained with InfoNCE on paired audio-video segments, and (2) preserving raw audio embeddings with minimal dimensional adjustments. Experiments with five state-of-the-art audio encoders reveal a fundamental trade-off. While audio-to-video retrieval performance increases dramatically (up to 44 percentage points in Top-1 accuracy) when audio is projected into CLIP's space, text generation quality declines. Encoders pre-trained with text supervision (CLAP, Whisper, ImageBind) maintain stronger generative capabilities than those focused primarily on audiovisual alignment (Wav2CLIP, AudioCLIP), highlighting the value of language exposure for generation tasks. We introduce WhisperCLIP, an architecture that fuses intermediate representations from Whisper, as well as AudioVisual Event Evaluation (AVE-2), a dataset of 580,147 three-second audiovisual clips with fine-grained alignment annotations. Our findings challenge the assumption that stronger cross-modal alignment necessarily benefits all multimodal tasks; instead, a Pareto frontier emerges wherein optimal performance depends on balancing retrieval accuracy with text generation quality. Codes and datasets: this https URL.
Abstract (translated)
尽管多模态系统已经取得了令人印象深刻的进展，但它们通常依赖于文本对齐的表示方式，而不是直接整合音频和视觉输入。这种依赖性可能会限制在需要细腻音频理解的任务中利用声学信息的能力。为此，SoundCLIP 探索了在多模态大型语言模型（MLLMs）中进行直接音视频集成的方法，通过将 CLIP 的视觉标记替换为音频表示，并在如 LLaVA 等模型中选择与声音相关的补丁标记。我们研究了两种配置：(1) 通过 InfoNCE 训练的多层感知机，将音频特征投影到 CLIP 的视觉流形空间；(2) 在保留原始音频嵌入的同时进行最少的维度调整。 使用五种最先进的音频编码器进行实验揭示了一个基本的权衡。当音频被投射进入 CLIP 空间时，音视频检索性能显著提高（Top-1 准确率最多提高了 44 个百分点），但文本生成质量下降了。经过文本监督预训练的编码器（如 CLAP、Whisper 和 ImageBind）比主要关注视听对齐的编码器（Wav2CLIP、AudioCLIP）具有更强的生成能力，这凸显了语言曝光对于生成任务的价值。 我们介绍了 WhisperCLIP 架构，该架构融合了来自 Whisper 的中间表示，并且引入了音频视频事件评估数据集 AVE-2，这是一个包含 580,147 条三秒音视频片段的数据集，每条片段都有精细的对齐注释。我们的研究结果挑战了更强跨模态对齐必然有益于所有多模态任务这一假设；相反地，一个帕累托前沿图出现了，在该图中，最佳性能取决于检索准确性和文本生成质量之间的平衡。 代码和数据集：[这个链接](https://this-url.com)。
URL
https://arxiv.org/abs/2506.10416
PDF
-
Q2E: Query-to-Event Decomposition for Zero-Shot Multilingual Text-to-Video Retrieval
2025-06-11 21:37:54Shubhashis Roy Dipta, Francis FerraroAbstract
Recent approaches have shown impressive proficiency in extracting and leveraging parametric knowledge from Large-Language Models (LLMs) and Vision-Language Models (VLMs). In this work, we consider how we can improve the identification and retrieval of videos related to complex real-world events by automatically extracting latent parametric knowledge about those events. We present Q2E: a Query-to-Event decomposition method for zero-shot multilingual text-to-video retrieval, adaptable across datasets, domains, LLMs, or VLMs. Our approach demonstrates that we can enhance the understanding of otherwise overly simplified human queries by decomposing the query using the knowledge embedded in LLMs and VLMs. We additionally show how to apply our approach to both visual and speech-based inputs. To combine this varied multimodal knowledge, we adopt entropy-based fusion scoring for zero-shot fusion. Through evaluations on two diverse datasets and multiple retrieval metrics, we demonstrate that Q2E outperforms several state-of-the-art baselines. Our evaluation also shows that integrating audio information can significantly improve text-to-video retrieval. We have released code and data for future research.
Abstract (translated)
近期的研究展示了从大型语言模型（LLMs）和视觉-语言模型（VLMs）中提取参数知识并加以利用的出色能力。在本工作中，我们探讨了如何通过自动抽取关于复杂现实事件的潜在参数化知识来改进视频识别与检索的方法。为此，我们提出了Q2E：一种适用于零样本跨语种文本到视频检索的查询拆解方法（Query-to-Event），此方法具有数据集、领域、LLMs或VLMs间的适应性。我们的方法表明可以通过使用嵌入在LLMs和VLMs中的知识来分解查询，从而增强对原本过于简化的用户查询的理解能力。此外，我们还展示了如何将这种方法应用于视觉与语音输入。 为了整合这种多样化的多模态信息，我们采用了基于熵的融合评分进行零样本融合。通过两个不同的数据集及多个检索指标的评估，证明了Q2E优于几个最新的基准方法。我们的评估结果也表明，结合音频信息可以显著提升文本到视频检索的效果。我们已发布了代码和数据以供未来研究之用。
URL
https://arxiv.org/abs/2506.10202
PDF
-
Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval
2025-06-09 06:44:45CH Cho, WJ Moon, W Jun, MS Jung, JP HeoAbstract
Partially Relevant Video Retrieval~(PRVR) aims to retrieve a video where a specific segment is relevant to a given text query. Typical training processes of PRVR assume a one-to-one relationship where each text query is relevant to only one video. However, we point out the inherent ambiguity between text and video content based on their conceptual scope and propose a framework that incorporates this ambiguity into the model learning process. Specifically, we propose Ambiguity-Restrained representation Learning~(ARL) to address ambiguous text-video pairs. Initially, ARL detects ambiguous pairs based on two criteria: uncertainty and similarity. Uncertainty represents whether instances include commonly shared context across the dataset, while similarity indicates pair-wise semantic overlap. Then, with the detected ambiguous pairs, our ARL hierarchically learns the semantic relationship via multi-positive contrastive learning and dual triplet margin loss. Additionally, we delve into fine-grained relationships within the video instances. Unlike typical training at the text-video level, where pairwise information is provided, we address the inherent ambiguity within frames of the same untrimmed video, which often contains multiple contexts. This allows us to further enhance learning at the text-frame level. Lastly, we propose cross-model ambiguity detection to mitigate the error propagation that occurs when a single model is employed to detect ambiguous pairs for its training. With all components combined, our proposed method demonstrates its effectiveness in PRVR.
Abstract (translated)
部分相关视频检索（PRVR）的目标是从大量视频中提取出特定段落与给定文本查询相关的视频。传统PRVR的训练过程假设一个一对一的关系，即每个文本查询仅对应一个视频。然而，我们指出了基于概念范围的文本和视频内容之间的固有歧义，并提出了一个将这种歧义纳入模型学习过程的框架。具体来说，我们提出了一种名为“歧义受限表示学习”（Ambiguity-Restrained Learning, ARL）的方法来解决含有模糊配对的文本-视频问题。 最初，ARL基于两个标准检测模糊配对：不确定性以及相似度。不确定性代表实例是否包含了跨整个数据集常见的上下文信息；而相似度则表明成对的语义重叠。然后，在检测到模糊配对之后，我们的ARL通过多正对比学习和双三元组边缘损失来分层学习语义关系。 此外，我们深入探讨了视频片段内的细粒度关联性。不同于典型的文本-视频层级训练中提供的两两信息，我们针对同一未修剪视频中的帧内固有的歧义进行了处理，通常这些视频包含多个上下文场景。这使我们在文本-帧层次上进一步增强了学习能力。 最后，我们提出了跨模型的歧义检测来缓解当单一模型被用于模糊配对训练时出现的错误传播问题。结合所有组件，我们的方法在PRVR中表现出显著的有效性。
URL
https://arxiv.org/abs/2506.07471
PDF
-
MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
2025-06-08 06:34:29Sanjoy Chowdhury, Mohamed Elmoghany, Yohan Abeysinghe, Junjie Fei, Sayan Nag, Salman Khan, Mohamed Elhoseiny, Dinesh ManochaAbstract
Large multimodal models (LMMs) have shown remarkable progress in audio-visual understanding, yet they struggle with real-world scenarios that require complex reasoning across extensive video collections. Existing benchmarks for video question answering remain limited in scope, typically involving one clip per query, which falls short of representing the challenges of large-scale, audio-visual retrieval and reasoning encountered in practical applications. To bridge this gap, we introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer. To this end, we present AVHaystacks, an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task. Additionally, we propose a model-agnostic, multi-agent framework MAGNET to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystacks. To enable robust evaluation of multi-video retrieval and temporal grounding for optimal response generation, we introduce two new metrics, STEM, which captures alignment errors between a ground truth and a predicted step sequence and MTGS, to facilitate balanced and interpretable evaluation of segment-level grounding performance. Project: this https URL
Abstract (translated)
大型多模态模型（LMMs）在音频-视觉理解方面取得了显著的进步，但它们在处理需要跨大量视频集合进行复杂推理的真实场景时仍面临挑战。现有的视频问答基准测试范围有限，通常每个查询仅涉及一个片段，这无法代表大规模、音视检索和推理的实际应用中的挑战。为了弥合这一差距，我们引入了一个名为AV-HaystacksQA的新任务，在该任务中，目标是在响应查询时识别不同视频之间的关键段落，并将它们链接起来以生成最具有信息量的答案。为此，我们提出了一个音频-视觉基准测试AVHaystacks，包含3100个带有标注的问答对，旨在评估LMMs在多视频检索和时间定位任务中的能力。此外，为解决这一挑战，我们提出了一种模型无关、多代理框架MAGNET，在我们提出的AVHaystacks基准上的QA任务中，与基线方法相比，其BLEU@4评分提升了高达89%，GPT评价分数提升了65%。为了实现对多视频检索和时间定位的有效评估以生成最佳响应，我们引入了两个新的指标：STEM用于捕捉实际步骤序列与预测步骤序列之间的对齐错误；MTGS则有助于平衡和可解释地评估段落级定位性能。 项目链接：[此链接](https://this-url.com)
URL
https://arxiv.org/abs/2506.07016
PDF
-
From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos
2025-06-05 17:31:17Animesh Gupta, Jay Parmar, Ishan Rajendrakumar Dave, Mubarak ShahAbstract
Composed Video Retrieval (CoVR) retrieves a target video given a query video and a modification text describing the intended change. Existing CoVR benchmarks emphasize appearance shifts or coarse event changes and therefore do not test the ability to capture subtle, fast-paced temporal differences. We introduce TF-CoVR, the first large-scale benchmark dedicated to temporally fine-grained CoVR. TF-CoVR focuses on gymnastics and diving and provides 180K triplets drawn from FineGym and FineDiving. Previous CoVR benchmarks focusing on temporal aspect, link each query to a single target segment taken from the same video, limiting practical usefulness. In TF-CoVR, we instead construct each <query, modification> pair by prompting an LLM with the label differences between clips drawn from different videos; every pair is thus associated with multiple valid target videos (3.9 on average), reflecting real-world tasks such as sports-highlight generation. To model these temporal dynamics we propose TF-CoVR-Base, a concise two-stage training framework: (i) pre-train a video encoder on fine-grained action classification to obtain temporally discriminative embeddings; (ii) align the composed query with candidate videos using contrastive learning. We conduct the first comprehensive study of image, video, and general multimodal embedding (GME) models on temporally fine-grained composed retrieval in both zero-shot and fine-tuning regimes. On TF-CoVR, TF-CoVR-Base improves zero-shot mAP@50 from 5.92 (LanguageBind) to 7.51, and after fine-tuning raises the state-of-the-art from 19.83 to 25.82.
Abstract (translated)
构成视频检索（CoVR）是指在给定查询视频和描述预期变化的修改文本的情况下，检索目标视频。现有的CoVR基准测试主要侧重于外观的变化或粗略事件的更改，因此无法检测捕捉细微、快速的时间差异的能力。我们引入了TF-CoVR，这是首个专注于时间细粒度构成视频检索的大规模基准测试。TF-CoVR聚焦于体操和跳水项目，并提供了180K个三元组，这些数据来源于FineGym和FineDiving。 之前的侧重于时间方面的CoVR基准测试将每个查询与同一视频中的单个目标片段相连，限制了其实用性。在TF-CoVR中，我们通过提示LLM使用从不同视频剪辑之间提取的标签差异来构建每个<query, modification>对；因此，每一对都关联到多个有效的目标视频（平均为3.9），反映了诸如体育精彩集锦生成等现实世界任务。 为了模拟这些时间动态变化，我们提出了TF-CoVR-Base，这是一种简洁的两阶段训练框架：(i) 在细粒度动作分类上对视频编码器进行预训练以获得具有时间区分性的嵌入；(ii) 使用对比学习方法将构成查询与候选视频对齐。我们在零样本和微调设置下，进行了首个关于图像、视频以及通用多模态嵌入（GME）模型在时间细粒度构成检索中的全面研究。 在TF-CoVR上，TF-CoVR-Base使零样本mAP@50从5.92（LanguageBind）提升到了7.51，在微调后将最先进的水平从19.83提高到了25.82。
URL
https://arxiv.org/abs/2506.05274
PDF
-
BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance
2025-06-04 05:40:54Huy Le, Nhat Chung, Tung Kieu, Anh Nguyen, Ngan LeAbstract
Text-video retrieval (TVR) systems often suffer from visual-linguistic biases present in datasets, which cause pre-trained vision-language models to overlook key details. To address this, we propose BiMa, a novel framework designed to mitigate biases in both visual and textual representations. Our approach begins by generating scene elements that characterize each video by identifying relevant entities/objects and activities. For visual debiasing, we integrate these scene elements into the video embeddings, enhancing them to emphasize fine-grained and salient details. For textual debiasing, we introduce a mechanism to disentangle text features into content and bias components, enabling the model to focus on meaningful content while separately handling biased information. Extensive experiments and ablation studies across five major TVR benchmarks (i.e., MSR-VTT, MSVD, LSMDC, ActivityNet, and DiDeMo) demonstrate the competitive performance of BiMa. Additionally, the model's bias mitigation capability is consistently validated by its strong results on out-of-distribution retrieval tasks.
Abstract (translated)
文本-视频检索（TVR）系统通常会受到数据集中存在的视觉-语言偏见的影响，导致预训练的视觉-语言模型忽视关键细节。为解决这一问题，我们提出了BiMa，这是一个旨在减轻视觉和文本表示中偏差的新颖框架。我们的方法从生成场景元素开始，这些元素通过识别相关的实体/对象和活动来表征每个视频。为了进行视觉去偏，我们将这些场景元素整合到视频嵌入中，增强它们以强调细微的和显著的细节。对于文本去偏，我们引入了一种机制，将文本特征分解为内容和偏差部分，使模型能够在处理有偏信息的同时专注于有意义的内容。 在五个主要的TVR基准测试（即MSR-VTT、MSVD、LSMDC、ActivityNet 和DiDeMo）上进行的广泛实验和消融研究显示了BiMa的竞争性能。此外，该模型通过其在外分布检索任务上的强大结果，持续验证了其减轻偏差的能力。
URL
https://arxiv.org/abs/2506.03589
PDF
-
MamFusion: Multi-Mamba with Temporal Fusion for Partially Relevant Video Retrieval
2025-06-04 01:08:03Xinru Ying, Jiaqi Mo, Jingyang Lin, Canghong Jin, Fangfang Wang, Lina WeiAbstract
Partially Relevant Video Retrieval (PRVR) is a challenging task in the domain of multimedia retrieval. It is designed to identify and retrieve untrimmed videos that are partially relevant to the provided query. In this work, we investigate long-sequence video content understanding to address information redundancy issues. Leveraging the outstanding long-term state space modeling capability and linear scalability of the Mamba module, we introduce a multi-Mamba module with temporal fusion framework (MamFusion) tailored for PRVR task. This framework effectively captures the state-relatedness in long-term video content and seamlessly integrates it into text-video relevance understanding, thereby enhancing the retrieval process. Specifically, we introduce Temporal T-to-V Fusion and Temporal V-to-T Fusion to explicitly model temporal relationships between text queries and video moments, improving contextual awareness and retrieval accuracy. Extensive experiments conducted on large-scale datasets demonstrate that MamFusion achieves state-of-the-art performance in retrieval effectiveness. Code is available at the link: this https URL.
Abstract (translated)
部分相关视频检索（PRVR）是多媒体检索领域中的一个挑战性任务，旨在识别和提取与提供的查询部分相关的未剪辑视频。在这项工作中，我们研究了长序列视频内容理解以解决信息冗余问题。通过利用Mamba模块出色的长期状态空间建模能力和线性扩展能力，我们引入了一个配备时间融合框架的多Mamba模块（MamFusion），专门用于PRVR任务。该框架能够有效捕捉长期视频内容中的状态相关性，并将其无缝集成到文本与视频的相关理解中，从而提升检索过程的效果。 具体来说，我们介绍了时间T-to-V融合和时间V-to-T融合，以明确建模文本查询和视频时刻之间的时间关系，提高上下文感知能力和检索准确性。在大规模数据集上进行的大量实验表明，MamFusion在检索效果方面达到了最先进的性能。代码可在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2506.03473
PDF
-
Respond Beyond Language: A Benchmark for Video Generation in Response to Realistic User Intents
2025-06-02 13:52:21Shuting Wang, Yunqi Liu, Zixin Yang, Ning Hu, Zhicheng Dou, Chenyan XiongAbstract
Querying generative AI models, e.g., large language models (LLMs), has become a prevalent method for information acquisition. However, existing query-answer datasets primarily focus on textual responses, making it challenging to address complex user queries that require visual demonstrations or explanations for better understanding. To bridge this gap, we construct a benchmark, RealVideoQuest, designed to evaluate the abilities of text-to-video (T2V) models in answering real-world, visually grounded queries. It identifies 7.5K real user queries with video response intents from Chatbot-Arena and builds 4.5K high-quality query-video pairs through a multistage video retrieval and refinement process. We further develop a multi-angle evaluation system to assess the quality of generated video answers. Experiments indicate that current T2V models struggle with effectively addressing real user queries, pointing to key challenges and future research opportunities in multimodal AI.
Abstract (translated)
查询生成式人工智能模型（如大型语言模型，LLMs）已成为信息获取的一种常见方法。然而，现有的查询-回答数据集主要侧重于文本响应，这使得难以解决那些需要视觉演示或解释以更好地理解的复杂用户问题成为挑战。为了弥合这一差距，我们构建了一个基准测试框架 RealVideoQuest，旨在评估文字转视频（T2V）模型在回答真实世界、基于视觉的问题时的能力。该框架从 Chatbot-Arena 中识别出具有视频回应意图的 7,500 条真实用户查询，并通过多阶段视频检索和优化过程构建了高质量的查询-视频对共计 4,500 对。此外，我们还开发了一个多角度评估系统来衡量生成的视频回答的质量。实验表明，当前的文字转视频模型在有效处理真实用户的查询方面存在困难，这指出了多模态人工智能领域中的关键挑战和未来研究机会。
URL
https://arxiv.org/abs/2506.01689
PDF
-
Uneven Event Modeling for Partially Relevant Video Retrieval
2025-06-01 08:21:45Sa Zhu, Huashan Chen, Wanqian Zhang, Jinchao Zhang, Zexian Yang, Xiaoshuai Hao, Bo LiAbstract
Given a text query, partially relevant video retrieval (PRVR) aims to retrieve untrimmed videos containing relevant moments, wherein event modeling is crucial for partitioning the video into smaller temporal events that partially correspond to the text. Previous methods typically segment videos into a fixed number of equal-length clips, resulting in ambiguous event boundaries. Additionally, they rely on mean pooling to compute event representations, inevitably introducing undesired misalignment. To address these, we propose an Uneven Event Modeling (UEM) framework for PRVR. We first introduce the Progressive-Grouped Video Segmentation (PGVS) module, to iteratively formulate events in light of both temporal dependencies and semantic similarity between consecutive frames, enabling clear event boundaries. Furthermore, we also propose the Context-Aware Event Refinement (CAER) module to refine the event representation conditioned the text's cross-attention. This enables event representations to focus on the most relevant frames for a given text, facilitating more precise text-video alignment. Extensive experiments demonstrate that our method achieves state-of-the-art performance on two PRVR benchmarks.
Abstract (translated)
给定一个文本查询，部分相关视频检索（PRVR）的目标是从未修剪的视频中检索包含相关内容片段的视频。在此过程中，事件建模对于将视频分割成较小的时间段事件至关重要，这些时间段与文本部分内容相对应。以往的方法通常会将视频分割为固定数量、等长的片段，这会导致模糊的事件边界。此外，它们依赖于均值池化来计算事件表示，不可避免地引入了不必要的错位。 为了应对这些问题，我们提出了一个不均匀事件建模（UEM）框架用于PRVR。首先，我们介绍了渐进分组视频分割（PGVS）模块，该模块可以迭代地根据连续帧之间的时序依赖性和语义相似性来制定事件，从而实现清晰的事件边界。此外，我们还提出了一种基于上下文感知事件细化（CAER）模块，以条件于文本跨注意力的方式对事件表示进行精炼。这使得事件表示能够专注于与给定文本最相关的帧上，有助于更精确地进行文本和视频之间的对齐。 广泛的实验表明，我们的方法在两个PRVR基准测试中达到了最先进的性能水平。
URL
https://arxiv.org/abs/2506.00891
PDF
-
Leveraging Auxiliary Information in Text-to-Video Retrieval: A Review
2025-05-29 19:02:48Adriano Fragomeni, Dima Damen, Michael WrayAbstract
Text-to-Video (T2V) retrieval aims to identify the most relevant item from a gallery of videos based on a user's text query. Traditional methods rely solely on aligning video and text modalities to compute the similarity and retrieve relevant items. However, recent advancements emphasise incorporating auxiliary information extracted from video and text modalities to improve retrieval performance and bridge the semantic gap between these modalities. Auxiliary information can include visual attributes, such as objects; temporal and spatial context; and textual descriptions, such as speech and rephrased captions. This survey comprehensively reviews 81 research papers on Text-to-Video retrieval that utilise such auxiliary information. It provides a detailed analysis of their methodologies; highlights state-of-the-art results on benchmark datasets; and discusses available datasets and their auxiliary information. Additionally, it proposes promising directions for future research, focusing on different ways to further enhance retrieval performance using this information.
Abstract (translated)
文本到视频（T2V）检索的目标是根据用户的文本查询从一系列视频中识别出最相关的视频。传统的方法主要依赖于对齐视频和文本模态以计算相似性并检索相关项。然而，最近的研究强调了利用从视频和文本模态中提取的辅助信息来改善检索性能，并弥合这些模态之间的语义差距的重要性。这种辅助信息可以包括视觉属性（如对象）、时间和空间上下文以及文本描述（如对话和重写的字幕）。本综述全面回顾了81篇关于利用此类辅助信息进行文本到视频检索的研究论文，详细分析了它们的方法论；强调了在基准数据集上的最新成果，并讨论了可用的数据集及其辅助信息。此外，它还提出了未来研究的有前景的方向，重点是通过使用这些信息来进一步提高检索性能的不同方法。
URL
https://arxiv.org/abs/2505.23952
PDF
-
Learning World Models for Interactive Video Generation
2025-05-28 05:55:44Taiye Chen, Xun Hu, Zihan Ding, Chi JinAbstract
Foundational world models must be both interactive and preserve spatiotemporal coherence for effective future planning with action choices. However, present models for long video generation have limited inherent world modeling capabilities due to two main challenges: compounding errors and insufficient memory mechanisms. We enhance image-to-video models with interactive capabilities through additional action conditioning and autoregressive framework, and reveal that compounding error is inherently irreducible in autoregressive video generation, while insufficient memory mechanism leads to incoherence of world models. We propose video retrieval augmented generation (VRAG) with explicit global state conditioning, which significantly reduces long-term compounding errors and increases spatiotemporal consistency of world models. In contrast, naive autoregressive generation with extended context windows and retrieval-augmented generation prove less effective for video generation, primarily due to the limited in-context learning capabilities of current video models. Our work illuminates the fundamental challenges in video world models and establishes a comprehensive benchmark for improving video generation models with internal world modeling capabilities.
Abstract (translated)
基础世界模型必须既具有交互性又能够保持时空一致性，以便在有效规划未来行动时做出选择。然而，目前用于长视频生成的模型由于两个主要挑战而具备有限的世界建模能力：累积误差和不足的记忆机制。我们通过添加行为条件化和自回归框架来增强图像到视频模型的互动功能，并揭示了自回归视频生成中的累积错误本质上是不可减少的，而缺乏记忆机制则会导致世界模型不一致。为此，我们提出了一种基于视频检索增强生成（VRAG）的方法，该方法通过明确的全局状态条件化显著减少了长期累积误差并增加了时空一致性。 相比之下，简单的具有扩展上下文窗口和检索增强的自回归生成对于视频生成效果较差，主要是因为目前的视频模型在上下文学习能力方面存在限制。我们的工作揭示了视频世界模型中的基本挑战，并为改进具备内部世界建模功能的视频生成模型建立了全面基准。
URL
https://arxiv.org/abs/2505.21996
PDF
-
LoVR: A Benchmark for Long Video Retrieval in Multimodal Contexts
2025-05-20 04:49:09Qifeng Cai, Hao Liang, Hejun Dong, Meiyi Qiang, Ruichuan An, Zhaoyang Han, Zhengzhou Zhu, Bin Cui, Wentao ZhangAbstract
Long videos contain a vast amount of information, making video-text retrieval an essential and challenging task in multimodal learning. However, existing benchmarks suffer from limited video duration, low-quality captions, and coarse annotation granularity, which hinder the evaluation of advanced video-text retrieval methods. To address these limitations, we introduce LoVR, a benchmark specifically designed for long video-text retrieval. LoVR contains 467 long videos and over 40,804 fine-grained clips with high-quality captions. To overcome the issue of poor machine-generated annotations, we propose an efficient caption generation framework that integrates VLM automatic generation, caption quality scoring, and dynamic refinement. This pipeline improves annotation accuracy while maintaining scalability. Furthermore, we introduce a semantic fusion method to generate coherent full-video captions without losing important contextual information. Our benchmark introduces longer videos, more detailed captions, and a larger-scale dataset, presenting new challenges for video understanding and retrieval. Extensive experiments on various advanced embedding models demonstrate that LoVR is a challenging benchmark, revealing the limitations of current approaches and providing valuable insights for future research. We release the code and dataset link at this https URL
Abstract (translated)
长时间视频包含大量的信息，使得视频-文本检索成为多模态学习中一个既重要又具挑战性的任务。然而，现有的基准测试由于视频时长有限、字幕质量低以及标注精细度不足等问题，在评估先进的视频-文本检索方法方面存在局限性。为解决这些问题，我们引入了LoVR这一专门用于长时间视频-文本检索的基准。 LoVR包含467段长时间视频和超过40,804个高质量细粒度片段。为了克服机器生成注释质量不佳的问题，我们提出了一种高效的字幕生成框架，该框架结合了视觉语言模型自动生成功能、字幕质量评分以及动态优化方法。这种流程在保证准确性的同时还保持了可扩展性。此外，我们引入了一个语义融合的方法来生成连贯的全视频描述而不丢失重要的上下文信息。 我们的基准测试提供了更长的视频、更为详细的注释和更大规模的数据集，为视频理解和检索带来了新的挑战。在各种先进的嵌入模型上的广泛实验表明，LoVR是一个具有挑战性的基准测试，揭示了现有方法的局限性，并为未来研究提供了有价值的见解。我们可以在这里（请将“this https URL”替换为实际链接）提供代码和数据集的访问。 通过这些改进，LoVR旨在促进视频-文本检索领域的发展，推动该领域的模型性能进一步提升。
URL
https://arxiv.org/abs/2505.13928
PDF
-
A Challenge to Build Neuro-Symbolic Video Agents
2025-05-20 02:53:21Sahil Shah, Harsh Goel, Sai Shankar Narasimhan, Minkyu Choi, S P Sharan, Oguzhan Akcin, Sandeep ChinchaliAbstract
Modern video understanding systems excel at tasks such as scene classification, object detection, and short video retrieval. However, as video analysis becomes increasingly central to real-world applications, there is a growing need for proactive video agents for the systems that not only interpret video streams but also reason about events and take informed actions. A key obstacle in this direction is temporal reasoning: while deep learning models have made remarkable progress in recognizing patterns within individual frames or short clips, they struggle to understand the sequencing and dependencies of events over time, which is critical for action-driven decision-making. Addressing this limitation demands moving beyond conventional deep learning approaches. We posit that tackling this challenge requires a neuro-symbolic perspective, where video queries are decomposed into atomic events, structured into coherent sequences, and validated against temporal constraints. Such an approach can enhance interpretability, enable structured reasoning, and provide stronger guarantees on system behavior, all key properties for advancing trustworthy video agents. To this end, we present a grand challenge to the research community: developing the next generation of intelligent video agents that integrate three core capabilities: (1) autonomous video search and analysis, (2) seamless real-world interaction, and (3) advanced content generation. By addressing these pillars, we can transition from passive perception to intelligent video agents that reason, predict, and act, pushing the boundaries of video understanding.
Abstract (translated)
现代视频理解系统在场景分类、目标检测和短片检索等任务上表现出色。然而，随着视频分析在现实世界应用中的重要性日益增加，对于能够不仅解读视频流还能够对事件进行推理并采取明智行动的主动型视频代理的需求也在增长。这一方向上的一个关键障碍是时间推理：虽然深度学习模型在识别单帧或短片段内的模式方面取得了显著进展，但它们难以理解事件随时间排列和依赖关系，这对于基于行为的决策至关重要。解决这个限制需要超越传统的深度学习方法。我们认为，应对这一挑战需要采用神经符号学视角，即将视频查询分解为原子事件、结构化成连贯序列，并根据时间约束进行验证。这种做法可以增强可解释性，支持结构化的推理过程，并提供更有力的行为保证，这些都是构建值得信赖的视频代理的关键属性。 为此，我们向研究界提出了一个重大挑战：开发下一代智能视频代理，整合三个核心能力：(1) 自主视频搜索和分析；(2) 无缝的现实世界交互；以及 (3) 高级内容生成。通过解决这些支柱问题，我们可以从被动感知过渡到能够推理、预测并采取行动的智能视频代理，推动视频理解技术的发展边界。
URL
https://arxiv.org/abs/2505.13851
PDF
-
Rebalancing Contrastive Alignment with Learnable Semantic Gaps in Text-Video Retrieval
2025-05-18 17:18:06Jian Xiao, Zijie Song, Jialong Hu, Hao Cheng, Zhenzhen Hu, Jia Li, Richang HongAbstract
Recent advances in text-video retrieval have been largely driven by contrastive learning frameworks. However, existing methods overlook a key source of optimization tension: the separation between text and video distributions in the representation space (referred to as the modality gap), and the prevalence of false negatives in batch sampling. These factors lead to conflicting gradients under the InfoNCE loss, impeding stable alignment. To mitigate this, we propose GARE, a Gap-Aware Retrieval framework that introduces a learnable, pair-specific increment Delta_ij between text t_i and video v_j to offload the tension from the global anchor representation. We first derive the ideal form of Delta_ij via a coupled multivariate first-order Taylor approximation of the InfoNCE loss under a trust-region constraint, revealing it as a mechanism for resolving gradient conflicts by guiding updates along a locally optimal descent direction. Due to the high cost of directly computing Delta_ij, we introduce a lightweight neural module conditioned on the semantic gap between each video-text pair, enabling structure-aware correction guided by gradient supervision. To further stabilize learning and promote interpretability, we regularize Delta using three components: a trust-region constraint to prevent oscillation, a directional diversity term to promote semantic coverage, and an information bottleneck to limit redundancy. Experiments across four retrieval benchmarks show that GARE consistently improves alignment accuracy and robustness to noisy supervision, confirming the effectiveness of gap-aware tension mitigation.
Abstract (translated)
最近在文本-视频检索领域的进展主要受到对比学习框架的推动。然而，现有的方法忽视了一个重要的优化张力来源：即文本和视频在表示空间中的分布分离（称为模态差距）以及批量采样中假阴性的普遍存在。这些因素导致了在InfoNCE损失下的梯度冲突，阻碍了稳定的对齐过程。 为了解决这个问题，我们提出了GARE（Gap-Aware Retrieval），这是一种基于感知间隙的检索框架，它引入了一个可学习的、特定于每对的增量Delta_ij，在文本t_i和视频v_j之间。通过这种方式，它可以将张力从全局锚表示卸载下来。首先，我们在受信任区域约束的情况下通过耦合多元一阶泰勒近似导出了Delta_ij的理想形式，揭示了它作为解决梯度冲突机制的作用，引导沿着局部最优下降方向的更新。 由于直接计算Delta_ij的成本很高，我们引入了一个轻量级神经模块，该模块基于每对视频-文本之间的语义差距进行条件化处理。这使得能够通过梯度监督来进行结构感知校正。为了进一步稳定学习并促进可解释性，我们使用三个部分来规范Delta：一个信任区域约束以防止振荡，一个方向多样性项以促进语义覆盖，并一个信息瓶颈以限制冗余。 在四个检索基准上的实验表明，GARE能够一致地提高对齐准确性和对噪声监督的鲁棒性，证实了感知间隙张力缓解的有效性。
URL
https://arxiv.org/abs/2505.12499
PDF
-
CMAWRNet: Multiple Adverse Weather Removal via a Unified Quaternion Neural Architecture
2025-05-03 18:02:19Vladimir Frants, Sos Agaian, Karen Panetta, Peter HuangAbstract
Images used in real-world applications such as image or video retrieval, outdoor surveillance, and autonomous driving suffer from poor weather conditions. When designing robust computer vision systems, removing adverse weather such as haze, rain, and snow is a significant problem. Recently, deep-learning methods offered a solution for a single type of degradation. Current state-of-the-art universal methods struggle with combinations of degradations, such as haze and rain-streak. Few algorithms have been developed that perform well when presented with images containing multiple adverse weather conditions. This work focuses on developing an efficient solution for multiple adverse weather removal using a unified quaternion neural architecture called CMAWRNet. It is based on a novel texture-structure decomposition block, a novel lightweight encoder-decoder quaternion transformer architecture, and an attentive fusion block with low-light correction. We also introduce a quaternion similarity loss function to preserve color information better. The quantitative and qualitative evaluation of the current state-of-the-art benchmarking datasets and real-world images shows the performance advantages of the proposed CMAWRNet compared to other state-of-the-art weather removal approaches dealing with multiple weather artifacts. Extensive computer simulations validate that CMAWRNet improves the performance of downstream applications such as object detection. This is the first time the decomposition approach has been applied to the universal weather removal task.
Abstract (translated)
在实际应用中，如图像或视频检索、户外监控和自动驾驶等领域使用的图片会受到恶劣天气条件的影响。设计稳健的计算机视觉系统时，去除雾气、雨水和雪等不利天气因素是一个重要的问题。最近，深度学习方法为单一类型的退化提供了解决方案，但目前最先进的通用方法在面对如雾霾与雨迹这样的多种降质组合时仍面临挑战。很少有算法能在处理包含多种恶劣天气条件的图像时表现出色。 本研究专注于开发一种高效的多不良天气去除解决方案，采用名为CMAWRNet（Color and Multi-Adverse Weather Removal Network）的统一四元数神经架构。该方法基于一个新颖的纹理结构分解块、一个轻量级编码器解码器四元数转换器架构以及一个带有低光校正功能的关注融合模块，并引入了一个四元数相似性损失函数，以更好地保留颜色信息。 当前最先进的基准数据集和实际世界图像的定量及定性评估显示，与处理多种天气效果的其他方法相比，所提出的CMAWRNet表现更优。广泛的计算机模拟验证了CMAWRNet能提升下游应用（如物体检测）的表现。这是首次将分解方法应用于通用天气去除任务中。
URL
https://arxiv.org/abs/2505.01882
PDF
