Abstract
We introduce ModaRoute, an LLM-based intelligent routing system that dynamically selects optimal modalities for multimodal video retrieval. While dense text captions can achieve 75.9% Recall@5, they require expensive offline processing and miss critical visual information present in 34% of clips with scene text not captured by ASR. By analyzing query intent and predicting information needs, ModaRoute reduces computational overhead by 41% while achieving 60.9% Recall@5. Our approach uses GPT-4.1 to route queries across ASR (speech), OCR (text), and visual indices, averaging 1.78 modalities per query versus exhaustive 3.0 modality search. Evaluation on 1.8M video clips demonstrates that intelligent routing provides a practical solution for scaling multimodal retrieval systems, reducing infrastructure costs while maintaining competitive effectiveness for real-world deployment.
Abstract (translated)
我们介绍了一种基于大型语言模型（LLM）的智能路由系统——ModaRoute，该系统能够动态选择最佳模式来进行多模态视频检索。尽管密集的文字字幕可以达到75.9%的召回率@5指标，但它们需要昂贵的离线处理，并且会错过34%包含场景文本的片段中的关键视觉信息（这些信息未被自动语音识别（ASR）捕捉到）。通过分析查询意图并预测信息需求，ModaRoute在减少计算开销41%的同时实现了60.9%的召回率@5。我们的方法使用GPT-4.1将查询路由到ASR（语音）、OCR（文本）和视觉索引之间，在每个查询中平均使用了1.78种模式，而全面搜索则需要3.0种模式。在对180万视频片段进行的评估显示，智能路由为扩展多模态检索系统提供了实际解决方案，能够降低基础设施成本并保持现实世界部署中的竞争力效果。
URL
https://arxiv.org/abs/2507.13374
