Abstract
With the rapid growth of multi-modal data from social media, short video platforms, and e-commerce, content-based retrieval has become essential for efficiently searching and utilizing heterogeneous information. Over time, retrieval techniques have evolved from Unimodal Retrieval (UR) to Cross-modal Retrieval (CR) and, more recently, to Composed Multi-modal Retrieval (CMR). CMR enables users to retrieve images or videos by integrating a reference visual input with textual modifications, enhancing search flexibility and precision. This paper provides a comprehensive review of CMR, covering its fundamental challenges, technical advancements, and categorization into supervised, zero-shot, and semi-supervised learning paradigms. We discuss key research directions, including data augmentation, model architecture, and loss optimization in supervised CMR, as well as transformation frameworks and external knowledge integration in zero-shot CMR. Additionally, we highlight the application potential of CMR in composed image retrieval, video retrieval, and person retrieval, which have significant implications for e-commerce, online search, and public security. Given its ability to refine and personalize search experiences, CMR is poised to become a pivotal technology in next-generation retrieval systems. A curated list of related works and resources is available at: this https URL
Abstract (translated)
随着来自社交媒体、短视频平台和电子商务的多模态数据快速增长，基于内容的信息检索已成为高效搜索和利用异构信息的关键。随着时间的推移，检索技术从单模态检索（UR）发展到跨模态检索（CR），最近又演进到了组合多模态检索（CMR）。CMR使用户能够通过将参考视觉输入与文本修改相结合来检索图像或视频，从而增强了搜索的灵活性和精度。本文全面回顾了CMR，涵盖了其基本挑战、技术进展以及监督学习、零样本学习和半监督学习范式下的分类。我们讨论了关键的研究方向，包括监督CMR中的数据增强、模型架构和损失优化，以及零样本CMR中的转换框架和外部知识集成。此外，本文还强调了CMR在组合图像检索、视频检索和个人检索方面的应用潜力，这些领域对于电子商务、在线搜索和公共安全具有重要意义。凭借其改善并个性化搜索体验的能力，CMR有望成为下一代检索系统中至关重要的技术。更多相关工作和资源的精选列表可在此网址获得：this https URL
URL
https://arxiv.org/abs/2503.01334
