Image_Retrieval
-
Patch-wise Retrieval: A Bag of Practical Techniques for Instance-level Matching
2025-12-14 09:24:51Wonseok Choi, Sohwi Lim, Nam Hyeon-Woo, Moon Ye-Bin, Dong-Ju Jeong, Jinyoung Hwang, Tae-Hyun OhAbstract
Instance-level image retrieval aims to find images containing the same object as a given query, despite variations in size, position, or appearance. To address this challenging task, we propose Patchify, a simple yet effective patch-wise retrieval framework that offers high performance, scalability, and interpretability without requiring fine-tuning. Patchify divides each database image into a small number of structured patches and performs retrieval by comparing these local features with a global query descriptor, enabling accurate and spatially grounded matching. To assess not just retrieval accuracy but also spatial correctness, we introduce LocScore, a localization-aware metric that quantifies whether the retrieved region aligns with the target object. This makes LocScore a valuable diagnostic tool for understanding and improving retrieval behavior. We conduct extensive experiments across multiple benchmarks, backbones, and region selection strategies, showing that Patchify outperforms global methods and complements state-of-the-art reranking pipelines. Furthermore, we apply Product Quantization for efficient large-scale retrieval and highlight the importance of using informative features during compression, which significantly boosts performance. Project website: this https URL
Abstract (translated)
实例级别的图像检索旨在找到包含与给定查询对象相同物体的图片，即使这些图片在大小、位置或外观上有所不同。为了应对这一挑战性任务，我们提出了Patchify，这是一种简单而有效的基于补丁的检索框架，它无需微调即可提供高性能、可扩展性和可解释性。Patchify将每个数据库图像分割成一小部分结构化的补丁，并通过比较这些局部特征与全局查询描述符来进行检索，从而实现准确且空间定位的匹配。 为了评估不仅仅是检索准确性，还包括空间正确性，我们引入了LocScore，这是一种具有定位感知能力的度量标准，用于量化所检索区域是否对准目标对象。这使得LocScore成为一种理解并改进检索行为的重要诊断工具。 我们在多个基准、骨干网络和区域选择策略上进行了广泛的实验，证明Patchify优于全局方法，并且可以补充当前最先进的重排名管道。此外，我们应用产品量化技术进行高效的大型规模检索，并强调在压缩过程中使用信息丰富的特征的重要性，这极大地提升了性能。 项目网站: [请参阅原文链接](this https URL)
URL
https://arxiv.org/abs/2512.12610
PDF
-
Beyond Pixels: A Training-Free, Text-to-Text Framework for Remote Sensing Image Retrieval
2025-12-11 12:43:41J. Xiao, Y. Guo, X. Zi, K. Thiyagarajan, C. Moreira, M. PrasadAbstract
Semantic retrieval of remote sensing (RS) images is a critical task fundamentally challenged by the \textquote{semantic gap}, the discrepancy between a model's low-level visual features and high-level human concepts. While large Vision-Language Models (VLMs) offer a promising path to bridge this gap, existing methods often rely on costly, domain-specific training, and there is a lack of benchmarks to evaluate the practical utility of VLM-generated text in a zero-shot retrieval context. To address this research gap, we introduce the Remote Sensing Rich Text (RSRT) dataset, a new benchmark featuring multiple structured captions per image. Based on this dataset, we propose a fully training-free, text-only retrieval reference called TRSLLaVA. Our methodology reformulates cross-modal retrieval as a text-to-text (T2T) matching problem, leveraging rich text descriptions as queries against a database of VLM-generated captions within a unified textual embedding space. This approach completely bypasses model training or fine-tuning. Experiments on the RSITMD and RSICD benchmarks show our training-free method is highly competitive with state-of-the-art supervised models. For instance, on RSITMD, our method achieves a mean Recall of 42.62\%, nearly doubling the 23.86\% of the standard zero-shot CLIP baseline and surpassing several top supervised models. This validates that high-quality semantic representation through structured text provides a powerful and cost-effective paradigm for remote sensing image retrieval.
Abstract (translated)
遥感（RS）图像的语义检索是一项关键任务，该任务主要受到所谓的“语义鸿沟”的挑战，即模型的低级视觉特征与高级人类概念之间的差异。虽然大规模的视觉语言模型（VLMs）为弥合这一差距提供了有希望的方法，但现有方法通常依赖于昂贵且特定领域的训练，并缺乏评估零样本检索上下文中VLM生成文本的实际效用基准。为了填补这一研究空白，我们引入了“遥感丰富文本”（RSRT）数据集，这是一个新基准，该数据集中每张图像包含多个结构化的描述文本。基于此数据集，我们提出了一个完全不需要训练的纯文本检索参考方法TRSLLaVA。我们的方法将跨模态检索重新表述为文本到文本（T2T）匹配问题，在统一的文本嵌入空间中利用丰富的文本描述作为查询，针对VLM生成的图像描述数据库进行匹配。这种方法彻底避免了模型训练或微调的需求。 在RSITMD和RSICD基准测试上的实验表明，我们无需训练的方法与最先进的监督模型具有高度竞争力。例如，在RSITMD上，我们的方法实现了平均召回率为42.62%，几乎是标准零样本CLIP基线的两倍（23.86%），并且超过了几个顶级的监督模型。这验证了通过结构化文本提供高质量语义表示为遥感图像检索提供了一种强大且成本效益高的范例。
URL
https://arxiv.org/abs/2512.10596
PDF
-
Language-driven Fine-grained Retrieval
2025-12-06 02:56:55Shijie Wang, Xin Yu, Yadan Luo, Zijian Wang, Pengfei Zhang, Zi HuangAbstract
Existing fine-grained image retrieval (FGIR) methods learn discriminative embeddings by adopting semantically sparse one-hot labels derived from category names as supervision. While effective on seen classes, such supervision overlooks the rich semantics encoded in category names, hindering the modeling of comparability among cross-category details and, in turn, limiting generalization to unseen categories. To tackle this, we introduce LaFG, a Language-driven framework for Fine-Grained Retrieval that converts class names into attribute-level supervision using large language models (LLMs) and vision-language models (VLMs). Treating each name as a semantic anchor, LaFG prompts an LLM to generate detailed, attribute-oriented descriptions. To mitigate attribute omission in these descriptions, it leverages a frozen VLM to project them into a vision-aligned space, clustering them into a dataset-wide attribute vocabulary while harvesting complementary attributes from related categories. Leveraging this vocabulary, a global prompt template selects category-relevant attributes, which are aggregated into category-specific linguistic prototypes. These prototypes supervise the retrieval model to steer
Abstract (translated)
现有的细粒度图像检索（FGIR）方法通过采用从类别名称衍生出的语义稀疏的一_hot标签作为监督来学习区分性嵌入。虽然在已见过的类上有效，但这种监督忽略了类别名称中包含的丰富语义信息，从而阻碍了跨类别细节可比性的建模，并进一步限制了对未见过类别的泛化能力。为了解决这一问题，我们引入了一种名为LaFG的语言驱动细粒度检索框架。该框架利用大型语言模型（LLMs）和视觉-语言模型（VLMs），将类别名称转换为属性级别的监督信息。 具体来说，LaFG将每个类别名称视为语义锚点，并提示一个LLM生成详细的、以属性为导向的描述。为了减轻这些描述中可能出现的属性遗漏问题，它利用了一个冻结的VLM将其投影到与视觉对齐的空间中，在此基础上进行聚类，形成整个数据集范围内的属性词汇表并收集相关类别中的互补属性。 基于此属性词汇表，LaFG使用一个全局提示模板选择出相关的属性，并将这些属性聚合为特定于每个类别的语言原型。这些原型作为监督信息引导检索模型，使其能够在细粒度图像检索任务中更好地泛化和区分不同类别的细节。
URL
https://arxiv.org/abs/2512.06255
PDF
-
HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval
2025-12-02 14:10:16Zhiwei Chen, Yupeng Hu, Zixu Li, Zhiheng Fu, Haokun Wen, Weili GuanAbstract
Composed Video Retrieval (CVR) is a challenging video retrieval task that utilizes multi-modal queries, consisting of a reference video and modification text, to retrieve the desired target video. The core of this task lies in understanding the multi-modal composed query and achieving accurate composed feature learning. Within multi-modal queries, the video modality typically carries richer semantic content compared to the textual modality. However, previous works have largely overlooked the disparity in information density between these two modalities. This limitation can lead to two critical issues: 1) modification subject referring ambiguity and 2) limited detailed semantic focus, both of which degrade the performance of CVR models. To address the aforementioned issues, we propose a novel CVR framework, namely the Hierarchical Uncertainty-aware Disambiguation network (HUD). HUD is the first framework that leverages the disparity in information density between video and text to enhance multi-modal query understanding. It comprises three key components: (a) Holistic Pronoun Disambiguation, (b) Atomistic Uncertainty Modeling, and (c) Holistic-to-Atomistic Alignment. By exploiting overlapping semantics through holistic cross-modal interaction and fine-grained semantic alignment via atomistic-level cross-modal interaction, HUD enables effective object disambiguation and enhances the focus on detailed semantics, thereby achieving precise composed feature learning. Moreover, our proposed HUD is also applicable to the Composed Image Retrieval (CIR) task and achieves state-of-the-art performance across three benchmark datasets for both CVR and CIR tasks. The codes are available on this https URL.
Abstract (translated)
组成视频检索（Composed Video Retrieval，CVR）是一项具有挑战性的视频检索任务，它利用包含参考视频和修改文本的多模态查询来检索所需的靶标视频。这项任务的核心在于理解多模态组合查询并实现准确的组合特征学习。在多模态查询中，视频模式通常携带比文本模式更丰富的语义内容。然而，先前的工作大多忽视了这两种模式之间信息密度差异的问题。这一限制可能导致两个关键问题：1）修改对象指代模糊性；2）有限的具体语义关注点，这些问题都会降低CVR模型的性能。 为解决上述问题，我们提出了一种新颖的CVR框架——分层不确定性感知消歧网络（Hierarchical Uncertainty-aware Disambiguation network，HUD）。HUD是第一个利用视频和文本信息密度差异来增强多模态查询理解的框架。它包含三个关键组成部分：(a) 全局代词消歧；(b) 原子级不确定性建模；(c) 全局到原子级对齐。通过全局跨模式交互利用重叠语义，并通过原子级跨模式交互实现精细语义对齐，HUD能够有效进行对象消歧并增强对具体语义的关注点，从而实现精确的组合特征学习。 此外，我们提出的HUD框架也适用于组成图像检索（Composed Image Retrieval，CIR）任务，在三个基准数据集上均取得了在CVR和CIR任务上的最佳性能。相关代码可在此链接访问：[https URL]。
URL
https://arxiv.org/abs/2512.02792
PDF
-
GeoBridge: A Semantic-Anchored Multi-View Foundation Model Bridging Images and Text for Geo-Localization
2025-12-02 12:28:22Zixuan Song, Jing Zhang, Di Wang, Zidie Zhou, Wenbin Liu, Haonan Guo, En Wang, Bo DuAbstract
Cross-view geo-localization infers a location by retrieving geo-tagged reference images that visually correspond to a query image. However, the traditional satellite-centric paradigm limits robustness when high-resolution or up-to-date satellite imagery is unavailable. It further underexploits complementary cues across views (e.g., drone, satellite, and street) and modalities (e.g., language and image). To address these challenges, we propose GeoBridge, a foundation model that performs bidirectional matching across views and supports language-to-image retrieval. Going beyond traditional satellite-centric formulations, GeoBridge builds on a novel semantic-anchor mechanism that bridges multi-view features through textual descriptions for robust, flexible localization. In support of this task, we construct GeoLoc, the first large-scale, cross-modal, and multi-view aligned dataset comprising over 50,000 pairs of drone, street-view panorama, and satellite images as well as their textual descriptions, collected from 36 countries, ensuring both geographic and semantic alignment. We performed broad evaluations across multiple tasks. Experiments confirm that GeoLoc pre-training markedly improves geo-location accuracy for GeoBridge while promoting cross-domain generalization and cross-modal knowledge transfer. The dataset, source code, and pretrained models were released at this https URL.
Abstract (translated)
跨视角地理定位通过检索与查询图像视觉上对应的地理标记参考图像来推断位置。然而，传统的以卫星为中心的范式在高分辨率或最新卫星影像不可用时会限制其鲁棒性，并且未能充分利用不同视角（如无人机、卫星和街景）以及模态（如语言和图像）之间的互补线索。为了解决这些挑战，我们提出了GeoBridge，这是一种基础模型，它可以在不同的视图之间进行双向匹配并支持从语言到图像的检索。超越传统的以卫星为中心的形式化方法，GeoBridge建立了一种新颖的语义锚机制，通过文本描述将多视角特征联系起来，从而实现稳健和灵活的位置定位。 为了支持这一任务，我们构建了GeoLoc，这是首个大规模、跨模态、多视图对齐的数据集，包含来自36个国家超过50,000对无人机、街景全景和卫星图像及其文本描述，确保地理和语义上的对齐。我们在多个任务上进行了广泛的评估。实验结果确认了GeoLoc预训练显著提高了GeoBridge的地理定位准确性，并促进了跨域泛化和跨模态知识转移。 数据集、源代码以及预训练模型可在[该网址]（请将URL替换为实际链接）获取。
URL
https://arxiv.org/abs/2512.02697
PDF
-
Generative Editing in the Joint Vision-Language Space for Zero-Shot Composed Image Retrieval
2025-12-01 13:04:55Xin Wang, Haipeng Zhang, Mang Li, Zhaohui Xia, Yueguo Chen, Yu Zhang, Chunyu WeiAbstract
Composed Image Retrieval (CIR) enables fine-grained visual search by combining a reference image with a textual modification. While supervised CIR methods achieve high accuracy, their reliance on costly triplet annotations motivates zero-shot solutions. The core challenge in zero-shot CIR (ZS-CIR) stems from a fundamental dilemma: existing text-centric or diffusion-based approaches struggle to effectively bridge the vision-language modality gap. To address this, we propose Fusion-Diff, a novel generative editing framework with high effectiveness and data efficiency designed for multimodal alignment. First, it introduces a multimodal fusion feature editing strategy within a joint vision-language (VL) space, substantially narrowing the modality gap. Second, to maximize data efficiency, the framework incorporates a lightweight Control-Adapter, enabling state-of-the-art performance through fine-tuning on only a limited-scale synthetic dataset of 200K samples. Extensive experiments on standard CIR benchmarks (CIRR, FashionIQ, and CIRCO) demonstrate that Fusion-Diff significantly outperforms prior zero-shot approaches. We further enhance the interpretability of our model by visualizing the fused multimodal representations.
Abstract (translated)
组合图像检索（Composed Image Retrieval，CIR）通过结合参考图像和文本修改来实现细粒度的视觉搜索。尽管监督式CIR方法能够达到高精度，但其依赖于成本高昂的三元组标注促使了零样本解决方案的发展。零样本CIR (ZS-CIR)的核心挑战在于一个基本困境：现有的以文本为中心或基于扩散的方法难以有效地弥合视觉与语言模态之间的差距。为此，我们提出了一种名为Fusion-Diff的新颖生成编辑框架，该框架旨在实现多模式对齐，并具备高效性和数据效率。 首先，Fusion-Diff在联合的视觉-语言（VL）空间内引入了多模式融合特征编辑策略，大大缩小了模态差距。其次，为了最大化数据效率，该框架整合了一个轻量级的Control-Adapter，在仅使用有限规模的20万样本合成数据集进行微调的情况下就能实现最先进的性能。 在标准CIR基准测试（如CIRR、FashionIQ和CIRCO）上的广泛实验表明，Fusion-Diff显著优于先前的零样本方法。为了增强模型的可解释性，我们通过可视化融合后的多模态表示进一步改进了该框架。
URL
https://arxiv.org/abs/2512.01636
PDF
-
UNION: A Lightweight Target Representation for Efficient Zero-Shot Image-Guided Retrieval with Optional Textual Queries
2025-11-27 09:28:28Hoang-Bao Le, Allie Tran, Binh T. Nguyen, Liting Zhou, Cathal GurrinAbstract
Image-Guided Retrieval with Optional Text (IGROT) is a general retrieval setting where a query consists of an anchor image, with or without accompanying text, aiming to retrieve semantically relevant target images. This formulation unifies two major tasks: Composed Image Retrieval (CIR) and Sketch-Based Image Retrieval (SBIR). In this work, we address IGROT under low-data supervision by introducing UNION, a lightweight and generalisable target representation that fuses the image embedding with a null-text prompt. Unlike traditional approaches that rely on fixed target features, UNION enhances semantic alignment with multimodal queries while requiring no architectural modifications to pretrained vision-language models. With only 5,000 training samples - from LlavaSCo for CIR and Training-Sketchy for SBIR - our method achieves competitive results across benchmarks, including CIRCO mAP@50 of 38.5 and Sketchy mAP@200 of 82.7, surpassing many heavily supervised baselines. This demonstrates the robustness and efficiency of UNION in bridging vision and language across diverse query types.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22253
PDF
-
New York Smells: A Large Multimodal Dataset for Olfaction
2025-11-25 17:44:50Ege Ozguroglu, Junbang Liang, Ruoshi Liu, Mia Chiquier, Michael DeTienne, Wesley Wei Qian, Alexandra Horowitz, Andrew Owens, Carl VondrickAbstract
While olfaction is central to how animals perceive the world, this rich chemical sensory modality remains largely inaccessible to machines. One key bottleneck is the lack of diverse, multimodal olfactory training data collected in natural settings. We present New York Smells, a large dataset of paired image and olfactory signals captured ``in the wild.'' Our dataset contains 7,000 smell-image pairs from 3,500 distinct objects across indoor and outdoor environments, with approximately 70$\times$ more objects than existing olfactory datasets. Our benchmark has three tasks: cross-modal smell-to-image retrieval, recognizing scenes, objects, and materials from smell alone, and fine-grained discrimination between grass species. Through experiments on our dataset, we find that visual data enables cross-modal olfactory representation learning, and that our learned olfactory representations outperform widely-used hand-crafted features.
Abstract (translated)
URL
https://arxiv.org/abs/2511.20544
PDF
-
ScenarioCLIP: Pretrained Transferable Visual Language Models and Action-Genome Dataset for Natural Scene Analysis
2025-11-25 12:59:31Advik Sinha, Saurabh Atreya, Aashutosh A V, Sk Aziz Ali, Abhijit DasAbstract
Until recently, the general corpus of CLIP-type fundamental models has widely explored either the retrieval of short descriptions or the classification of objects in the scene as SINGLE-object image classification task. The same holds for retrieving the image embedding (image retrieval task) given a text prompt. However, real-world scene images exhibit rich compositional structure involving multiple objects and actions. The latest methods in the CLIP-based literature improve class-level discrimination by mining harder negative image-text pairs and by refining permanent text prompts, often using LLMs. However, these improvements remain confined to predefined class lists and do not explicitly model relational or compositional structure. PyramidCLIP partially addresses this gap by aligning global and local visual features, yet it still lacks explicit modeling of inter-object relations. Hence, to further leverage this aspect for scene analysis, the proposed ScenarioCLIP model accepts input texts, grounded relations, and input images, along with focused regions highlighting relations. The proposed model is pretrained on curated scenario data, and finetuned for specialized downstream tasks, such as cross-modal retrieval and fine-grained visual understanding tasks. To address the lack of domain-specific datasets, we generate a novel dataset by extending image-text pairs from existing diverse indoor and outdoor scenario datasets that are publicly available. We used a pipeline of existing language models to ground action, object, and relations, filled by manual and automatic curation. We established a comprehensive benchmark for several scenario-based tasks and compared it with many baseline methods. ScenarioCLIP demonstrates robust zero-shot and finetune performance on various domain-specific tasks. Our code and dataset are available at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2511.20274
PDF
-
Intelligent Image Search Algorithms Fusing Visual Large Models
2025-11-25 04:54:49Kehan Wang, Tingqiong Cui, Yang Zhang, Yu Chen, Shifeng Wu, Zhenzhang LiAbstract
Fine-grained image retrieval, which aims to find images containing specific object components and assess their detailed states, is critical in fields like security and industrial inspection. However, conventional methods face significant limitations: manual features (e.g., SIFT) lack robustness; deep learning-based detectors (e.g., YOLO) can identify component presence but cannot perform state-specific retrieval or zero-shot search; Visual Large Models (VLMs) offer semantic and zero-shot capabilities but suffer from poor spatial grounding and high computational cost, making them inefficient for direct retrieval. To bridge these gaps, this paper proposes DetVLM, a novel intelligent image search framework that synergistically fuses object detection with VLMs. The framework pioneers a search-enhancement paradigm via a two-stage pipeline: a YOLO detector first conducts efficient, high-recall component-level screening to determine component presence; then, a VLM acts as a recall-enhancement unit, performing secondary verification for components missed by the detector. This architecture directly enables two advanced capabilities: 1) State Search: Guided by task-specific prompts, the VLM refines results by verifying component existence and executing sophisticated state judgments (e.g., "sun visor lowered"), allowing retrieval based on component state. 2) Zero-shot Search: The framework leverages the VLM's inherent zero-shot capability to recognize and retrieve images containing unseen components or attributes (e.g., "driver wearing a mask") without any task-specific training. Experiments on a vehicle component dataset show DetVLM achieves a state-of-the-art overall retrieval accuracy of 94.82\%, significantly outperforming detection-only baselines. It also attains 94.95\% accuracy in zero-shot search for driver mask-wearing and over 90\% average accuracy in state search tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2511.19920
PDF
-
Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Models
2025-11-22 17:04:30Dachuan Zhao, Weiyue Li, Zhenda Shen, Yushu Qiu, Bowen Xu, Haoyu Chen, Yongchao ChenAbstract
Vision-Language Models (VLMs) have become indispensable for multimodal reasoning, yet their representations often encode and amplify demographic biases, resulting in biased associations and misaligned predictions in downstream tasks. Such behavior undermines fairness and distorts the intended alignment between vision and language. Recent post-hoc approaches attempt to mitigate bias by replacing the most attribute-correlated embedding coordinates with neutral values. However, our systematic analysis reveals three critical failures of this coordinate-wise approach: feature entanglement, poor cross-dataset generalization, and incomplete bias removal. We find that bias is not localized to a few coordinates but is instead distributed across a few linear subspaces. To address these limitations, we propose $\textbf{S}$ubspace $\textbf{P}$rojection $\textbf{D}$ebiasing ($\textbf{SPD}$), a geometrically principled framework that identifies and removes the entire subspace of linearly decodable bias while reinserting a neutral mean component to preserve semantic fidelity. Extensive experiments across zero-shot classification, text-to-image retrieval, and image generation validate the effectiveness of SPD: our method achieves more robust debiasing with an average improvement of $18.5\%$ across four fairness metrics, while maintaining minimal loss in task performance compared to the best debiasing baseline.
Abstract (translated)
URL
https://arxiv.org/abs/2511.18123
PDF
-
A Little More Like This: Text-to-Image Retrieval with Vision-Language Models Using Relevance Feedback
2025-11-21 14:01:36Bulat Khaertdinov, Mirela Popa, Nava TintarevAbstract
Large vision-language models (VLMs) enable intuitive visual search using natural language queries. However, improving their performance often requires fine-tuning and scaling to larger model variants. In this work, we propose a mechanism inspired by traditional text-based search to improve retrieval performance at inference time: relevance feedback. While relevance feedback can serve as an alternative to fine-tuning, its model-agnostic design also enables use with fine-tuned VLMs. Specifically, we introduce and evaluate four feedback strategies for VLM-based retrieval. First, we revise classical pseudo-relevance feedback (PRF), which refines query embeddings based on top-ranked results. To address its limitations, we propose generative relevance feedback (GRF), which uses synthetic captions for query refinement. Furthermore, we introduce an attentive feedback summarizer (AFS), a custom transformer-based model that integrates multimodal fine-grained features from relevant items. Finally, we simulate explicit feedback using ground-truth captions as an upper-bound baseline. Experiments on Flickr30k and COCO with the VLM backbones show that GRF, AFS, and explicit feedback improve retrieval performance by 3-5% in MRR@5 for smaller VLMs, and 1-3% for larger ones, compared to retrieval with no feedback. Moreover, AFS, similarly to explicit feedback, mitigates query drift and is more robust than GRF in iterative, multi-turn retrieval settings. Our findings demonstrate that relevance feedback can consistently enhance retrieval across VLMs and open up opportunities for interactive and adaptive visual search.
Abstract (translated)
URL
https://arxiv.org/abs/2511.17255
PDF
-
DIR-TIR: Dialog-Iterative Refinement for Text-to-Image Retrieval
Abstract
This paper addresses the task of interactive, conversational text-to-image retrieval. Our DIR-TIR framework progressively refines the target image search through two specialized modules: the Dialog Refiner Module and the Image Refiner Module. The Dialog Refiner actively queries users to extract essential information and generate increasingly precise descriptions of the target image. Complementarily, the Image Refiner identifies perceptual gaps between generated images and user intentions, strategically reducing the visual-semantic discrepancy. By leveraging multi-turn dialogues, DIR-TIR provides superior controllability and fault tolerance compared to conventional single-query methods, significantly improving target image hit accuracy. Comprehensive experiments across diverse image datasets demonstrate our dialogue-based approach substantially outperforms initial-description-only baselines, while the synergistic module integration achieves both higher retrieval precision and enhanced interactive experience.
Abstract (translated)
URL
https://arxiv.org/abs/2511.14449
PDF
-
O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model
2025-11-18 11:18:08Rishi Gupta, Mukilan Karuppasamy, Shyam Marjit, Aditay Tripathi, Anirban ChakrabortyAbstract
While Large Vision Language Models (LVLMs) are increasingly deployed in real-world applications, their ability to interpret abstract visual inputs remains limited. Specifically, they struggle to comprehend hand-drawn sketches, a modality that offers an intuitive means of expressing concepts that are difficult to describe textually. We identify the primary bottleneck as the absence of a large-scale dataset that jointly models sketches, photorealistic images, and corresponding natural language instructions. To address this, we present two key contributions: (1) a new, large-scale dataset of image-sketch-instruction triplets designed to facilitate both pretraining and instruction tuning, and (2) O3SLM, an LVLM trained on this dataset. Comprehensive evaluations on multiple sketch-based tasks: (a) object localization, (b) counting, (c) image retrieval i.e., (SBIR and fine-grained SBIR), and (d) visual question answering (VQA); while incorporating the three existing sketch datasets, namely QuickDraw!, Sketchy, and Tu Berlin, along with our generated SketchVCL dataset, show that O3SLM achieves state-of-the-art performance, substantially outperforming existing LVLMs in sketch comprehension and reasoning.
Abstract (translated)
URL
https://arxiv.org/abs/2511.14368
PDF
-
THIR: Topological Histopathological Image Retrieval
2025-11-17 09:18:54Zahra Tabatabaei, Jon SporringAbstract
According to the World Health Organization, breast cancer claimed the lives of approximately 685,000 women in 2020. Early diagnosis and accurate clinical decision making are critical in reducing this global burden. In this study, we propose THIR, a novel Content-Based Medical Image Retrieval (CBMIR) framework that leverages topological data analysis specifically, Betti numbers derived from persistent homology to characterize and retrieve histopathological images based on their intrinsic structural patterns. Unlike conventional deep learning approaches that rely on extensive training, annotated datasets, and powerful GPU resources, THIR operates entirely without supervision. It extracts topological fingerprints directly from RGB histopathological images using cubical persistence, encoding the evolution of loops as compact, interpretable feature vectors. The similarity retrieval is then performed by computing the distances between these topological descriptors, efficiently returning the top-K most relevant matches. Extensive experiments on the BreaKHis dataset demonstrate that THIR outperforms state of the art supervised and unsupervised methods. It processes the entire dataset in under 20 minutes on a standard CPU, offering a fast, scalable, and training free solution for clinical image retrieval.
Abstract (translated)
URL
https://arxiv.org/abs/2511.13170
PDF
-
Visible Structure Retrieval for Lightweight Image-Based Relocalisation
2025-11-16 08:32:18Fereidoon Zangeneh, Leonard Bruns, Amit Dekel, Alessandro Pieropan, Patric JensfeltAbstract
Accurate camera pose estimation from an image observation in a previously mapped environment is commonly done through structure-based methods: by finding correspondences between 2D keypoints on the image and 3D structure points in the map. In order to make this correspondence search tractable in large scenes, existing pipelines either rely on search heuristics, or perform image retrieval to reduce the search space by comparing the current image to a database of past observations. However, these approaches result in elaborate pipelines or storage requirements that grow with the number of past observations. In this work, we propose a new paradigm for making structure-based relocalisation tractable. Instead of relying on image retrieval or search heuristics, we learn a direct mapping from image observations to the visible scene structure in a compact neural network. Given a query image, a forward pass through our novel visible structure retrieval network allows obtaining the subset of 3D structure points in the map that the image views, thus reducing the search space of 2D-3D correspondences. We show that our proposed method enables performing localisation with an accuracy comparable to the state of the art, while requiring lower computational and storage footprint.
Abstract (translated)
URL
https://arxiv.org/abs/2511.12503
PDF
-
Shrinking the Teacher: An Adaptive Teaching Paradigm for Asymmetric EEG-Vision Alignment
2025-11-14 15:52:00Lukun Wu, Jie Li, Ziqi Ren, Kaifan Zhang, Xinbo GaoAbstract
Decoding visual features from EEG signals is a central challenge in neuroscience, with cross-modal alignment as the dominant approach. We argue that the relationship between visual and brain modalities is fundamentally asymmetric, characterized by two critical gaps: a Fidelity Gap (stemming from EEG's inherent noise and signal degradation, vs. vision's high-fidelity features) and a Semantic Gap (arising from EEG's shallow conceptual representation, vs. vision's rich semantic depth). Previous methods often overlook this asymmetry, forcing alignment between the two modalities as if they were equal partners and thereby leading to poor generalization. To address this, we propose the adaptive teaching paradigm. This paradigm empowers the ``teacher" modality (vision) to dynamically shrink and adjust its knowledge structure under task guidance, tailoring its semantically dense features to match the ``student" modality (EEG)'s capacity. We implement this paradigm with the ShrinkAdapter, a simple yet effective module featuring a residual-free design and a bottleneck structure. Through extensive experiments, we validate the underlying rationale and effectiveness of our paradigm. Our method achieves a top-1 accuracy of 60.2\% on the zero-shot brain-to-image retrieval task, surpassing previous state-of-the-art methods by a margin of 9.8\%. Our work introduces a new perspective for asymmetric alignment: the teacher must shrink and adapt to bridge the vision-brain gap.
Abstract (translated)
URL
https://arxiv.org/abs/2511.11422
PDF
-
From Street to Orbit: Training-Free Cross-View Retrieval via Location Semantics and LLM Guidance
2025-11-12 23:51:46Jeongho Min, Dongyoung Kim, Jaehyup LeeAbstract
Cross-view image retrieval, particularly street-to-satellite matching, is a critical task for applications such as autonomous navigation, urban planning, and localization in GPS-denied environments. However, existing approaches often require supervised training on curated datasets and rely on panoramic or UAV-based images, which limits real-world deployment. In this paper, we present a simple yet effective cross-view image retrieval framework that leverages a pretrained vision encoder and a large language model (LLM), requiring no additional training. Given a monocular street-view image, our method extracts geographic cues through web-based image search and LLM-based location inference, generates a satellite query via geocoding API, and retrieves matching tiles using a pretrained vision encoder (e.g., DINOv2) with PCA-based whitening feature refinement. Despite using no ground-truth supervision or finetuning, our proposed method outperforms prior learning-based approaches on the benchmark dataset under zero-shot settings. Moreover, our pipeline enables automatic construction of semantically aligned street-to-satellite datasets, which is offering a scalable and cost-efficient alternative to manual annotation. All source codes will be made publicly available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.09820
PDF
-
DiffusionUavLoc: Visually Prompted Diffusion for Cross-View UAV Localization
2025-11-09 15:27:17Tao Liu, Kan Ren, Qian ChenAbstract
With the rapid growth of the low-altitude economy, unmanned aerial vehicles (UAVs) have become key platforms for measurement and tracking in intelligent patrol systems. However, in GNSS-denied environments, localization schemes that rely solely on satellite signals are prone to failure. Cross-view image retrieval-based localization is a promising alternative, yet substantial geometric and appearance domain gaps exist between oblique UAV views and nadir satellite orthophotos. Moreover, conventional approaches often depend on complex network architectures, text prompts, or large amounts of annotation, which hinders generalization. To address these issues, we propose DiffusionUavLoc, a cross-view localization framework that is image-prompted, text-free, diffusion-centric, and employs a VAE for unified representation. We first use training-free geometric rendering to synthesize pseudo-satellite images from UAV imagery as structural prompts. We then design a text-free conditional diffusion model that fuses multimodal structural cues to learn features robust to viewpoint changes. At inference, descriptors are computed at a fixed time step t and compared using cosine similarity. On University-1652 and SUES-200, the method performs competitively for cross-view localization, especially for satellite-to-drone in this http URL data and code will be published at the following URL: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.06422
PDF
-
Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era
2025-11-08 14:35:11Feng Lu, Tong Jin, Canming Ye, Yunpeng Liu, Xiangyuan Lan, Chun YuanAbstract
Visual place recognition (VPR) is typically regarded as a specific image retrieval task, whose core lies in representing images as global descriptors. Over the past decade, dominant VPR methods (e.g., NetVLAD) have followed a paradigm that first extracts the patch features/tokens of the input image using a backbone, and then aggregates these patch features into a global descriptor via an aggregator. This backbone-plus-aggregator paradigm has achieved overwhelming dominance in the CNN era and remains widely used in transformer-based models. In this paper, however, we argue that a dedicated aggregator is not necessary in the transformer era, that is, we can obtain robust global descriptors only with the backbone. Specifically, we introduce some learnable aggregation tokens, which are prepended to the patch tokens before a particular transformer block. All these tokens will be jointly processed and interact globally via the intrinsic self-attention mechanism, implicitly aggregating useful information within the patch tokens to the aggregation tokens. Finally, we only take these aggregation tokens from the last output tokens and concatenate them as the global representation. Although implicit aggregation can provide robust global descriptors in an extremely simple manner, where and how to insert additional tokens, as well as the initialization of tokens, remains an open issue worthy of further exploration. To this end, we also propose the optimal token insertion strategy and token initialization method derived from empirical studies. Experimental results show that our method outperforms state-of-the-art methods on several VPR datasets with higher efficiency and ranks 1st on the MSLS challenge leaderboard. The code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.06024
PDF
