Abstract
Multimodal search has become increasingly important in providing users with a natural and effective way to ex-press their search intentions. Images offer fine-grained details of the desired products, while text allows for easily incorporating search modifications. However, some existing multimodal search systems are unreliable and fail to address simple queries. The problem becomes harder with the large variability of natural language text queries, which may contain ambiguous, implicit, and irrelevant in-formation. Addressing these issues may require systems with enhanced matching capabilities, reasoning abilities, and context-aware query parsing and rewriting. This paper introduces a novel multimodal search model that achieves a new performance milestone on the Fashion200K dataset. Additionally, we propose a novel search interface integrating Large Language Models (LLMs) to facilitate natural language interaction. This interface routes queries to search systems while conversationally engaging with users and considering previous searches. When coupled with our multimodal search model, it heralds a new era of shopping assistants capable of offering human-like interaction and enhancing the overall search experience.
Abstract (translated)
多模态搜索已经成为为用户提供自然且有效表达其搜索意图的方法变得越来越重要。图像提供了所需产品的精细细节，而文本允许轻松地包括搜索修改。然而，一些现有的多模态搜索系统不可靠，未能解决简单的查询。随着自然语言文本查询的大幅波动，可能包含模糊、隐含和无关信息，这个问题变得更加严重。解决这些问题可能需要具有增强的匹配能力、推理能力和上下文感知查询解析和重写能力的系统。本文介绍了一种新颖的多模态搜索模型，在Fashion200K数据集上实现了新的性能里程碑。此外，我们提出了一种新颖的搜索接口，整合了大型语言模型（LLMs），以促进自然语言交互。此接口在与用户进行交互并考虑之前搜索的同时将查询路由到搜索系统。与我们的多模态搜索模型结合，它预示着能提供类似人类交互的购物助手的新时代的来临。
URL
https://arxiv.org/abs/2404.15790
