Abstract
Embedding models play a crucial role in representing and retrieving information across various NLP applications. Recent advancements in Large Language Models (LLMs) have further enhanced the performance of embedding models, which are trained on massive amounts of text covering almost every domain. These models are often benchmarked on general-purpose datasets like Massive Text Embedding Benchmark (MTEB), where they demonstrate superior performance. However, a critical question arises: Is the development of domain-specific embedding models necessary when general-purpose models are trained on vast corpora that already include specialized domain texts? In this paper, we empirically investigate this question, choosing the finance domain as an example. We introduce the Finance Massive Text Embedding Benchmark (FinMTEB), a counterpart to MTEB that consists of financial domain-specific text datasets. We evaluate the performance of seven state-of-the-art embedding models on FinMTEB and observe a significant performance drop compared to their performance on MTEB. To account for the possibility that this drop is driven by FinMTEB's higher complexity, we propose four measures to quantify dataset complexity and control for this factor in our analysis. Our analysis provides compelling evidence that state-of-the-art embedding models struggle to capture domain-specific linguistic and semantic patterns, even when trained on large general-purpose corpora. This study sheds light on the necessity of developing domain-specific embedding models in the LLM era, offering valuable insights for researchers and practitioners.
Abstract (translated)
嵌入模型在各种自然语言处理应用中扮演着至关重要的角色，尤其是在大型语言模型（LLMs） recent 进步的加强作用下。这些模型在训练过程中使用大量覆盖几乎所有领域的文本数据，通常在通用数据集上进行比较，表现出优异的性能。然而，一个关键的问题出现了：在训练通用模型以涵盖庞大文本数据集的同时，是否需要开发特定领域的嵌入模型？在本文中，我们通过实证研究来回答这个问题，并选择金融领域作为例子。我们引入了金融百万文本嵌入基准（FinMTEB），这是 MTEB 的一个对应物，由金融领域的特定文本数据组成。我们评估了七个最先进的嵌入模型在 FinMTEB 上的性能，观察到它们的性能明显低于它们在 MTEB 上的性能。为了考虑到 FinMTEB 更高的复杂性，我们提出了四个度量来量化数据集复杂度，并控制这个因素在我们的分析中。我们的分析提供了令人信服的证据，即使训练在大规模通用数据集上，最先进的嵌入模型也无法捕捉到领域特定的语言和语义模式。这项研究阐明了在 LLM 时代开发领域特定嵌入模型的必要性，为研究人员和实践者提供了宝贵的洞见。
URL
https://arxiv.org/abs/2409.18511
