Abstract
Text embedding models are essential for various natural language processing tasks, enabling the effective encoding of semantic information into dense vector representations. These models are typically optimized using triplets of (query, positive, negative) data pairs for contrastive learning, where the negative samples play a critical role in enhancing the model's ability to discern subtle semantic distinctions. In this work, we introduce a Multi-Granularity Hard-negative (MGH) synthesis framework that leverages large language models (LLMs) to generate diverse negative samples with varying levels of similarity with the query. This approach facilitates a coarse-to-fine curriculum learning strategy during supervised training, allowing the embedding model to progressively learn more nuanced semantic representations. Meanwhile, we propose an Anchor Token Aware (ATA) pooling method that assigns higher weights to anchor tokens based on aggregation patterns observed in LLMs, improving text embedding accuracy without increasing model complexity. Comprehensive experiments on the MTEB benchmark demonstrate that our methods achieve state-of-the-art performance, surpassing existing synthesis strategies both with synthetic data and when combined with public retrieval datasets.
Abstract (translated)
文本嵌入模型对于各种自然语言处理任务至关重要，能够有效地将语义信息编码为密集的向量表示。这些模型通常使用包含（查询、正样本、负样本）的数据三元组进行对比学习来优化，并且其中负样本在提升模型区分细微语义差异的能力方面起着关键作用。在这项工作中，我们引入了一种多粒度难例生成框架（Multi-Granularity Hard-negative, MGH），利用大规模语言模型（LLMs）生成与查询具有不同相似程度的多样化负样本。这种方法促进了监督训练中的从粗到细的教学策略，使嵌入模型能够逐步学习更细腻的语义表示。同时，我们提出了一种基于LLM聚合模式为锚定标记分配更高权重的Anchor Token Aware (ATA) 池化方法，在不增加模型复杂性的情况下提高了文本嵌入精度。在MTEB基准测试中的全面实验表明，我们的方法达到了最先进的性能，并且无论是使用合成数据还是与公共检索数据集结合时，都超过了现有的生成策略。
URL
https://arxiv.org/abs/2509.00842
