Abstract
Domain specificity of embedding models is critical for effective performance. However, existing benchmarks, such as FinMTEB, are primarily designed for high-resource languages, leaving low-resource settings, such as Korean, under-explored. Directly translating established English benchmarks often fails to capture the linguistic and cultural nuances present in low-resource domains. In this paper, titled TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Models Bring? A Case Study on Korea Financial Texts, we introduce KorFinMTEB, a novel benchmark for the Korean financial domain, specifically tailored to reflect its unique cultural characteristics in low-resource languages. Our experimental results reveal that while the models perform robustly on a translated version of FinMTEB, their performance on KorFinMTEB uncovers subtle yet critical discrepancies, especially in tasks requiring deeper semantic understanding, that underscore the limitations of direct translation. This discrepancy highlights the necessity of benchmarks that incorporate language-specific idiosyncrasies and cultural nuances. The insights from our study advocate for the development of domain-specific evaluation frameworks that can more accurately assess and drive the progress of embedding models in low-resource settings.
Abstract (translated)
领域特定的嵌入模型对于有效性能至关重要。然而，现有的基准测试（如FinMTEB）主要针对高资源语言设计，这导致像韩语这样的低资源环境被忽视。直接将已建立的英语基准翻译到其他语言往往无法捕捉这些语言中的文化和语言细微差别。在题为《TWICE：低资源领域特定嵌入模型能带来哪些优势？韩国金融文本案例研究》的研究中，我们介绍了KorFinMTEB，这是一个针对韩语金融领域的全新基准测试，特别注重反映其独特的文化特性。实验结果显示，虽然这些模型在翻译版本的FinMTEB上表现出色，但在KorFinMTEB上的表现却揭示了细微但关键的区别，特别是在需要更深层次语义理解的任务中。这种差异突显了将语言特定的独特性及文化细微差纳入基准测试的重要性。 我们研究中的见解倡导开发领域特定的评估框架，以更加准确地衡量和推动低资源环境下的嵌入模型进步。
URL
https://arxiv.org/abs/2502.07131
