Abstract
Document chunking is a crucial component of Retrieval-Augmented Generation (RAG), as it directly affects the retrieval of relevant and precise context. Conventional fixed-length and recursive splitters often produce arbitrary, incoherent segments that fail to preserve semantic structure. Although semantic chunking has gained traction, its influence on generation quality remains underexplored. This paper introduces two efficient semantic chunking methods, Projected Similarity Chunking (PSC) and Metric Fusion Chunking (MFC), trained on PubMed data using three different embedding models. We further present an evaluation framework that measures the effect of chunking on both retrieval and generation by augmenting PubMedQA with full-text PubMed Central articles. Our results show substantial retrieval improvements (24x with PSC) in MRR and higher Hits@k on PubMedQA. We provide a comprehensive analysis, including statistical significance and response-time comparisons with common chunking libraries. Despite being trained on a single domain, PSC and MFC also generalize well, achieving strong out-of-domain generation performance across multiple datasets. Overall, our findings confirm that our semantic chunkers, especially PSC, consistently deliver superior performance.
Abstract (translated)
文档分块是检索增强生成（RAG）中的一个关键组件，因为它直接影响到相关和精确上下文的检索。传统的固定长度和递归分割方法通常会产生任意且不连贯的片段，无法保留语义结构。尽管基于语义的分块已逐渐受到关注，但其对生成质量的影响仍缺乏深入研究。本文介绍了两种高效的语义分块方法：投影相似度分块（PSC）和度量融合分块（MFC），这些方法是在PubMed数据上使用三种不同的嵌入模型进行训练的。我们进一步提出了一种评估框架，该框架通过将PubMedQA与全文PubMed Central文章结合来衡量分块对检索和生成的影响。我们的结果表明，在MRR（平均倒数排名）指标上有显著提升（PSC提高了24倍），并在PubMedQA上的Hits@k也有所提高。我们还提供了全面的分析，包括统计显著性和响应时间对比与常见分割库的结果。尽管是在单一领域内进行训练，但PSC和MFC方法也能很好地泛化，在多个数据集上实现了强大的跨域生成性能。总体而言，我们的研究结果证实了我们的语义分块器（特别是PSC）在各种任务中始终如一地表现出色。
URL
https://arxiv.org/abs/2512.00367
