Abstract
Retrieval-Augmented Generation (RAG) fine-tuning has shown substantial improvements over vanilla RAG, yet most studies target document question answering and often rely on standard NLP metrics that can obscure factual differences. We evaluate RAG fine-tuning for long-form text generation in electronic design automation, adapting a 7B model under five context augmentation strategies with varying retrieval conditions. We introduce TriFEX, a human-validated, triple-based evaluation pipeline that attributes generated claims to their origin-user query, context and reference-and propose Parametric Knowledge Precision (PKP), which isolates internalized knowledge by filtering out claims leaked in the prompt. We show that ROUGE and BERTScore fail to detect factual differences that our triple-based evaluation reveals. Additionally, we demonstrate that an existing metric for knowledge internalization is retrieva-sensitive, with about 75% of its cross-condition variance driven by changes in the rate at which internal knowledge is expressed (PR), rather than by changes in its actual correctness (PKP). The fine-tuned 7B variants outperform a 72B baseline on most metrics, further showing generalization across conditions and on a related benchmark. These results underscore the limitations of available metrics in RAG evaluation and show that smaller models could be reasonably well adapted to specialized tasks for cost-efficient, on-premises deployment.
Abstract (translated)
检索增强生成（RAG）微调相较于原始RAG已展现出显著提升，但现有研究多聚焦于文档问答任务，且常依赖可能掩盖事实差异的标准自然语言处理指标。本研究评估电子设计自动化领域的长文本生成场景下的RAG微调效果，采用五种不同检索条件的上下文增强策略对70亿参数模型进行适配。提出经人工验证的三元组评估流程TriFEX，可将生成声明溯源至原始用户查询、上下文及参考依据；并设计参数化知识精确度（PKP）指标，通过过滤提示中泄露的内容来隔离模型内部知识。研究发现ROUGE与BERTScore无法检测出三元组评估揭示的事实差异，且现有知识内化指标对检索敏感——其跨条件约75%的方差源于内部知识表达率（PR）变化，而非实际正确性（PKP）变化。经微调的70亿参数模型在多数指标上超越720亿参数基线，并在跨条件及关联基准测试中展现泛化能力。这些结果凸显了当前RAG评估指标的局限性，并表明较小模型可通过合理适配在专业任务中实现成本效益更高的本地部署。
URL
https://arxiv.org/abs/2603.23047
