Abstract
Retrieval-Augmented Generation (RAG) often struggles with knowledge conflicts, where model-internal parametric knowledge overrides retrieved evidence, leading to unfaithful outputs. Existing approaches are often limited, relying either on superficial decoding adjustments or weight editing that necessitates ground-truth targets. Through layer-wise analysis, we attribute this failure to a parametric suppression phenomenon: specifically, in deep layers, certain FFN layers overwrite context-sensitive representations with memorized priors. To address this, we propose CoRect (Context-Aware Logit Contrast for Hidden State Rectification). By contrasting logits from contextualized and non-contextualized forward passes, CoRect identifies layers that exhibit high parametric bias without requiring ground-truth labels. It then rectifies the hidden states to preserve evidence-grounded information. Across question answering (QA) and summarization benchmarks, CoRect consistently improves faithfulness and reduces hallucinations compared to strong baselines.
Abstract (translated)
基于检索的生成（Retrieval-Augmented Generation，RAG）常常在知识冲突中遇到困难，在这种情况下，模型内部参数化的知识会覆盖从外部检索到的证据，导致输出不准确。现有方法通常有限，要么依赖于浅层解码调整，要么依靠需要真实标签的权重编辑来解决问题。通过逐层分析，我们将这一失败归因于一个参数抑制现象：具体来说，在深层中，某些全连接网络（FFN）层会用记忆化的先验覆盖上下文敏感表示。 为了应对这个问题，我们提出了CoRect（Context-Aware Logit Contrast for Hidden State Rectification），即具有上下文感知的逻辑对比用于隐藏状态校正。通过比较有上下文和无上下文前向传递中的逻辑值，CoRect能够识别那些表现出高参数偏差的层，并且无需真实标签即可进行这一过程。然后它会修正这些隐藏状态以保留基于证据的信息。 在问题回答（QA）和摘要生成基准上，与强大的基线方法相比，CoRect在提高输出准确性、减少幻觉方面表现出了持续改进的效果。
URL
https://arxiv.org/abs/2602.08221
