Abstract
Scientific writing is an expert-domain task that demands deep domain knowledge, task-specific requirements and reasoning capabilities that leverage the domain knowledge to satisfy the task specifications. While scientific text generation has been widely studied, its evaluation remains a challenging and open problem. It is critical to develop models that can be reliably deployed for evaluating diverse open-ended scientific writing tasks while adhering to their distinct requirements. However, existing LLM-based judges and reward models are primarily optimized for general-purpose benchmarks with fixed scoring rubrics and evaluation criteria. Consequently, they often fail to reason over sparse knowledge of scientific domains when interpreting task-dependent and multi-faceted criteria. Moreover, fine-tuning for each individual task is costly and impractical for low-resource settings. To bridge these gaps, we propose cost-efficient, open-source reward models tailored for scientific writing evaluation. We introduce a two-stage training framework that initially optimizes scientific evaluation preferences and then refines reasoning capabilities. Our multi-aspect evaluation design and joint training across diverse tasks enable fine-grained assessment and robustness to dynamic criteria and scoring rubrics. Experimental analysis shows that our training regime strongly improves LLM-based scientific writing evaluation. Our models generalize effectively across tasks and to previously unseen scientific writing evaluation settings, allowing a single trained evaluator to be reused without task-specific retraining.
Abstract (translated)
科学写作是一项专业领域任务，需要深厚的专业知识、具体任务的要求以及能够运用专业知识来满足这些要求的推理能力。尽管对科学文本生成的研究已经很广泛，但其评估仍然是一个具有挑战性和尚未解决的问题。开发能够可靠地用于评估各种开放式科学写作任务的模型，同时遵循它们各自的特定要求，这一点至关重要。然而，现有的基于大语言模型（LLM）的评判和奖励模型主要是为带有固定评分标准和评价准则的一般性基准进行优化的。因此，当解释与任务相关的多方面标准时，它们往往难以利用稀疏的专业领域知识进行推理。此外，在每个单独的任务上进行微调对于低资源环境来说成本高昂且不切实际。 为了弥合这些差距，我们提出了一种经济高效、开源的奖励模型，专门用于科学写作评估。我们引入了一个两阶段训练框架，首先优化科学评估偏好，然后细化推理能力。我们的多方面评价设计和跨多种任务的联合训练使细粒度评估成为可能，并增强了对动态标准和评分准则变化的适应性。实验分析表明，我们的训练方案显著提升了基于大语言模型的科学写作评估效果。我们的模型在不同的任务之间以及从未见过的科学写作评估情景中表现良好，使得单一经过训练的评判者能够在不进行特定任务重新训练的情况下复用。
URL
https://arxiv.org/abs/2601.11374
