Abstract
Reinforcement Learning with Human Feedback (RLHF) has proven effective in image generation field guided by reward models to align human preferences. Motivated by this, adapting RLHF for Image Super-Resolution (ISR) tasks has shown promise in optimizing perceptual quality with Image Quality Assessment (IQA) model as reward models. However, the traditional IQA model usually output a single global score, which are exceptionally insensitive to local and fine-grained distortions. This insensitivity allows ISR models to produce perceptually undesirable artifacts that yield spurious high scores, misaligning optimization objectives with perceptual quality and results in reward hacking. To address this, we propose a Fine-grained Perceptual Reward Model (FinPercep-RM) based on an Encoder-Decoder architecture. While providing a global quality score, it also generates a Perceptual Degradation Map that spatially localizes and quantifies local defects. We specifically introduce the FGR-30k dataset to train this model, consisting of diverse and subtle distortions from real-world super-resolution models. Despite the success of the FinPercep-RM model, its complexity introduces significant challenges in generator policy learning, leading to training instability. To address this, we propose a Co-evolutionary Curriculum Learning (CCL) mechanism, where both the reward model and the ISR model undergo synchronized curricula. The reward model progressively increases in complexity, while the ISR model starts with a simpler global reward for rapid convergence, gradually transitioning to the more complex model outputs. This easy-to-hard strategy enables stable training while suppressing reward hacking. Experiments validates the effectiveness of our method across ISR models in both global quality and local realism on RLHF methods.
Abstract (translated)
基于人类反馈的强化学习（RLHF）在通过奖励模型引导图像生成领域时，已被证明能够有效地对齐人类偏好。受此启发，在图像超分辨率（ISR）任务中采用RLHF显示出潜力，可以利用图像质量评估（IQA）模型作为奖励模型来优化感知质量。然而，传统的IQA模型通常输出单一的全局评分，对于局部和细微失真非常不敏感。这种不敏感性允许ISR模型产生一些在视觉上令人不满但能获得高分数的伪影，从而导致优化目标与感知质量脱节，并且会导致奖励欺骗（reward hacking）。 为了解决这个问题，我们提出了一种基于编码器-解码器架构的细粒度感知奖励模型（Fine-grained Perceptual Reward Model, FinPercep-RM）。除了提供一个全局评分之外，该模型还会生成一张感知退化图，用于空间定位并量化局部缺陷。为了训练这一模型，我们特别引入了FGR-30k数据集，其中包含来自实际超分辨率模型的各种细微失真。 尽管FinPercep-RM模型取得了一定的成功，但其复杂性为生成器策略学习带来了重大挑战，并导致了训练的不稳定性。为此，我们提出了一种协同进化课程学习（Co-evolutionary Curriculum Learning, CCL）机制，在这种机制中，奖励模型和ISR模型都经历了同步的课程。在这一过程中，奖励模型逐步增加其复杂性，而ISR模型则从简单的全局奖励开始快速收敛，然后逐渐过渡到更复杂的模型输出。这一由简至难的战略不仅能够使训练过程更加稳定，还能抑制奖励欺骗。 实验验证了我们方法的有效性，在各种ISR模型中，无论是全局质量还是局部真实性方面，RLHF方法都取得了显著的改进。
URL
https://arxiv.org/abs/2512.22647
