Abstract
Incorporating automatically predicted human feedback into the process of training generative models has attracted substantial recent interest, while feedback at inference time has received less attention. The typical feedback at training time, i.e., preferences of choice given two samples, does not naturally transfer to the inference phase. We introduce a novel type of feedback -- caption reformulations -- and train models to mimic reformulation feedback based on human annotations. Our method does not require training the image captioning model itself, thereby demanding substantially less computational effort. We experiment with two types of reformulation feedback: first, we collect a dataset of human reformulations that correct errors in the generated captions. We find that incorporating reformulation models trained on this data into the inference phase of existing image captioning models results in improved captions, especially when the original captions are of low quality. We apply our method to non-English image captioning, a domain where robust models are less prevalent, and gain substantial improvement. Second, we apply reformulations to style transfer. Quantitative evaluations reveal state-of-the-art performance on German image captioning and English style transfer, while human validation with a detailed comparative framework exposes the specific axes of improvement.
Abstract (translated)
将自动预测的人类反馈融入生成模型的训练过程引起了近期广泛的关注，而推理阶段的反馈则较少受到重视。在训练过程中常见的反馈类型（即给定两个样本时的选择偏好）并不自然地适用于推理阶段。我们提出了一种新型反馈——标题改写，并基于人类注释训练模型模仿这种改写反馈。我们的方法不需要重新训练图像描述生成模型，从而大大减少了计算需求。我们在两种类型的改写反馈上进行了实验：首先，收集了一个数据集，其中包含纠正生成标题错误的人类改写。我们发现，在现有图像描述模型的推理阶段集成基于这些数据训练的改写模型可以提高标题质量，尤其是在原始标题质量较低的情况下效果更佳。我们将该方法应用于非英语图像描述领域——这是一个稳健模型较少见的领域，并获得了显著改进。其次，我们在风格迁移中应用了这种改写技术。定量评估显示，在德语图像描述和英语风格迁移方面达到了最先进的性能，而详细对比框架下的人类验证揭示了具体的改进维度。
URL
https://arxiv.org/abs/2501.04513
