Abstract
To explore a more scalable path for adding multimodal capabilities to existing LLMs, this paper addresses a fundamental question: Can a unimodal LLM, relying solely on text, reason about its own informational needs and provide effective feedback to optimize a multimodal model? To answer this, we propose a method that enables a language agent to give feedback to a vision-language model (VLM) to adapt text generation to the agent's preferences. Our results from different experiments affirm this hypothesis, showing that LLM preference feedback significantly enhances VLM descriptions. Using our proposed method, we find that the VLM can generate multimodal scene descriptions to help the LLM better understand multimodal context, leading to improvements of maximum 13% in absolute accuracy compared to the baseline multimodal approach. Furthermore, a human study validated our AI-driven feedback, showing a 64.6% preference alignment rate between the LLM's choices and human judgments. Extensive experiments provide insights on how and why the method works and its limitations.
Abstract (translated)
为了探索将多模态功能添加到现有大规模语言模型（LLM）的更可扩展路径，本文提出并回答了一个基本问题：一个仅依赖文本的一模态LLM是否能够根据自己的信息需求进行推理，并提供有效反馈以优化多模态模型？为解答这一问题，我们提出了一种方法，该方法使语言代理能够向视觉-语言模型（VLM）提供反馈，从而适应其偏好调整文本生成。来自不同实验的结果证实了这个假设，表明LLM偏好的反馈显著增强了VLM的描述效果。使用我们提出的这种方法，发现VLM可以生成多模态场景描述，以帮助LLM更好地理解多模态背景信息，与基线方法相比，在绝对准确性方面最多可提高13%。此外，一项人类研究验证了我们的AI驱动反馈的有效性，结果显示在LLM选择和人工判断之间达到了64.6%的偏好一致性率。广泛的实验提供了关于该方法如何以及为何有效及其局限性的见解。
URL
https://arxiv.org/abs/2601.06424
