Abstract
We present MILS: Multimodal Iterative LLM Solver, a surprisingly simple, training-free approach, to imbue multimodal capabilities into your favorite LLM. Leveraging their innate ability to perform multi-step reasoning, MILS prompts the LLM to generate candidate outputs, each of which are scored and fed back iteratively, eventually generating a solution to the task. This enables various applications that typically require training specialized models on task-specific data. In particular, we establish a new state-of-the-art on emergent zero-shot image, video and audio captioning. MILS seamlessly applies to media generation as well, discovering prompt rewrites to improve text-to-image generation, and even edit prompts for style transfer! Finally, being a gradient-free optimization approach, MILS can invert multimodal embeddings into text, enabling applications like cross-modal arithmetic.
Abstract (translated)
我们介绍了MILS：多模态迭代LLM求解器，这是一种令人惊讶地简单且无需训练的方法，用于赋予您喜欢的大型语言模型（LLM）多模态能力。通过利用它们内在的执行多步推理的能力，MILS提示LLM生成候选输出，并对每个输出进行评分并反复反馈，最终生成任务解决方案。这使得各种通常需要在特定任务数据上训练专用模型的应用程序成为可能。特别是，我们在新兴的零样本图像、视频和音频描述方面建立了新的最先进的成果。MILS还无缝地适用于媒体生成，通过发现提示重写来改进文本到图像生成，并且甚至可以编辑提示以进行风格转换！最后，作为无梯度优化方法，MILS能够将多模态嵌入逆向映射为文本，从而使跨模态算术等应用成为可能。
URL
https://arxiv.org/abs/2501.18096
