Abstract
Multimodal Large Language Models (MLLMs) hold great promise for advanced reasoning at the intersection of text and images, yet they have not fully realized this potential. MLLMs typically integrate an LLM, a vision encoder, and a connector that maps the vision encoder's embeddings into the LLM's text embedding space. Although each component is pretrained on massive datasets with billions of samples, the entire multimodal model is typically trained on only thousands (or a few million) samples, which can result in weak performance on complex reasoning tasks. To address these shortcomings, instead of relying on extensive labeled datasets for fine-tuning, we propose a Test-Time Warmup method that adapts the MLLM per test instance by leveraging data from weakly supervised auxiliary tasks. With our approach, we observe a relative performance improvement of 4.03% on MMMU, 5.28% on VQA-Rad, and 1.63% on GQA on the Llama-Vision-Instruct model. Our method demonstrates that 'warming up' before inference can enhance MLLMs' robustness across diverse reasoning tasks.
Abstract (translated)
多模态大型语言模型（MLLMs）在文本和图像交叉领域中的高级推理方面展现出巨大的潜力，但尚未完全实现这一潜能。MLLMs 通常集成了一个大型语言模型（LLM）、一个视觉编码器以及一个将视觉编码器的嵌入映射到 LLM 的文本嵌入空间中的连接器。尽管每个组件都在包含数十亿样本的大规模数据集上进行了预训练，整个多模态模型通常仅在数千个或几百万个样本上进行微调，这可能导致在复杂推理任务上的表现较弱。 为了弥补这些不足，我们提出了一种名为“测试时预热”（Test-Time Warmup）的方法。该方法通过利用来自弱监督辅助任务的数据，在每个测试实例中适应 MLLM 而无需依赖于广泛标记的数据集进行微调。采用我们的方法后，在 Llama-Vision-Instruct 模型上，MMMU 数据集的相对性能提升了 4.03%，VQA-Rad 数据集提升了 5.28%，GQA 数据集提升了 1.63%。 该方法表明，“预热”之后进行推理能够增强 MLLMs 在各种复杂推理任务中的鲁棒性。
URL
https://arxiv.org/abs/2509.10641
