Abstract
Multimodal large language models (MLLMs) have significantly advanced the integration of visual and textual understanding. However, their ability to generate code from multimodal inputs remains limited. In this work, we introduce VisCodex, a unified framework that seamlessly merges vision and coding language models to empower MLLMs with strong multimodal code generation abilities. Leveraging a task vector-based model merging technique, we integrate a state-of-the-art coding LLM into a strong vision-language backbone, while preserving both visual comprehension and advanced coding skills. To support training and evaluation, we introduce the Multimodal Coding Dataset (MCD), a large-scale and diverse collection of 598k samples, including high-quality HTML code, chart image-code pairs, image-augmented StackOverflow QA, and algorithmic problems. Furthermore, we propose InfiBench-V, a novel and challenging benchmark specifically designed to assess models on visually-rich, real-world programming questions that demand a nuanced understanding of both textual and visual contexts. Extensive experiments show that VisCodex achieves state-of-the-art performance among open-source MLLMs and approaches proprietary models like GPT-4o, highlighting the effectiveness of our model merging strategy and new datasets.
Abstract (translated)
多模态大型语言模型（MLLMs）在视觉和文本理解的融合上取得了显著进步。然而，它们从多模态输入生成代码的能力仍然有限。在这项工作中，我们介绍了VisCodex，这是一个统一的框架，它无缝地将视觉与编码语言模型结合起来，赋予MLLM强大的多模态代码生成能力。通过采用基于任务向量的模型融合技术，我们将最先进的编码LLM集成到一个强大的视觉-语言骨干网络中，同时保持其对图像的理解能力和高级编程技能。 为了支持训练和评估，我们引入了多模态编码数据集（Multimodal Coding Dataset, MCD），这是一个包含598k个样本的大规模多样化数据集，其中包括高质量的HTML代码、图表图片-代码配对、带图增强的StackOverflow问答以及算法问题。此外，我们还提出了InfiBench-V，一个新的具有挑战性的基准测试，专门用于评估模型在视觉丰富的真实世界编程任务上的表现能力，这些任务需要同时理解文本和视觉上下文。 广泛的实验表明，VisCodex在开源MLLM中表现出最先进的性能，并且接近专有模型如GPT-4o的水平。这突显了我们的模型融合策略以及新数据集的有效性。
URL
https://arxiv.org/abs/2508.09945
