Abstract
Compositional training has been the de-facto paradigm in existing Multimodal Large Language Models (MLLMs), where pre-trained vision encoders are connected with pre-trained LLMs through continuous multimodal pre-training. However, the multimodal scaling property of this paradigm remains difficult to explore due to the separated training. In this paper, we focus on the native training of MLLMs in an end-to-end manner and systematically study its design space and scaling property under a practical setting, i.e., data constraint. Through careful study of various choices in MLLM, we obtain the optimal meta-architecture that best balances performance and training cost. After that, we further explore the scaling properties of the native MLLM and indicate the positively correlated scaling relationship between visual encoders and LLMs. Based on these findings, we propose a native MLLM called NaViL, combined with a simple and cost-effective recipe. Experimental results on 14 multimodal benchmarks confirm the competitive performance of NaViL against existing MLLMs. Besides that, our findings and results provide in-depth insights for the future study of native MLLMs.
Abstract (translated)
现有多模态大型语言模型（MLLMs）的组成训练已成为默认范式，在这种模式下，预训练的视觉编码器通过连续的多模态预训练与预训练的语言模型相连。然而，由于分阶段训练的原因，这一框架下的多模态扩展属性难以探索。本文专注于在实际数据约束条件下对多模态大型语言模型进行端到端原生训练，并系统地研究其设计空间和可扩展性。通过对各种MLLM选择的仔细研究，我们获得了能够最佳平衡性能与训练成本的元架构。之后，进一步探讨了原生MLLM的扩展属性，并指出视觉编码器与LLM之间存在积极相关的扩展关系。基于这些发现，我们提出了一种名为NaViL的原生多模态语言模型，结合了一个简单且经济高效的配置方案。在14个多模态基准测试中的实验结果证实了NaViL相对于现有MLLM的竞争性性能。此外，我们的研究和成果为未来对原生MLLM的研究提供了深刻的见解。
URL
https://arxiv.org/abs/2510.08565
