Abstract
The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance on resolution-sensitive tasks, such as optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vision encoders. Despite their success, there is a lack of systematic comparisons and detailed ablation studies addressing critical aspects, such as expert selection and the integration of multiple vision experts. This study provides an extensive exploration of the design space for MLLMs using a mixture of vision encoders and resolutions. Our findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. We additionally introduce Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks. Models and code: this https URL
Abstract (translated)
准确地解释复杂视觉信息是一个多模态大型语言模型（MLLM）的重要话题。最近的工作表明，增强视觉感知在显著减少幻觉和提高分辨率敏感任务（如光学字符识别和文档分析）的表现方面至关重要。许多最近的MLLM使用混合视觉编码器来实现这一目标。尽管它们取得了成功，但缺乏系统性的比较和详细消融研究解决了一些关键方面，如专家选择和多视觉专家的集成。 本研究对使用混合视觉编码器的设计空间进行了深入探索，以设计MLLM。我们的发现揭示了各种现有策略背后的共同原理，从而实现了简洁有效的设计方法。我们发现，简单地将来自互补视觉编码器的视觉标记串联起来与更复杂的混合架构或策略相当有效。此外，我们还引入了预对齐来连接视觉关注编码器和语言标记，增强了模型的连贯性。 所得到的一组MLLM家族，Eagle，在主要MLLM基准测试中超过了其他领先的开源模型。模型和代码：这个链接
URL
https://arxiv.org/abs/2408.15998
