Abstract
Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM reasoning. To address this issue, we present Penguin-VL, whose vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for multimodal understanding. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: this https URL
Abstract (translated)
视觉语言模型（VLM）的发展很大程度上依赖于扩大模型规模，这阻碍了其在计算资源受限的移动和边缘设备（如智能手机和机器人）上的部署。在这项工作中，我们探讨了紧凑型（例如20亿参数和80亿参数）VLMs性能极限的问题。我们挑战了一种流行做法：最先进的VLM必须依赖于通过大规模对比预训练（如CLIP/SigLIP）初始化的视觉编码器。我们发现了一个目标上的不匹配点：对比学习，专注于区分性优化，强加了粗粒度和类别级别的不变性抑制，阻碍了密集描述和复杂VLM推理所需的细粒度视觉线索。 为了解决这个问题，我们介绍了Penguin-VL，其视觉编码器是从文本语言模型初始化的。我们的实验表明，Penguin-Encoder作为传统对比预训练的一个更优替代方案，解锁了多模态理解中更高的视觉真实性和数据效率。在各种图像和视频基准上，Penguin-VL达到了与最先进的VLM（例如Qwen3-VL）相当的数学推理性能，并且在文档理解、视觉知识以及多视角视频理解等任务上超过了它们的表现。值得注意的是，这些改进是通过一个轻量级架构实现的，这表明更好的视觉表示而不是模型规模扩展才是提高性能的主要驱动因素。 我们的消融研究显示，Penguin-Encoder始终优于对比预训练编码器，在密集感知和复杂推理中保持了关键的细粒度空间和时间线索。这使其成为计算效率型VLM的一个强有力的替代选择，并在资源受限环境中也能实现高性能。代码：[请在这里插入实际链接]
URL
https://arxiv.org/abs/2603.06569
