Abstract
Multimodal Large Language Models (MLLMs) have recently achieved remarkable success in visual-language understanding, demonstrating superior high-level semantic alignment within their vision encoders. An important question thus arises: Can these encoders serve as versatile vision backbones, capable of reliably performing classic vision-centric tasks as well? To address the question, we make the following contributions: (i) we identify that the vision encoders within MLLMs exhibit deficiencies in their dense feature representations, as evidenced by their suboptimal performance on dense prediction tasks (e.g., semantic segmentation, depth estimation); (ii) we propose VersaViT, a well-rounded vision transformer that instantiates a novel multi-task framework for collaborative post-training. This framework facilitates the optimization of the vision backbone via lightweight task heads with multi-granularity supervision; (iii) extensive experiments across various downstream tasks demonstrate the effectiveness of our method, yielding a versatile vision backbone suited for both language-mediated reasoning and pixel-level understanding.
Abstract (translated)
多模态大型语言模型（MLLM）近期在视觉-语言理解方面取得了显著成功，展示了其视觉编码器中出色的高级语义对齐能力。由此引发了一个重要的问题：这些编码器能否作为通用的视觉骨干网络，可靠地执行经典的以视觉为中心的任务？为了解决这个问题，我们做出了以下贡献： (i) 我们识别出多模态大型语言模型中的视觉编码器在密集特征表示方面存在不足，在密集预测任务（如语义分割、深度估计）中表现出性能不佳。 (ii) 我们提出了一种全面的视觉变换器——VersaViT，该变换器实例化了一个新颖的多任务框架，用于协作式后期训练。此框架通过带有多种粒度监督的轻量级任务头优化视觉骨干网络。 (iii) 在各种下游任务中的广泛实验表明了我们方法的有效性，生成了一种适用于语言介导推理和像素级理解的通用视觉骨干网络。 通过以上贡献，VersaViT不仅提升了MLLM在视觉密集预测方面的性能，还为开发能够同时支持高层次语义理解和精确像素级分析的多功能视觉模型铺平了道路。
URL
https://arxiv.org/abs/2602.09934
