Abstract
We introduce V-SONAR, a vision-language embedding space extended from the text-only embedding space SONAR (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To construct V-SONAR, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the SONAR space. We thoroughly evaluate V-SONAR and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the OMNISONAR text decoder, V-SONAR further surpasses state-of-the-art vision-language models on video captioning tasks, including DREAM-1K (BLEU 23.9 vs. 19.6) and PE-VIDEO (BLEU 39.0 vs. 30.0). Leveraging V-SONAR, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in SONAR and trained with English text only, can perform both single- and multi-visual concept understanding in a zero-shot manner. Finally, we introduce V-LCM, which extends the LCM with vision-language instruction tuning. V-LCM encodes vision and language inputs into an unified sequence of latent embeddings via V-SONAR and SONAR, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM's text-only pre-training. Experiments on a large-scale multilingual and -modal instruction-tuning data mixture highlight the potential of V-LCM: V-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.
Abstract (translated)
我们介绍了V-SONAR，这是一个从仅文本嵌入空间SONAR（Omnilingual Embeddings Team等，2026年）扩展而来的视觉-语言嵌入空间，支持1500种文字语言和177种语音语言。为了构建V-SONAR，我们提出了一种后处理对齐流程，该流程将现有视觉编码器的表示映射到SONAR空间中。我们全面评估了V-SONAR，并展示了其嵌入在文本到视频检索任务上取得了竞争性性能。配备了OMNISONAR文本解码器，V-SONAR进一步超越了最新的视觉-语言模型，在包括DREAM-1K（BLEU 23.9 vs. 19.6）和PE-VIDEO（BLEU 39.0 vs. 30.0）的视频字幕任务上。 利用V-SONAR，我们首先展示了在SONAR中运行并仅使用英语文本进行训练的大概念模型(LCM；LCM团队等,2024)可以零样本地执行单个和多个视觉概念理解。最后，我们介绍了V-LCM，它将视觉-语言指令微调扩展到了LCM中。V-LCM通过V-SONAR和SONAR将视觉和语言输入编码为统一的潜在嵌入序列，并且它的训练使用了与LCM仅文本预训练相同的潜扩散目标进行下一个嵌入预测。 在大规模多语种和多模态指令微调数据集上的实验突显了V-LCM的潜力：在涵盖图像/视频字幕和问答任务上，V-LCM的表现可媲美最新的视觉-语言模型；而在所有62种测试语言中，其在包含从丰富资源到低资源在内的61种语言中的表现显著超越。
URL
https://arxiv.org/abs/2603.01096
