Abstract
We introduce VL-JEPA, a vision-language model built on a Joint Embedding Predictive Architecture (JEPA). Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts. By learning in an abstract representation space, the model focuses on task-relevant semantics while abstracting away surface-level linguistic variability. In a strictly controlled comparison against standard token-space VLM training with the same vision encoder and training data, VL-JEPA achieves stronger performance while having 50% fewer trainable parameters. At inference time, a lightweight text decoder is invoked only when needed to translate VL-JEPA predicted embeddings into text. We show that VL-JEPA natively supports selective decoding that reduces the number of decoding operations by 2.85x while maintaining similar performance compared to non-adaptive uniform decoding. Beyond generation, the VL-JEPA's embedding space naturally supports open-vocabulary classification, text-to-video retrieval, and discriminative VQA without any architecture modification. On eight video classification and eight video retrieval datasets, the average performance VL-JEPA surpasses that of CLIP, SigLIP2, and Perception Encoder. At the same time, the model achieves comparable performance as classical VLMs (InstructBLIP, QwenVL) on four VQA datasets: GQA, TallyQA, POPE and POPEv2, despite only having 1.6B parameters.
Abstract (translated)
我们介绍了一种基于联合嵌入预测架构（JEPA）的视觉-语言模型——VL-JEPA。不同于传统的视觉-语言模型（VLMs），在生成时逐个自回归地产生标记，VL-JEPA 预测目标文本的连续嵌入。通过在一个抽象表示空间中学习，该模型专注于任务相关的语义，并忽略表面级的语言变异性。在使用相同的视觉编码器和训练数据进行严格控制对比的情况下，与标准词元空间VLM训练相比，VL-JEPA 达到了更强的性能表现，同时其可训练参数减少了50%。推理时，当需要将VL-JEPA预测的嵌入转换为文本时才调用轻量级文本解码器。 我们展示了VL-JEPA原生支持选择性解码，在保持与非自适应均匀解码相似性能的同时，减少了解码操作次数2.85倍。除了生成任务外，VL-JEPA 的嵌入空间自然地支持开放词汇分类、文本到视频检索和判别式视觉问答（VQA），无需进行任何架构修改。 在八个视频分类数据集和八个视频检索数据集中，平均性能表明VL-JEPA超越了CLIP、SigLIP2和感知编码器。同时，在GQA、TallyQA、POPE 和 POPEv2这四个VQA 数据集上，尽管参数量仅为1.6B，其性能与经典的视觉-语言模型（如InstructBLIP, QwenVL）相当。
URL
https://arxiv.org/abs/2512.10942
