Abstract
Inspired by the success of generative pretraining in natural language, we ask whether the same principles can yield strong self-supervised visual learners. Instead of training models to output features for downstream use, we train them to generate embeddings to perform predictive tasks directly. This work explores such a shift from learning representations to learning models. Specifically, models learn to predict future patch embeddings conditioned on past ones, using causal masking and stop gradient, which we refer to as Next-Embedding Predictive Autoregression (NEPA). We demonstrate that a simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole learning objective is effective - no pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads. This formulation retains architectural simplicity and scalability, without requiring additional design complexity. NEPA achieves strong results across tasks, attaining 83.8% and 85.3% top-1 accuracy on ImageNet-1K with ViT-B and ViT-L backbones after fine-tuning, and transferring effectively to semantic segmentation on ADE20K. We believe generative pretraining from embeddings provides a simple, scalable, and potentially modality-agnostic alternative to visual self-supervised learning.
Abstract (translated)
受自然语言生成预训练成功的启发，我们探讨了同样的原则是否可以培养出强大的自监督视觉学习者。与其训练模型输出用于下游任务的特征，我们将它们训练成能够直接执行预测性任务的嵌入生成器。这项工作探索了一种从学习表示转向学习模型的方法。具体而言，模型学会了根据过去的嵌入来预测未来的补丁(embedding)信息，使用因果屏蔽和停止梯度技术，我们称之为“Next-Embedding 预测自回归”(NEPA)。 研究表明，一个仅通过预测下一个嵌入作为其唯一训练目标的简单Transformer，在ImageNet-1k上进行预训练是有效的——这种方法不需要像素重建、离散令牌(token)、对比损失或特定任务的头部设计。这种形式保持了架构上的简洁性和可扩展性，并且无需额外的设计复杂度。 NEPA在多项任务中取得了优异的成绩，例如使用ViT-B和ViT-L基础模型对ImageNet-1K进行微调后分别达到了83.8%和85.3%的top-1准确率；此外，在ADE20K数据集上的语义分割转移学习也表现良好。我们认为从嵌入生成的预训练提供了一种简单、可扩展且可能跨模态(即适用于多种类型的数据输入)的方法，来替代现有的视觉自监督学习方法。
URL
https://arxiv.org/abs/2512.16922
