Abstract
Unifying visual representation learning and text-to-image (T2I) generation within a single model remains a central challenge in multimodal learning. We introduce DREAM, a unified framework that jointly optimizes discriminative and generative objectives, while learning strong visual representations. DREAM is built on two key techniques: During training, Masking Warmup, a progressive masking schedule, begins with minimal masking to establish the contrastive alignment necessary for representation learning, then gradually transitions to full masking for stable generative training. At inference, DREAM employs Semantically Aligned Decoding to align partially masked image candidates with the target text and select the best one for further decoding, improving text-image fidelity (+6.3%) without external rerankers. Trained solely on CC12M, DREAM achieves 72.7% ImageNet linear-probing accuracy (+1.1% over CLIP) and an FID of 4.25 (+6.2% over FLUID), with consistent gains in few-shot classification, semantic segmentation, and depth estimation. These results demonstrate that discriminative and generative objectives can be synergistic, allowing unified multimodal models that excel at both visual understanding and generation.
Abstract (translated)
统一视觉表示学习和文本到图像（T2I）生成模型的单一框架仍然是多模态学习中的一个核心挑战。我们引入了DREAM，这是一个统一的框架，它同时优化判别性和生成性目标，并在此过程中学习强大的视觉表示。DREAM基于两种关键技术构建：在训练阶段，遮罩预热（Masking Warmup），一种渐进式的遮罩调度策略，从最小化遮罩开始建立对比对齐以支持表示学习，然后逐渐过渡到完全遮罩以实现稳定的生成性训练；在推理阶段，DREAM采用语义对齐解码技术来将部分遮罩的图像候选与目标文本进行对齐，并选择最佳选项继续解码，从而提高图文一致性（+6.3%），而无需外部重排序器。仅使用CC12M数据集进行训练，DREAM达到了72.7%的ImageNet线性探测精度（比CLIP高1.1%）和FID为4.25（比FLUID高6.2%），并且在少量样本分类、语义分割和深度估计方面表现出一致的优势。这些结果表明，判别性和生成性的目标可以协同工作，使得统一的多模态模型能够在视觉理解和生成两方面都表现优异。
URL
https://arxiv.org/abs/2603.02667
