Abstract
A significant ``modality gap" exists between the abundance of text-only data and the increasing power of multimodal models. This work systematically investigates whether images generated on-the-fly by Text-to-Image (T2I) models can serve as a valuable complementary modality for text-centric tasks. Through a comprehensive evaluation framework on text classification, we analyze the impact of critical variables, including T2I model quality, prompt engineering strategies, and multimodal fusion architectures. Our findings demonstrate that this``synthetic perception" can yield significant performance gains, even when augmenting strong large language model baselines. However, we find the effectiveness of this approach is highly conditional, depending critically on the semantic alignment between text and the generated image, the inherent ``visual groundability" of the task, and the generative fidelity of the T2I model. Our work establishes the first rigorous benchmark for this paradigm, providing a clear analysis of its potential and current limitations, and demonstrating its viability as a pathway to enrich language understanding in traditionally unimodal scenarios.
Abstract (translated)
在文本数据丰富而多模态模型能力日益增强的情况下，存在一个显著的“模式差距”。本项研究系统地探讨了通过文字生成图像（T2I）的模型即时生成的图片能否作为以文本为中心的任务的一种有价值的补充模式。我们通过全面的评估框架，在文本分类任务上分析了关键变量的影响，包括T2I模型的质量、提示工程策略和多模态融合架构。我们的研究结果表明，“合成感知”即使在增强强大的大规模语言模型基准的情况下也能带来显著的性能提升。然而，这种方法的有效性高度依赖于文本与生成图像之间的语义一致性、任务本身的“视觉基础性”，以及T2I模型的生成保真度。 本项工作建立了这一范例的第一个严格基准，对它的潜力和当前限制进行了清晰分析，并展示了其作为丰富传统单一模式场景中语言理解途径的可能性。
URL
https://arxiv.org/abs/2506.17623
