Abstract
The integration of prompt tuning with multimodal learning has shown significant generalization abilities for various downstream tasks. Despite advancements, existing methods heavily depend on massive modality-specific labeled data (e.g., video, audio, and image), or are customized for a single modality. In this study, we present Text as Any-Modality by Consistent Prompt Tuning (TaAM-CPT), a scalable approach for constructing a general representation model toward unlimited modalities using solely text data. TaAM-CPT comprises modality prompt pools, text construction, and modality-aligned text encoders from pre-trained models, which allows for extending new modalities by simply adding prompt pools and modality-aligned text encoders. To harmonize the learning across different modalities, TaAM-CPT designs intra- and inter-modal learning objectives, which can capture category details within modalities while maintaining semantic consistency across different modalities. Benefiting from its scalable architecture and pre-trained models, TaAM-CPT can be seamlessly extended to accommodate unlimited modalities. Remarkably, without any modality-specific labeled data, TaAM-CPT achieves leading results on diverse datasets spanning various modalities, including video classification, image classification, and audio classification. The code is available at this https URL.
Abstract (translated)
将提示调优与多模态学习相结合已经展示了在各种下游任务中的显著泛化能力。尽管有所进展，现有的方法仍然严重依赖大量特定模式的标注数据（例如视频、音频和图像），或者只为单一模式定制。在这项研究中，我们提出了由一致提示调优实现的“文本作为任何模态”（Text as Any-Modality by Consistent Prompt Tuning，简称TaAM-CPT），这是一种可扩展的方法，仅使用文本数据即可构建面向无限数量模态的一般表示模型。 TaAM-CPT 包括模式提示池、文本构造和从预训练模型中获取的与模式一致的文本编码器。这使得通过简单地添加提示池和与模式一致的文本编码器就可以扩展新的模态。为了在不同模态之间实现和谐学习，TaAM-CPT 设计了内模态和跨模态的学习目标，这些目标可以在捕获模态内部类别细节的同时保持跨不同模态的语义一致性。 得益于其可扩展架构和预训练模型，TaAM-CPT 可以无缝地扩展以适应无限数量的模式。特别值得注意的是，在没有特定模式标注数据的情况下，TaAM-CPT 在跨越视频分类、图像分类和音频分类等不同模态的各种数据集上取得了领先的结果。 该项目代码可在以下链接获取：[https URL]（请将 [https URL] 替换为实际提供的URL）。
URL
https://arxiv.org/abs/2508.06382
