Abstract
Multimodal foundation models can process several modalities. However, since the space of possible modalities is large and evolving over time, training a model from scratch to encompass all modalities is unfeasible. Moreover, integrating a modality into a pre-existing foundation model currently requires a significant amount of paired data, which is often not available for low-resource modalities. In this paper, we introduce a method for sample-efficient modality integration (SEMI) into Large Language Models (LLMs). To this end, we devise a hypernetwork that can adapt a shared projector -- placed between modality-specific encoders and an LLM -- to any modality. The hypernetwork, trained on high-resource modalities (i.e., text, speech, audio, video), is conditioned on a few samples from any arbitrary modality at inference time to generate a suitable adapter. To increase the diversity of training modalities, we artificially multiply the number of encoders through isometric transformations. We find that SEMI achieves a significant boost in sample efficiency during few-shot integration of new modalities (i.e., satellite images, astronomical images, inertial measurements, and molecules) with encoders of arbitrary embedding dimensionality. For instance, to reach the same accuracy as 32-shot SEMI, training the projector from scratch needs 64$\times$ more data. As a result, SEMI holds promise to extend the modality coverage of foundation models.
Abstract (translated)
多模态基础模型可以处理多种模式。然而，由于可能的模式空间庞大且随时间演变，从头开始训练一个能够涵盖所有模式的模型是不可行的。此外，目前将一种模式整合到现有的基础模型中通常需要大量的配对数据，而这种数据对于低资源模式往往无法获得。在本文中，我们提出了一种用于大型语言模型（LLM）样例高效模态集成（SEMI）的方法。为此，我们设计了一个超网络，该网络可以在推理时根据任意模式的少量样本生成一个合适的适配器来适应放置在特定于模式编码器和LLM之间的共享投影器。这个超网络是在高资源模式（即文本、语音、音频、视频）上训练的。 为了增加训练模态的多样性，我们通过等距变换人工增加了编码器的数量。研究发现，在少量样本集成新模态（例如卫星图像、天文图像、惯性测量和分子）的过程中，SEMI在任意嵌入维度的编码器下实现了显著的样例效率提升。例如，为了达到与32次投射后SEMI相同的精度，从头开始训练投影器需要64倍更多的数据。 因此，SEMI有望扩展基础模型的模态覆盖范围。
URL
https://arxiv.org/abs/2509.04606
