Abstract
Multimodal Large Language Models (MLLMs) have demonstrated profound capabilities in understanding multimodal information, covering from Image LLMs to the more complex Video LLMs. Numerous studies have illustrated their exceptional cross-modal comprehension. Recently, integrating video foundation models with large language models to build a comprehensive video understanding system has been proposed to overcome the limitations of specific pre-defined vision tasks. However, the current advancements in Video LLMs tend to overlook the foundational contributions of Image LLMs, often opting for more complicated structures and a wide variety of multimodal data for pre-training. This approach significantly increases the costs associated with these this http URL response to these challenges, this work introduces an efficient method that strategically leverages the priors of Image LLMs, facilitating a resource-efficient transition from Image to Video LLMs. We propose RED-VILLM, a Resource-Efficient Development pipeline for Video LLMs from Image LLMs, which utilizes a temporal adaptation plug-and-play structure within the image fusion module of Image LLMs. This adaptation extends their understanding capabilities to include temporal information, enabling the development of Video LLMs that not only surpass baseline performances but also do so with minimal instructional data and training resources. Our approach highlights the potential for a more cost-effective and scalable advancement in multimodal models, effectively building upon the foundational work of Image LLMs.
Abstract (translated)
多模态大型语言模型（MLLMs）在理解多模态信息方面取得了深刻的潜力，从图像LL模型到更复杂的视频LL模型。许多研究都证明了它们的跨模态理解能力。最近，将视频基础模型与大型语言模型集成以构建全面视频理解系统的主张，以克服特定预定义的视觉任务的局限性。然而，当前的视频LL模型的发展往往忽视了图像LL模型的基础贡献，通常选择更复杂的设计和各种多模态数据进行预训练。这种方法显著增加了为应对这些挑战而产生的成本，本文提出了一种有效的策略，通过战略性地利用图像LL模型的先验知识，促使从图像到视频LL模型的资源高效过渡。我们提出了RED-VILLM，一个资源高效的图像LL模型开发流程，该流程利用了图像LL模型的图像融合模块中的时间适应插件和 play-pause 结构。这个适应扩展了他们的理解能力，使他们能够开发出不仅超越基线性能，而且可以用最少的数据和训练资源实现的视频LL模型。我们的方法突出了在多模态模型中实现更成本效益和可扩展性的发展的潜力，有效地建立在图像LL模型的基础工作之上。
URL
https://arxiv.org/abs/2404.11865
