Abstract
Vision Language Models (VLMs) integrate visual and text modalities to enable multimodal understanding and generation. These models typically combine a Vision Transformer (ViT) as an image encoder and a Large Language Model (LLM) for text generation. LoRA (Low-Rank Adaptation) is an efficient fine-tuning method to adapt pre-trained models to new tasks by introducing low-rank updates to their weights. While LoRA has emerged as a powerful technique for fine-tuning large models by introducing low-rank updates, current implementations assume a fixed rank, potentially limiting flexibility and efficiency across diverse tasks. This paper introduces \textit{LangVision-LoRA-NAS}, a novel framework that integrates Neural Architecture Search (NAS) with LoRA to optimize VLMs for variable-rank adaptation. Our approach leverages NAS to dynamically search for the optimal LoRA rank configuration tailored to specific multimodal tasks, balancing performance and computational efficiency. Through extensive experiments using the LLaMA-3.2-11B model on several datasets, LangVision-LoRA-NAS demonstrates notable improvement in model performance while reducing fine-tuning costs. Our Base and searched fine-tuned models on LLaMA-3.2-11B-Vision-Instruct can be found \href{this https URL}{\textcolor{blue}{here}} and the code for LangVision-LoRA-NAS can be found \href{this https URL}{\textcolor{blue}{here}}.
Abstract (translated)
视觉语言模型（VLMs）结合了视觉和文本模态，以实现多模态理解和生成。这些模型通常将视觉变压器（ViT）用作图像编码器，并使用大型语言模型（LLM）进行文本生成。低秩适应（LoRA）是一种高效的微调方法，通过向预训练模型的权重引入低秩更新来使其适应新任务。虽然LoRA已成为一种强大的技术，用于通过引入低秩更新来微调大规模模型，但目前的实现假设固定的秩，这可能限制了在各种任务中的灵活性和效率。本文介绍了**LangVision-LoRA-NAS**，这是一种新颖的框架，它将神经架构搜索（NAS）与LoRA结合使用，以优化视觉语言模型进行变秩适应。我们的方法利用NAS动态地为特定多模态任务寻找最优的LoRA秩配置，在性能和计算效率之间取得平衡。通过在LLaMA-3.2-11B模型上对多个数据集进行广泛的实验，LangVision-LoRA-NAS展示了显著提高模型性能的同时降低了微调成本。 我们的基础和搜索微调后的LLaMA-3.2-11B-Vision-Instruct模型可在此[链接]处获取，而LangVision-LoRA-NAS的代码可在另一链接处找到。请注意，上述链接应替换为实际提供的URL地址。
URL
https://arxiv.org/abs/2508.12512
