Abstract
The Segment Anything Model (SAM) has emerged as a powerful visual foundation model for image segmentation. However, adapting SAM to specific downstream tasks, such as medical and agricultural imaging, remains a significant challenge. To address this, Low-Rank Adaptation (LoRA) and its variants have been widely employed to enhancing SAM's adaptation performance on diverse domains. Despite advancements, a critical question arises: can we integrate inductive bias into the model? This is particularly relevant since the Transformer encoder in SAM inherently lacks spatial priors within image patches, potentially hindering the acquisition of high-level semantic information. In this paper, we propose NAS-LoRA, a new Parameter-Efficient Fine-Tuning (PEFT) method designed to bridge the semantic gap between pre-trained SAM and specialized domains. Specifically, NAS-LoRA incorporates a lightweight Neural Architecture Search (NAS) block between the encoder and decoder components of LoRA to dynamically optimize the prior knowledge integrated into weight updates. Furthermore, we propose a stage-wise optimization strategy to help the ViT encoder balance weight updates and architectural adjustments, facilitating the gradual learning of high-level semantic information. Various Experiments demonstrate our NAS-LoRA improves existing PEFT methods, while reducing training cost by 24.14% without increasing inference cost, highlighting the potential of NAS in enhancing PEFT for visual foundation models.
Abstract (translated)
《段落任何模型（SAM）》作为图像分割的强大视觉基础模型已经出现。然而，将SAM适应特定下游任务（如医学和农业成像）仍然是一项重大挑战。为此，低秩适配（LoRA）及其变体已被广泛用于增强SAM在不同领域的适应性能。尽管取得了进展，但仍有一个关键问题亟待解决：我们能否将归纳偏置整合到模型中？这尤其重要，因为SAM中的Transformer编码器本身缺乏图像补丁内的空间先验信息，可能妨碍高级语义信息的获取。 为此，在本文中，我们提出了NAS-LoRA，这是一种新的参数高效微调（PEFT）方法，旨在弥合预训练SAM与专业化领域之间的语义差距。具体而言，NAS-LoRA在LoRA的编码器和解码器组件之间引入了一个轻量级的神经架构搜索（NAS）模块，以动态优化融入权重更新中的先验知识。此外，我们提出了一种分阶段优化策略，帮助ViT编码器平衡权重更新与架构调整，从而逐步学习高级语义信息。 各种实验表明，我们的NAS-LoRA方法改进了现有的PEFT方法，并且在不增加推理成本的情况下，将训练成本降低了24.14%，这突显了NAS在增强视觉基础模型的PEFT方面的潜力。
URL
https://arxiv.org/abs/2512.03499
