Abstract
Pretrained vision-language models (VLMs), such as CLIP, have shown remarkable potential in few-shot image classification and led to numerous effective transfer learning strategies. These methods leverage the pretrained knowledge of VLMs to enable effective domain adaptation while mitigating overfitting through parameter-efficient tuning or instance-based consistency constraints. However, such regularizations often neglect the geometric structure of data distribution, which may lead to distortion of the overall semantic representation. To overcome this limitation, we propose a novel fine-tuning method, Manifold-Preserving and Sculpting Tuning (MPS-Tuning). Regarding the data distribution in feature space as a semantic manifold, MPS-Tuning explicitly constrains the intrinsic geometry of this manifold while further sculpting it to enhance class separability. Specifically, MPS-Tuning preserves both macroscopic and microscopic topological structures of the original manifold by aligning Gram matrices of features before and after fine-tuning. Theoretically, this constraint is shown to approximate an upper bound of the Gromov-Wasserstein distance. Furthermore, features from the image and text modalities are paired, and pairwise similarities are optimized to enhance the manifold's class discriminability. Extensive experiments demonstrate that MPS-Tuning significantly improves model performance while effectively preserving the structure of the semantic manifold. The code will be released.
Abstract (translated)
预训练的视觉-语言模型（VLMs），如CLIP，在少量样本图像分类中表现出显著潜力，并促成了许多有效的迁移学习策略。这些方法利用了VLMs的预训练知识，以实现有效的领域适应性并通过参数高效的微调或基于实例的一致性约束来减轻过拟合问题。然而，这种正则化通常忽视了数据分布的几何结构，这可能导致整体语义表示失真。为了克服这一局限，我们提出了一种新的微调方法：保形雕塑微调（MPS-Tuning）。将特征空间中的数据分布视为语义流形，MPS-Tuning 明确地约束了这种流形的内在几何形状，并进一步对其进行塑形以增强类别可分离性。具体来说，通过在微调前后的特征Gram矩阵对齐来保留原始流形的宏观和微观拓扑结构。从理论上讲，这一限制显示其接近Gromov-Wasserstein距离的一个上限。此外，图像和文本模式中的特征被配对，并且成对相似度被优化以增强流形的类别判别能力。广泛的实验表明，MPS-Tuning 显著提高了模型性能，同时有效地保持了语义流形的结构。代码将会发布。
URL
https://arxiv.org/abs/2508.12877
