Abstract
Pretrained vision-language models (VLMs) such as CLIP excel in multimodal understanding but struggle with contextually relevant fine-grained visual features, making it difficult to distinguish visually similar yet culturally distinct concepts. This limitation stems from the scarcity of high-quality culture-specific datasets, the lack of integrated contextual knowledge, and the absence of hard negatives highlighting subtle distinctions. To address these challenges, we first design a data curation pipeline that leverages open-sourced VLMs and text-to-image diffusion models to construct CulTwin, a synthetic cultural dataset. This dataset consists of paired concept-caption-image triplets, where concepts visually resemble each other but represent different cultural contexts. Then, we fine-tune CLIP on CulTwin to create CultureCLIP, which aligns cultural concepts with contextually enhanced captions and synthetic images through customized contrastive learning, enabling finer cultural differentiation while preserving generalization capabilities. Experiments on culturally relevant benchmarks show that CultureCLIP outperforms the base CLIP, achieving up to a notable 5.49% improvement in fine-grained concept recognition on certain tasks, while preserving CLIP's original generalization ability, validating the effectiveness of our data synthesis and VLM backbone training paradigm in capturing subtle cultural distinctions.
Abstract (translated)
预训练的视觉-语言模型（VLMs），如CLIP，在多模态理解方面表现出色，但在处理上下文相关的细粒度视觉特征时却面临困难。这使得区分在视觉上相似但文化背景不同的概念变得很困难。这种限制源自高质量的文化特有数据集的稀缺、集成上下文知识的缺失以及缺乏强调微妙区别的硬负样本（hard negatives）。为了解决这些挑战，我们首先设计了一种数据整理流水线，该流水线利用开源VLM和文本到图像扩散模型来构建CulTwin这一合成文化数据集。该数据集由成对的概念-描述-图片三元组组成，其中概念在视觉上看起来相似但代表不同的文化背景。 然后，我们通过定制的对比学习方法，在CulTwin上微调CLIP以创建CultureCLIP，使文化概念与上下文增强的描述和合成图像相匹配。这种方法能够在保持一般化能力的同时实现更精细的文化区分。在相关文化的基准测试中进行的实验表明，CultureCLIP的表现优于基础CLIP模型，在某些任务中的细粒度概念识别上实现了高达5.49%的显著改进，并且保留了CLIP原有的泛化能力。这验证了我们的数据合成和VLM骨干训练范式在捕捉微妙文化差异方面是有效的。
URL
https://arxiv.org/abs/2507.06210
