Abstract
We propose ObjMST, an object-focused multimodal style transfer framework that provides separate style supervision for salient objects and surrounding elements while addressing alignment issues in multimodal representation learning. Existing image-text multimodal style transfer methods face the following challenges: (1) generating non-aligned and inconsistent multimodal style representations; and (2) content mismatch, where identical style patterns are applied to both salient objects and their surrounding elements. Our approach mitigates these issues by: (1) introducing a Style-Specific Masked Directional CLIP Loss, which ensures consistent and aligned style representations for both salient objects and their surroundings; and (2) incorporating a salient-to-key mapping mechanism for stylizing salient objects, followed by image harmonization to seamlessly blend the stylized objects with their environment. We validate the effectiveness of ObjMST through experiments, using both quantitative metrics and qualitative visual evaluations of the stylized outputs. Our code is available at: this https URL.
Abstract (translated)
我们提出了ObjMST，这是一种以对象为中心的多模态风格转换框架，它为显著对象和周围元素提供了独立的风格监督，并解决了多模态表示学习中的对齐问题。现有的图像-文本多模态风格转换方法面临以下挑战：(1) 生成不一致且不对齐的多模态风格表示；(2) 内容不匹配，在这种情况下，相同的风格模式会被应用于显著对象及其周围元素上。 我们的方法通过以下方式缓解这些问题： (1) 引入了一种特定于风格的掩码方向CLIP损失（Style-Specific Masked Directional CLIP Loss），确保了对于显著对象及其周围环境的一致且对齐的风格表示； (2) 采用了一种从显著到关键映射机制，用于给显著对象进行风格化处理，之后通过图像调和来无缝地将这些经过风格化的对象与其背景融合在一起。 我们通过实验验证了ObjMST的有效性，其中包括定量指标以及对风格化输出的定性视觉评估。我们的代码可在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2503.04353
