Abstract
Generating multi-subject stylized images remains a significant challenge due to the ambiguity in defining style attributes (e.g., color, texture, atmosphere, and structure) and the difficulty in consistently applying them across multiple subjects. Although recent diffusion-based text-to-image models have achieved remarkable progress, existing methods typically rely on computationally expensive inversion procedures or large-scale stylized datasets. Moreover, these methods often struggle with maintaining multi-subject semantic fidelity and are limited by high inference costs. To address these limitations, we propose ICAS (IP-Adapter and ControlNet-based Attention Structure), a novel framework for efficient and controllable multi-subject style transfer. Instead of full-model tuning, ICAS adaptively fine-tunes only the content injection branch of a pre-trained diffusion model, thereby preserving identity-specific semantics while enhancing style controllability. By combining IP-Adapter for adaptive style injection with ControlNet for structural conditioning, our framework ensures faithful global layout preservation alongside accurate local style synthesis. Furthermore, ICAS introduces a cyclic multi-subject content embedding mechanism, which enables effective style transfer under limited-data settings without the need for extensive stylized corpora. Extensive experiments show that ICAS achieves superior performance in structure preservation, style consistency, and inference efficiency, establishing a new paradigm for multi-subject style transfer in real-world applications.
Abstract (translated)
生成多主体风格化图像仍然面临重大挑战，主要是由于定义风格属性（如颜色、纹理、氛围和结构）的模糊性以及在多个主体之间一致应用这些属性的难度。尽管最近基于扩散模型的文本到图像方法取得了显著进展，现有的方法通常依赖于计算成本高昂的逆向过程或大规模风格化数据集。此外，这些方法往往难以保持多主体语义保真度，并且受到高推断成本的限制。 为了克服这些局限性，我们提出了ICAS（IP-Adapter 和 ControlNet 基础注意力结构），这是一个新型框架，旨在实现高效可控的多主体风格转换。不同于全模型微调，ICAS 仅对预训练扩散模型的内容注入分支进行自适应微调，从而在保留特定身份语义的同时增强风格控制性。通过结合IP-Adapter 进行自适应风格注入与ControlNet 的结构条件设定，我们的框架确保了忠实的全局布局保存及精确的地方风格合成。 此外，ICAS 引入了一种循环多主体内容嵌入机制，在数据有限的情况下实现了有效的风格转换，无需依赖大规模风格化语料库。广泛的实验表明，ICAS 在结构保留、风格一致性和推断效率方面取得了卓越性能，为现实世界应用中的多主题风格转移开创了新的范式。
URL
https://arxiv.org/abs/2504.13224
