Abstract
Unified diffusion editors often rely on a fixed, shared backbone for diverse tasks, suffering from task interference and poor adaptation to heterogeneous demands (e.g., local vs global, semantic vs photometric). In particular, prevalent ControlNet and OmniControl variants combine multiple conditioning signals (e.g., text, mask, reference) via static concatenation or additive adapters which cannot dynamically prioritize or suppress conflicting modalities, thus resulting in artifacts like color bleeding across mask boundaries, identity or style drift, and unpredictable behavior under multi-condition inputs. To address this, we propose Condition-Aware Routing of Experts (CARE-Edit) that aligns model computation with specific editing competencies. At its core, a lightweight latent-attention router assigns encoded diffusion tokens to four specialized experts--Text, Mask, Reference, and Base--based on multi-modal conditions and diffusion timesteps: (i) a Mask Repaint module first refines coarse user-defined masks for precise spatial guidance; (ii) the router applies sparse top-K selection to dynamically allocate computation to the most relevant experts; (iii) a Latent Mixture module subsequently fuses expert outputs, coherently integrating semantic, spatial, and stylistic information to the base images. Experiments validate CARE-Edit's strong performance on contextual editing tasks, including erasure, replacement, text-driven edits, and style transfer. Empirical analysis further reveals task-specific behavior of specialized experts, showcasing the importance of dynamic, condition-aware processing to mitigate multi-condition conflicts.
Abstract (translated)
统一扩散编辑器通常依赖于一个固定的、共享的骨干网络来执行各种任务，这会导致任务干扰和对异构需求（例如局部与全局、语义与光度）适应性差的问题。特别是流行的ControlNet和OmniControl变体通过静态拼接或加法适配器结合多种条件信号（如文本、掩码、参考图像），但这些方法无法动态地优先处理或抑制冲突模态，因此在多条件输入下会出现诸如跨掩码边界的颜色溢出、身份或风格漂移以及行为不可预测等问题。为解决这些问题，我们提出了Condition-Aware Routing of Experts (CARE-Edit)，该方法使模型计算与特定编辑任务的能力相匹配。核心思想是一个轻量级的潜在注意力路由器根据多模态条件和扩散时间步骤将编码的扩散令牌分配给四个专门化的专家——文本、掩码、参考图像和基础图像： 1. 掩码重绘模块首先对粗略的用户定义掩码进行细化，以提供精确的空间指导。 2. 路由器应用稀疏的Top-K选择来动态地将计算资源分配给最相关的专家。 3. 潜在混合模块随后融合各个专家的输出，将语义、空间和风格信息连贯地整合到基础图像中。 实验验证了CARE-Edit在上下文编辑任务（包括擦除、替换、文本驱动的编辑和风格转移）中的强大性能。实证分析进一步揭示了专门化专家的任务特定行为，展示了动态、条件感知处理的重要性以减轻多条件冲突的影响。
URL
https://arxiv.org/abs/2603.08589
