Abstract
Despite recent advances in diffusion models, top-tier text-to-image (T2I) models still struggle to achieve precise spatial layout control, i.e. accurately generating entities with specified attributes and locations. Segmentation-mask-to-image (S2I) generation has emerged as a promising solution by incorporating pixel-level spatial guidance and regional text prompts. However, existing S2I methods fail to simultaneously ensure semantic consistency and shape consistency. To address these challenges, we propose Seg2Any, a novel S2I framework built upon advanced multimodal diffusion transformers (e.g. FLUX). First, to achieve both semantic and shape consistency, we decouple segmentation mask conditions into regional semantic and high-frequency shape components. The regional semantic condition is introduced by a Semantic Alignment Attention Mask, ensuring that generated entities adhere to their assigned text prompts. The high-frequency shape condition, representing entity boundaries, is encoded as an Entity Contour Map and then introduced as an additional modality via multi-modal attention to guide image spatial structure. Second, to prevent attribute leakage across entities in multi-entity scenarios, we introduce an Attribute Isolation Attention Mask mechanism, which constrains each entity's image tokens to attend exclusively to themselves during image self-attention. To support open-set S2I generation, we construct SACap-1M, a large-scale dataset containing 1 million images with 5.9 million segmented entities and detailed regional captions, along with a SACap-Eval benchmark for comprehensive S2I evaluation. Extensive experiments demonstrate that Seg2Any achieves state-of-the-art performance on both open-set and closed-set S2I benchmarks, particularly in fine-grained spatial and attribute control of entities.
Abstract (translated)
尽管在扩散模型方面取得了近期进展，顶级的文本到图像（T2I）模型仍然难以实现精确的空间布局控制，即准确生成具有指定属性和位置的实体。分割掩码到图像（S2I）生成作为一种有前景的解决方案应运而生，它通过引入像素级别的空间指导和区域文本提示来改进这一过程。然而，现有的S2I方法未能同时确保语义一致性和形状一致性。 为了解决这些挑战，我们提出了Seg2Any，这是一种基于先进多模态扩散变换器（例如FLUX）构建的新型S2I框架。首先，为了实现语义和形状的一致性，我们将分割掩码条件解耦为区域语义和高频形状成分。通过引入一个语义对齐注意力掩码来提出区域语义条件，确保生成的实体符合其分配的文本提示。高频形状条件表示实体边界，并被编码成实体轮廓图，然后作为额外模态通过多模态注意机制引入，以指导图像的空间结构。 其次，在涉及多个实体的情况下，为了防止属性在不同实体之间泄漏，我们引入了一种属性隔离注意力掩码机制，该机制约束每个实体的图像令牌仅在图像自我关注时关注自身。为支持开放集S2I生成，我们构建了SACap-1M，这是一个大规模数据集，包含100万张图片和590万个分割实体及其详细的区域描述文本，以及一个用于全面评估的SACap-Eval基准。 广泛的实验表明，Seg2Any在开放集和封闭集S2I基准测试中均表现出色，在细粒度的空间和属性控制方面尤其突出。
URL
https://arxiv.org/abs/2506.00596
