Abstract
As powerful generative models, text-to-image diffusion models have recently been explored for discriminative tasks. A line of research focuses on adapting a pre-trained diffusion model to semantic segmentation without any further training, leading to what training-free diffusion segmentors. These methods typically rely on cross-attention maps from the model's attention layers, which are assumed to capture semantic relationships between image pixels and text tokens. Ideally, such approaches should benefit from more powerful diffusion models, i.e., stronger generative capability should lead to better segmentation. However, we observe that existing methods often fail to scale accordingly. To understand this issue, we identify two underlying gaps: (i) cross-attention is computed across multiple heads and layers, but there exists a discrepancy between these individual attention maps and a unified global representation. (ii) Even when a global map is available, it does not directly translate to accurate semantic correlation for segmentation, due to score imbalances among different text tokens. To bridge these gaps, we propose two techniques: auto aggregation and per-pixel rescaling, which together enable training-free segmentation to better leverage generative capability. We evaluate our approach on standard semantic segmentation benchmarks and further integrate it into a generative technique, demonstrating both improved performance broad applicability. Codes are at this https URL.
Abstract (translated)
作为强大的生成模型，文本到图像的扩散模型最近被探索用于判别任务。一项研究重点是将预训练的扩散模型适应于语义分割任务而不进行进一步的训练，从而产生无需训练的扩散分割器（training-free diffusion segmentors）。这些方法通常依赖于来自模型注意层的交叉注意力图，假设它们捕获了图像像素和文本标记之间的语义关系。理想情况下，这样的方法应该受益于更强大的扩散模型，即更强的生成能力应导致更好的分割效果。然而，我们观察到现有方法往往无法相应地扩展。为了理解这一问题，我们确定了两个潜在差距：(i) 交叉注意力在多个头和层之间进行计算，但这些单独的注意图与统一的整体表示之间存在差异。(ii) 即使可以获取整体映射，在分割过程中它也无法直接转化为准确的语义相关性，由于不同文本标记之间的评分不平衡。为了弥合这两个差距，我们提出了两种技术：自动聚合（auto aggregation）和逐像素重缩放（per-pixel rescaling），这些技术结合使用可以使无训练分割更好地利用生成能力。我们在标准语义分割基准上评估了我们的方法，并将其进一步集成到一种生成技术中，展示了改进的性能和广泛的应用性。代码可在以下链接找到：[请在此处插入实际URL]。
URL
https://arxiv.org/abs/2603.06178
