Abstract
We introduce the notion of point affiliation into feature upsampling. By abstracting a feature map into non-overlapped semantic clusters formed by points of identical semantic meaning, feature upsampling can be viewed as point affiliation -- designating a semantic cluster for each upsampled point. In the framework of kernel-based dynamic upsampling, we show that an upsampled point can resort to its low-res decoder neighbors and high-res encoder point to reason the affiliation, conditioned on the mutual similarity between them. We therefore present a generic formulation for generating similarity-aware upsampling kernels and prove that such kernels encourage not only semantic smoothness but also boundary sharpness. This formulation constitutes a novel, lightweight, and universal upsampling solution, Similarity-Aware Point Affiliation (SAPA). We show its working mechanism via our preliminary designs with window-shape kernel. After probing the limitations of the designs on object detection, we reveal additional insights for upsampling, leading to SAPA with the dynamic kernel shape. Extensive experiments demonstrate that SAPA outperforms prior upsamplers and invites consistent performance improvements on a number of dense prediction tasks, including semantic segmentation, object detection, instance segmentation, panoptic segmentation, image matting, and depth estimation. Code is made available at: this https URL
Abstract (translated)
我们引入了点关联的概念,将其引入特征插值。通过将特征映射抽象为由具有相同语义意义的点组成的不重叠的语义簇,特征插值可以被视为点关联——为每个插值点指定一个语义簇。在基于内核的动态插值框架中,我们表明,插值点可以通过其低分辨率解码邻居和高分辨率编码点推理关联,根据它们之间的相互相似性条件。因此,我们提出了一种通用表达式来生成具有相似性感知特征插值内核,并证明了这种内核不仅鼓励语义平滑,还鼓励边界尖锐化。这种表达式构成了一种新型、轻便且通用的特征插值解决方案,称为相似性点关联(SAPA)。我们通过窗口形状内核的初步设计展示了其工作原理。在测试对象检测的设计限制后,我们揭示了增加插值点额外见解的方法,导致动态内核形状的SAPA。广泛的实验结果表明,SAPA优于先前的插值方案,并在许多密集预测任务中表现出一致的性能改进,包括语义分割、对象检测、实例分割、全景分割、图像拼接和深度估计。代码在此https URL上提供。
URL
https://arxiv.org/abs/2307.08198
