Abstract
Semantic segmentation plays a crucial role in enabling comprehensive scene understanding for robotic systems. However, generating annotations is challenging, requiring labels for every pixel in an image. In scenarios like autonomous driving, there's a need to progressively incorporate new classes as the operating environment of the deployed agent becomes more complex. For enhanced annotation efficiency, ideally, only pixels belonging to new classes would be annotated. This approach is known as Continual Semantic Segmentation (CSS). Besides the common problem of classical catastrophic forgetting in the continual learning setting, CSS suffers from the inherent ambiguity of the background, a phenomenon we refer to as the "background shift'', since pixels labeled as background could correspond to future classes (forward background shift) or previous classes (backward background shift). As a result, continual learning approaches tend to fail. This paper proposes a Backward Background Shift Detector (BACS) to detect previously observed classes based on their distance in the latent space from the foreground centroids of previous steps. Moreover, we propose a modified version of the cross-entropy loss function, incorporating the BACS detector to down-weight background pixels associated with formerly observed classes. To combat catastrophic forgetting, we employ masked feature distillation alongside dark experience replay. Additionally, our approach includes a transformer decoder capable of adjusting to new classes without necessitating an additional classification head. We validate BACS's superior performance over existing state-of-the-art methods on standard CSS benchmarks.
Abstract (translated)
语义分割在使机器人系统全面理解场景方面扮演了关键角色。然而，生成注释是具有挑战性的，需要为图像的每个像素提供标签。在自动驾驶等场景中，需要随着部署代理的操作环境变得越来越复杂而逐步引入新的类别。为了提高注释效率，理想情况下，只注释属于新类别的像素。这种方法被称为连续语义分割（CSS）。 除了在连续学习设置中经典灾难性遗忘的问题之外，CSS还受到背景固有不确定性的困扰，这种现象我们称之为“背景漂移”，因为被标注为背景的像素可能对应未来的类别（前向背景漂移）或过去的类别（后向背景漂移）。因此，连续学习方法往往失败。本文提出了一种基于背景漂移检测器（BACS）的Backward Background Shift Detector（BACS）来检测以前观察到的类别，根据它们在先前步骤中前景集中点的距离。此外，我们还提出了一个修改过的交叉熵损失函数，该函数包含BACS检测器，用于减轻与以前观察到的类别相关的背景像素的权重。 为了应对灾难性遗忘，我们还采用遮罩特征蒸馏以及暗经验回放。此外，我们的方法包括一个能够适应新类别的Transformer解码器，而无需增加额外的分类头。我们在标准CSS基准测试上验证BACS的卓越性能。
URL
https://arxiv.org/abs/2404.13148
