Abstract
Few-shot segmentation is a task to segment objects or regions of novel classes within an image given only a few annotated examples. In the generalized setting, the task extends to segment both the base and the novel classes. The main challenge is how to train the model such that the addition of novel classes does not hurt the base classes performance, also known as catastrophic forgetting. To mitigate this issue, we use SegGPT as our base model and train it on the base classes. Then, we use separate learnable prompts to handle predictions for each novel class. To handle various object sizes which typically present in remote sensing domain, we perform patch-based prediction. To address the discontinuities along patch boundaries, we propose a patch-and-stitch technique by re-framing the problem as an image inpainting task. During inference, we also utilize image similarity search over image embeddings for prompt selection and novel class filtering to reduce false positive predictions. Based on our experiments, our proposed method boosts the weighted mIoU of a simple fine-tuned SegGPT from 15.96 to 35.08 on the validation set of few-shot OpenEarthMap dataset given in the challenge.
Abstract (translated)
少样本分割是在只有几篇注释示例的情况下，对图像中 novel 类别的对象或区域进行分割的任务。在扩展设置中，任务扩展到同时分割基础类和 novel 类别。主要挑战是训练模型，使得 novel 类别的添加不会损害基础类别的性能，也就是灾难性遗忘（catastrophic forgetting）。为了减轻这个问题，我们使用 SegGPT 作为基础模型，并在其基础上进行训练。然后，我们使用独立的可学习提示来处理每个 novel 类别的预测。为了处理遥感领域中通常存在的各种对象大小，我们采用基于补丁的预测。为了处理补丁边界上的不连续性，我们提出了通过重新将问题重构为图像修复任务来解决补丁和缝合技术。在推理过程中，我们还利用图像相似搜索来选择提示和过滤 novel 类别，以降低虚假阳性预测。根据我们的实验，我们对简单微调的 SegGPT 的加权 mIoU 在 few-shot OpenEarthMap 数据集的验证集上从 15.96 提高到了 35.08。
URL
https://arxiv.org/abs/2404.10307
