Abstract
Current remote-sensing interpretation models often focus on a single task such as detection, segmentation, or caption. However, the task-specific designed models are unattainable to achieve the comprehensive multi-level interpretation of images. The field also lacks support for multi-task joint interpretation datasets. In this paper, we propose Panoptic Perception, a novel task and a new fine-grained dataset (FineGrip) to achieve a more thorough and universal interpretation for RSIs. The new task, 1) integrates pixel-level, instance-level, and image-level information for universal image perception, 2) captures image information from coarse to fine granularity, achieving deeper scene understanding and description, and 3) enables various independent tasks to complement and enhance each other through multi-task learning. By emphasizing multi-task interactions and the consistency of perception results, this task enables the simultaneous processing of fine-grained foreground instance segmentation, background semantic segmentation, and global fine-grained image captioning. Concretely, the FineGrip dataset includes 2,649 remote sensing images, 12,054 fine-grained instance segmentation masks belonging to 20 foreground things categories, 7,599 background semantic masks for 5 stuff classes and 13,245 captioning sentences. Furthermore, we propose a joint optimization-based panoptic perception model. Experimental results on FineGrip demonstrate the feasibility of the panoptic perception task and the beneficial effect of multi-task joint optimization on individual tasks. The dataset will be publicly available.
Abstract (translated)
目前，遥感解释模型通常集中于单一任务，如检测、分割或注释。然而，针对任务的定制模型对于实现图像的全面多层次解释是不现实的。该领域也缺乏支持多任务联合解释数据集。在本文中，我们提出了全面感知（Panoptic Perception），一种新的任务和新精细数据集（FineGrip），以实现对RSIs的更全面和通用的解释。新任务1）将像素级别、实例级别和图像级别信息集成到通用图像感知中，实现更深层次的场景理解和描述；2）从粗到细粒度捕捉图像信息，实现更深的场景理解和描述；3）通过多任务学习使各种独立任务互补和增强。通过强调多任务交互和感知结果的一致性，这项任务实现了同时处理细粒度前景实例分割、背景语义分割和全局细粒度图像注释的同时处理。具体来说，FineGrip数据集包括2,649个遥感图像，20个前景类别的细粒度实例分割掩码，5个类别和13,245个注释句子。此外，我们提出了一个基于联合优化方案的全面感知模型。FineGrip实验结果表明，全面感知任务是可行的，多任务联合优化对各个任务具有积极影响。该数据集将公开发布。
URL
https://arxiv.org/abs/2404.04608
