Abstract
Broad-scale marine surveys performed by underwater vehicles significantly increase the availability of coral reef imagery, however it is costly and time-consuming for domain experts to label images. Point label propagation is an approach used to leverage existing image data labeled with sparse point labels. The resulting augmented ground truth generated is then used to train a semantic segmentation model. Here, we first demonstrate that recent advances in foundation models enable generation of multi-species coral augmented ground truth masks using denoised DINOv2 features and K-Nearest Neighbors (KNN), without the need for any pre-training or custom-designed algorithms. For extremely sparsely labeled images, we propose a labeling regime based on human-in-the-loop principles, resulting in significant improvement in annotation efficiency: If only 5 point labels per image are available, our proposed human-in-the-loop approach improves on the state-of-the-art by 17.3% for pixel accuracy and 22.6% for mIoU; and by 10.6% and 19.1% when 10 point labels per image are available. Even if the human-in-the-loop labeling regime is not used, the denoised DINOv2 features with a KNN outperforms the prior state-of-the-art by 3.5% for pixel accuracy and 5.7% for mIoU (5 grid points). We also provide a detailed analysis of how point labeling style and the quantity of points per image affects the point label propagation quality and provide general recommendations on maximizing point label efficiency.
Abstract (translated)
广泛的海洋调查通过水下机器人进行的珊瑚礁图像调查显著增加了珊瑚礁图像的可用性，然而领域专家花时间和精力来标注图像是非常昂贵和费时的。点标签传播是一种利用带有稀疏点标签的现有图像数据来训练模型的方法。然后，生成的增强真实值被用于训练语义分割模型。在这里，我们首先证明， recent advances in foundation models enable the generation of multi-species coral augmented ground truth masks using denoised DINOv2 features and K-Nearest Neighbors (KNN)，而无需进行预训练或自定义算法。对于极度稀疏的标注图像，我们提出了基于人类闭环原则的标注方案，从而在像素准确度和mIoU方面实现了显著的改进：如果每张图片只有5个点标签可用，我们的人类闭环方法在像素准确度和mIoU方面的表现与最先进的水平相当，相差17.3%；而当每张图片有10个点标签时，相差22.6%。即使不使用人类闭环标注方案， denoised DINOv2 features with KNN也超越了之前的状态水平，在像素准确度和mIoU（5个网格点）方面分别提高了3.5%和5.7%。我们还对点标记样式和每张图片的点数对点标记传播质量的影响进行了详细分析，并提供了关于最大化点标记效率的一般建议。
URL
https://arxiv.org/abs/2404.09406
