Abstract
Mask-guided matting networks have achieved significant improvements and have shown great potential in practical applications in recent years. However, simply learning matting representation from synthetic and lack-of-real-world-diversity matting data, these approaches tend to overfit low-level details in wrong regions, lack generalization to objects with complex structures and real-world scenes such as shadows, as well as suffer from interference of background lines or textures. To address these challenges, in this paper, we propose a novel auxiliary learning framework for mask-guided matting models, incorporating three auxiliary tasks: semantic segmentation, edge detection, and background line detection besides matting, to learn different and effective representations from different types of data and annotations. Our framework and model introduce the following key aspects: (1) to learn real-world adaptive semantic representation for objects with diverse and complex structures under real-world scenes, we introduce extra semantic segmentation and edge detection tasks on more diverse real-world data with segmentation annotations; (2) to avoid overfitting on low-level details, we propose a module to utilize the inconsistency between learned segmentation and matting representations to regularize detail refinement; (3) we propose a novel background line detection task into our auxiliary learning framework, to suppress interference of background lines or textures. In addition, we propose a high-quality matting benchmark, Plant-Mat, to evaluate matting methods on complex structures. Extensively quantitative and qualitative results show that our approach outperforms state-of-the-art mask-guided methods.
Abstract (translated)
近年来，随着计算机图形学和计算机视觉领域的快速发展，遮罩引导的 matting 网络已经取得了显著的改进，并在实际应用中展示了巨大的潜力。然而，这些方法往往从合成和缺乏真实世界多样性的 matting 数据中学习 matting 表示，并倾向于在错误的区域过度拟合低级细节，缺乏对具有复杂结构和真实世界场景的对象的泛化，同时还受到背景线或纹理的干扰。为了应对这些挑战，在本文中，我们提出了一个名为“合议引导的遮罩指导模型”的新辅助学习框架，结合三个辅助任务：语义分割、边缘检测和背景线检测，从不同类型的数据和注释中学习不同的有效表示。我们的框架和模型包括以下关键方面： 1. 在真实世界场景中，为具有多样化和复杂结构的物体学习真实世界自适应语义表示，我们在更丰富的真实世界数据上进行语义分割和边缘检测，并具有分割注释； 2. 为了避免在低级细节上过拟合，我们引入了一个模块，利用学习和 matting 表示之间的不一致性来对细节进行规范； 3. 我们在辅助学习框架中引入了一个新的背景线检测任务，以抑制背景线或纹理的干扰。 此外，我们还提出了一个名为“植物-mat”的高质量 mat 基准，用于评估 mat 方法在复杂结构上的效果。广泛的定量结果和定性结果表明，我们的方法超越了最先进的遮罩引导方法。
URL
https://arxiv.org/abs/2403.19213
