Abstract
Recently, plain vision Transformers (ViTs) have shown impressive performance on various computer vision tasks, thanks to their strong modeling capacity and large-scale pretraining. However, they have not yet conquered the problem of image matting. We hypothesize that image matting could also be boosted by ViTs and present a new efficient and robust ViT-based matting system, named ViTMatte. Our method utilizes (i) a hybrid attention mechanism combined with a convolution neck to help ViTs achieve an excellent performance-computation trade-off in matting tasks. (ii) Additionally, we introduce the detail capture module, which just consists of simple lightweight convolutions to complement the detailed information required by matting. To the best of our knowledge, ViTMatte is the first work to unleash the potential of ViT on image matting with concise adaptation. It inherits many superior properties from ViT to matting, including various pretraining strategies, concise architecture design, and flexible inference strategies. We evaluate ViTMatte on Composition-1k and Distinctions-646, the most commonly used benchmark for image matting, our method achieves state-of-the-art performance and outperforms prior matting works by a large margin.
Abstract (translated)
最近，普通的视觉转换器(ViTs)在各种计算机视觉任务中表现出出色的性能，因为它们具有强大的建模能力和大规模的预训练。然而，他们还没有克服图像拼接的问题。我们假设图像拼接也可以由ViTs来提升，并提出了名为ViTMatte的新高效、可靠的ViT拼接系统。我们的方法和(i)采用混合注意力机制和卷积颈部来帮助ViTs在拼接任务中实现出色的性能-计算权衡。(ii)我们还引入了细节捕捉模块，它仅仅是简单的 lightweight 卷积来补充拼接任务所需的详细信息。据我们所知，ViTMatte是第一款通过简洁适应性释放ViT在图像拼接中的潜力的工作。它继承了ViT在拼接中许多优越的性质，包括各种预训练策略、简洁的建筑设计和灵活的推理策略。我们在Composition-1k和distinction-646等常用的图像拼接基准上评估了ViTMatte，我们的方法和之前的图像拼接工作相比实现了先进的性能，并大幅超越了之前的工作。
URL
https://arxiv.org/abs/2305.15272
