Abstract
Holistic scene understanding includes semantic segmentation, surface normal estimation, object boundary detection, depth estimation, etc. The key aspect of this problem is to learn representation effectively, as each subtask builds upon not only correlated but also distinct attributes. Inspired by visual-prompt tuning, we propose a Task-Specific Prompts Transformer, dubbed TSP-Transformer, for holistic scene understanding. It features a vanilla transformer in the early stage and tasks-specific prompts transformer encoder in the lateral stage, where tasks-specific prompts are augmented. By doing so, the transformer layer learns the generic information from the shared parts and is endowed with task-specific capacity. First, the tasks-specific prompts serve as induced priors for each task effectively. Moreover, the task-specific prompts can be seen as switches to favor task-specific representation learning for different tasks. Extensive experiments on NYUD-v2 and PASCAL-Context show that our method achieves state-of-the-art performance, validating the effectiveness of our method for holistic scene understanding. We also provide our code in the following link this https URL.
Abstract (translated)
整体场景理解包括语义分割、表面法线估计、物体边界检测、深度估计等。这个问题的关键在于有效地学习表示，因为每个子任务不仅依赖于相关属性，而且还依赖于独特的属性。受到视觉提示调整的启发，我们提出了一个任务特定提示的Transformer，称之为TSP-Transformer，用于整体场景理解。它具有一个基本的变压器（在早期阶段）和一个任务特定提示的变压器（在横向阶段），其中任务特定提示被增强。通过这样做，变压器层从共享部分学习通用信息，并具备任务特定能力。首先，任务特定提示可以作为每个任务的诱发先验。此外，任务特定提示可以被视为对不同任务为任务特定表示学习提供开关。在NYUD-v2和PASCAL-Context的实验中，我们的方法实现了最先进的性能，验证了我们对整体场景理解的有效性。我们还在下面这个链接的代码中提供了我们的方法：<https://github.com/Vision_Transformer/TSP-Transformer>
URL
https://arxiv.org/abs/2311.03427
