Abstract
Large vision-language models have impressively promote the performance of 2D visual recognition under zero/few-shot scenarios. In this paper, we focus on exploiting the large vision-language model, i.e., CLIP, to address zero/few-shot 3D shape recognition based on multi-view representations. The key challenge for both tasks is to generate a discriminative descriptor of the 3D shape represented by multiple view images under the scenarios of either without explicit training (zero-shot 3D shape recognition) or training with a limited number of data (few-shot 3D shape recognition). We analyze that both tasks are relevant and can be considered simultaneously. Specifically, leveraging the descriptor which is effective for zero-shot inference to guide the tuning of the aggregated descriptor under the few-shot training can significantly improve the few-shot learning efficacy. Hence, we propose Prompt-Enhanced View Aggregation Network (PEVA-Net) to simultaneously address zero/few-shot 3D shape recognition. Under the zero-shot scenario, we propose to leverage the prompts built up from candidate categories to enhance the aggregation process of multiple view-associated visual features. The resulting aggregated feature serves for effective zero-shot recognition of the 3D shapes. Under the few-shot scenario, we first exploit a transformer encoder to aggregate the view-associated visual features into a global descriptor. To tune the encoder, together with the main classification loss, we propose a self-distillation scheme via a feature distillation loss by treating the zero-shot descriptor as the guidance signal for the few-shot descriptor. This scheme can significantly enhance the few-shot learning efficacy.
Abstract (translated)
大型视觉语言模型在零/少样本场景中显著提高了2D视觉识别的表现。在本文中，我们将重点利用CLIP（大型视觉语言模型）来解决基于多视图表示的零/少样本3D形状识别。这两个任务的关键挑战是在没有明确训练（零样本3D形状识别）或有限数据训练（少样本3D形状识别）的场景中生成对多视图表示的3D形状的区分性描述。我们分析认为，这两个任务是相关的，可以同时考虑。具体来说，利用在零样本推理中有效的描述器来引导在少样本训练中聚合描述符可以显著提高少样本学习效果。因此，我们提出了Prompt-Enhanced View Aggregation Network（PEVA-Net）来同时解决零/少样本3D形状识别。在零样本场景中，我们希望通过利用从候选类别中构建的提示来增强多视图相关视觉特征的聚合过程。生成的聚合特征可用于有效的零样本3D形状识别。在少样本场景中，我们首先利用Transformer编码器将视图相关视觉特征聚合为全局描述符。为了调整编码器，我们通过一个通过特征蒸馏损失对零样本描述进行自监督学习的方案来提出一个自监督学习方案。这个方案可以显著增强少样本学习效果。
URL
https://arxiv.org/abs/2404.19168
