Abstract
In recent years, zero-shot learning has attracted the focus of many researchers, due to its flexibility and generality. Many approaches have been proposed to achieve the zero-shot classification of the point clouds for 3D object understanding, following the schema of CLIP. However, in the real world, the point clouds could be extremely sparse, dramatically limiting the effectiveness of the 3D point cloud encoders, and resulting in the misalignment of point cloud features and text embeddings. To the point cloud encoders to fit the extremely sparse point clouds without re-running the pre-training procedure which could be time-consuming and expensive, in this work, we propose an unsupervised model adaptation approach to enhance the point cloud encoder for the extremely sparse point clouds. We propose a novel fused-cross attention layer that expands the pre-trained self-attention layer with additional learnable tokens and attention blocks, which effectively modifies the point cloud features while maintaining the alignment between point cloud features and text embeddings. We also propose a complementary learning-based self-distillation schema that encourages the modified features to be pulled apart from the irrelevant text embeddings without overfitting the feature space to the observed text embeddings. Extensive experiments demonstrate that the proposed approach effectively increases the zero-shot capability on extremely sparse point clouds, and overwhelms other state-of-the-art model adaptation approaches.
Abstract (translated)
近年来，由于其灵活性和普适性，零样本学习（Zero-Shot Learning）吸引了许多研究人员的关注。为了实现3D物体理解中点云的零样本分类，许多方法提出了基于CLIP的方案。然而，在现实生活中，点云可能非常稀疏，极大地限制了3D点云编码器的有效性，并导致点云特征与文本嵌入之间的不匹配。为了适应稀疏的点云，避免重新进行预训练，我们在这个工作中提出了一个无监督的模型适应方法，以增强适应稀疏点云的点云编码器。我们提出了一个新颖的融合跨注意层，通过增加可学习标记和注意力模块，扩展了预训练的自注意力层，有效地修改点云特征，同时保持点云特征与文本嵌入之间的对齐。我们还提出了一个基于互补学习的自监督损失模式，鼓励修改后的特征从相关的文本嵌入中分离出来，以避免对特征空间对观察到的文本嵌入过拟合。大量实验证明，与最先进的模型适应方法相比，所提出的方案在稀疏点云上显著提高了零样本能力，并超越了其他方法。
URL
https://arxiv.org/abs/2404.19639
