Abstract
Large-scale vision 2D vision language models, such as CLIP can be aligned with a 3D encoder to learn generalizable (open-vocabulary) 3D vision models. However, current methods require supervised pre-training for such alignment, and the performance of such 3D zero-shot models remains sub-optimal for real-world adaptation. In this work, we propose an optimization framework: Cross-MoST: Cross-Modal Self-Training, to improve the label-free classification performance of a zero-shot 3D vision model by simply leveraging unlabeled 3D data and their accompanying 2D views. We propose a student-teacher framework to simultaneously process 2D views and 3D point clouds and generate joint pseudo labels to train a classifier and guide cross-model feature alignment. Thereby we demonstrate that 2D vision language models such as CLIP can be used to complement 3D representation learning to improve classification performance without the need for expensive class annotations. Using synthetic and real-world 3D datasets, we further demonstrate that Cross-MoST enables efficient cross-modal knowledge exchange resulting in both image and point cloud modalities learning from each other's rich representations.
Abstract (translated)
大规模视觉2D视觉语言模型，如CLIP，可以通过与3D编码器对齐来学习具有通用的（开放词汇）3D视觉模型。然而，现有的方法需要进行有监督的预训练才能实现这种对齐，而这种3D零 shot模型的性能在现实世界适应中仍然存在很大的提升空间。在这项工作中，我们提出了一个优化框架：Cross-MoST：跨模态自训练，通过简单地利用未标注的3D数据及其相关2D视图来提高零 shot 3D视觉模型的无标签分类性能。我们提出了一个学生-教师框架来同时处理2D视图和3D点云，并生成联合伪标签来训练分类器和指导跨模型特征对齐。因此，我们证明了CLIP这样的2D视觉语言模型可以作为补充3D表示学习来提高分类性能，而无需进行昂贵的类注释。使用合成和真实世界3D数据集，我们进一步证明了Cross-MoST能够实现高效的跨模态知识交流，从而实现图像和点云模态之间的知识学习。
URL
https://arxiv.org/abs/2404.10146
