Abstract
Recent advances in Vision and Language Models (VLMs) have improved open-world 3D representation, facilitating 3D zero-shot capability in unseen categories. Existing open-world methods pre-train an extra 3D encoder to align features from 3D data (e.g., depth maps or point clouds) with CAD-rendered images and corresponding texts. However, the limited color and texture variations in CAD images can compromise the alignment robustness. Furthermore, the volume discrepancy between pre-training datasets of the 3D encoder and VLM leads to sub-optimal 2D to 3D knowledge transfer. To overcome these issues, we propose OpenDlign, a novel framework for learning open-world 3D representations, that leverages depth-aligned images generated from point cloud-projected depth maps. Unlike CAD-rendered images, our generated images provide rich, realistic color and texture diversity while preserving geometric and semantic consistency with the depth maps. OpenDlign also optimizes depth map projection and integrates depth-specific text prompts, improving 2D VLM knowledge adaptation for 3D learning efficient fine-tuning. Experimental results show that OpenDlign significantly outperforms existing benchmarks in zero-shot and few-shot 3D tasks, exceeding prior scores by 8.0% on ModelNet40 and 16.4% on OmniObject3D with just 6 million tuned parameters. Moreover, integrating generated depth-aligned images into existing 3D learning pipelines consistently improves their performance.
Abstract (translated)
近年来，在Vision和语言模型（VLMs）方面的进步已经提高了开放世界3D表示，推动了在未见类别的3D零击能力。现有的开放世界方法在预训练3D编码器时添加了一个额外的3D编码器，使其将来自3D数据（如深度图或点云）的特征与CAD渲染图像和相关文本对齐。然而，CAD图像中有限的颜色和纹理变化可能会削弱对齐稳健性。此外，预训练3D编码器数据集和VLM数据集之间的体积差异导致了2D到3D知识传递的低效。为了克服这些问题，我们提出了OpenDlign，一种学习开放世界3D表示的新框架，它利用点云投影得到的深度图生成的深度对齐图像。与CAD渲染图像不同，我们的生成图像在保持几何和语义一致性的同时，提供了丰富、逼真的颜色和纹理多样性。此外，OpenDlign还优化了深度图投影并集成了深度特定文本提示，提高了2D VLM对3D学习的知识迁移效率。实验结果表明，OpenDlign在零击和少击3D任务上显著优于现有基准，在仅600万调整参数的情况下，超过了ModelNet40和OmniObject3D的分数。此外，将生成的深度对齐图像集成到现有的3D学习流程中，显著提高了它们的性能。
URL
https://arxiv.org/abs/2404.16538
