Abstract
Point cloud video understanding is critical for robotics as it accurately encodes motion and scene interaction. We recognize that 4D datasets are far scarcer than 3D ones, which hampers the scalability of self-supervised 4D models. A promising alternative is to transfer 3D pre-trained models to 4D perception tasks. However, rigorous empirical analysis reveals two critical limitations that impede transfer capability: overfitting and the modality gap. To overcome these challenges, we develop a novel "Align then Adapt" (PointATA) paradigm that decomposes parameter-efficient transfer learning into two sequential stages. Optimal-transport theory is employed to quantify the distributional discrepancy between 3D and 4D datasets, enabling our proposed point align embedder to be trained in Stage 1 to alleviate the underlying modality gap. To mitigate overfitting, an efficient point-video adapter and a spatial-context encoder are integrated into the frozen 3D backbone to enhance temporal modeling capacity in Stage 2. Notably, with the above engineering-oriented designs, PointATA enables a pre-trained 3D model without temporal knowledge to reason about dynamic video content at a smaller parameter cost compared to previous work. Extensive experiments show that PointATA can match or even outperform strong full fine-tuning models, whilst enjoying the advantage of parameter efficiency, e.g. 97.21 \% accuracy on 3D action recognition, $+8.7 \%$ on 4 D action segmentation, and 84.06\% on 4D semantic segmentation.
Abstract (translated)
点云视频理解在机器人技术中至关重要，因为它能准确地编码运动和场景交互。我们认识到，相较于3D数据集而言，4D数据集稀缺得多，这阻碍了自监督4D模型的可扩展性。一种有前景的替代方案是从3D预训练模型迁移至4D感知任务。然而，严格的实证分析揭示出两种关键限制，即过度拟合和模态差距问题，这些问题阻碍了迁移能力。 为了解决这些挑战，我们开发了一种新颖的“对齐后适应”（PointATA）范式，将参数效率高的迁移学习分解成两个连续阶段。利用最优传输理论来量化3D与4D数据集之间的分布差异，使我们的提议点对齐嵌入器能够在第一阶段训练以缓解潜在的模态差距问题。为减轻过度拟合，在第二阶段集成高效的点视频适配器和空间上下文编码器到冻结的3D主干中，从而增强时间建模能力。 值得注意的是，通过上述工程设计，PointATA使一个无时间知识的预训练3D模型能够以较小的参数成本推理出动态视频内容。广泛的实验表明，PointATA可以与强大的全微调模型相匹配甚至超越其性能，同时享受参数效率的优势，例如在3D动作识别中达到97.21%的准确率，在4D动作分割中提高8.7%，以及在4D语义分割中达到84.06%的成绩。
URL
https://arxiv.org/abs/2602.23069
