Abstract
Point cloud video representation learning is primarily built upon the masking strategy in a self-supervised manner. However, the progress is slow due to several significant challenges: (1) existing methods learn the motion particularly with hand-crafted designs, leading to unsatisfactory motion patterns during pre-training which are non-transferable on fine-tuning scenarios. (2) previous Masked AutoEncoder (MAE) frameworks are limited in resolving the huge representation gap inherent in 4D data. In this study, we introduce the first self-disentangled MAE for learning discriminative 4D representations in the pre-training stage. To address the first challenge, we propose to model the motion representation in a latent space. The second issue is resolved by introducing the latent tokens along with the typical geometry tokens to disentangle high-level and low-level features during decoding. Extensive experiments on MSR-Action3D, NTU-RGBD, HOI4D, NvGesture, and SHREC'17 verify this self-disentangled learning framework. We demonstrate that it can boost the fine-tuning performance on all 4D tasks, which we term Uni4D. Our pre-trained model presents discriminative and meaningful 4D representations, particularly benefits processing long videos, as Uni4D gets $+3.8\%$ segmentation accuracy on HOI4D, significantly outperforming either self-supervised or fully-supervised methods after end-to-end fine-tuning.
Abstract (translated)
点云视频表示学习主要基于自监督方式下的掩码策略构建。然而，由于几个显著的挑战，其进展缓慢：（1）现有方法通过手工设计来学习运动模式，在预训练阶段产生了不令人满意的、在微调场景中不可转移的运动模式。（2）之前的遮罩自动编码器(Masked AutoEncoder, MAE)框架难以解决4D数据固有的巨大表示差距。在此研究中，我们引入了第一个自我解耦式MAE来学习鉴别性的4D表示，并应用于预训练阶段。为了解决第一项挑战，我们在潜在空间中建模运动表示；第二项问题通过在解码过程中加入潜在令牌与典型几何令牌一起使用的方法得到解决，从而实现了高级和低级特征的分离。在MSR-Action3D、NTU-RGBD、HOI4D、NvGesture以及SHREC'17数据集上的广泛实验验证了这种自我解耦式学习框架的有效性。我们展示了它可以在所有4D任务上提升微调性能，我们将这一模型称为Uni4D。我们的预训练模型呈现出具有鉴别性和意义的4D表示，特别是对于处理长视频非常有益，因为Uni4D在HOI4D上的分割准确率提高了3.8%，在端到端微调后显著优于任何自我监督或全监督方法。
URL
https://arxiv.org/abs/2504.04837
