Abstract
We present a new self-supervised paradigm on point cloud sequence understanding. Inspired by the discriminative and generative self-supervised methods, we design two tasks, namely point cloud sequence based Contrastive Prediction and Reconstruction (CPR), to collaboratively learn more comprehensive spatiotemporal representations. Specifically, dense point cloud segments are first input into an encoder to extract embeddings. All but the last ones are then aggregated by a context-aware autoregressor to make predictions for the last target segment. Towards the goal of modeling multi-granularity structures, local and global contrastive learning are performed between predictions and targets. To further improve the generalization of representations, the predictions are also utilized to reconstruct raw point cloud sequences by a decoder, where point cloud colorization is employed to discriminate against different frames. By combining classic contrast and reconstruction paradigms, it makes the learned representations with both global discrimination and local perception. We conduct experiments on four point cloud sequence benchmarks, and report the results on action recognition and gesture recognition under multiple experimental settings. The performances are comparable with supervised methods and show powerful transferability.
Abstract (translated)
我们提出了一种新的点云序列理解自监督范式。借鉴了具有选择和生成自监督方法，我们设计了两个任务，即点云序列基于对比预测和重建(CPR)，旨在协作学习更全面的时间空间表示。具体来说，密集点云片段首先输入到编码器中以提取嵌入。除了最后一个片段，所有片段都被基于上下文自回归器聚合起来，以预测最后一个目标片段。为了建模多粒度结构，在预测和目标之间进行 local 和 global 对比学习。为了进一步提高表示的泛化能力，预测也被用于解码器中重建原始点云序列，其中点云颜色化用于区分不同的帧。通过结合经典对比和重建范式，它使学习到的表示具有全球区分性和 local 感知。我们对四个点云序列基准进行了实验，并在不同的实验设置下报告了动作识别和手势识别的结果。表现与监督方法相当，表现出强大的转移性。
URL
https://arxiv.org/abs/2305.12959
