Abstract
In this paper, we present UniPose, a unified cross-modality pose prior propagation method for weakly supervised 3D human pose estimation (HPE) using unannotated single-view RGB-D sequences (RGB, depth, and point cloud data). UniPose transfers 2D HPE annotations from large-scale RGB datasets (e.g., MS COCO) to the 3D domain via self-supervised learning on easily acquired RGB-D sequences, eliminating the need for labor-intensive 3D keypoint annotations. This approach bridges the gap between 2D and 3D domains without suffering from issues related to multi-view camera calibration or synthetic-to-real data shifts. During training, UniPose leverages off-the-shelf 2D pose estimations as weak supervision for point cloud networks, incorporating spatial-temporal constraints like body symmetry and joint motion. The 2D-to-3D back-projection loss and cross-modality interaction further enhance this process. By treating the point cloud network's 3D HPE results as pseudo ground truth, our anchor-to-joint prediction method performs 3D lifting on RGB and depth networks, making it more robust against inaccuracies in 2D HPE results compared to state-of-the-art methods. Experiments on CMU Panoptic and ITOP datasets show that UniPose achieves comparable performance to fully supervised methods. Incorporating large-scale unlabeled data (e.g., NTU RGB+D 60) enhances its performance under challenging conditions, demonstrating its potential for practical applications. Our proposed 3D lifting method also achieves state-of-the-art results.
Abstract (translated)
在这篇论文中，我们介绍了UniPose，这是一种统一的跨模态姿态先验传播方法，用于使用未标注的单视图RGB-D序列（包括RGB、深度和点云数据）进行弱监督下的三维人体姿态估计（HPE）。UniPose通过在易于获取的RGB-D序列上进行自监督学习，将大规模RGB数据集（如MS COCO）中的2D HPE注释转移到3D领域，从而消除了劳动密集型3D关键点标注的需求。这种方法弥合了2D和3D领域的差距，并避免了与多视角相机校准或合成到现实的数据偏移相关的问题。 在训练过程中，UniPose利用现成的2D姿态估计作为弱监督信号，用于点云网络，并结合空间-时间约束（如身体对称性和关节运动）。2D至3D反投影损失和跨模态交互进一步增强了这一过程。通过将点云网络的3D HPE结果视为伪真实值，我们的锚点到关节预测方法在RGB和深度网络上执行3D提升操作，与最先进的方法相比，在处理2D HPE结果中的不准确性时更为鲁棒。 在CMU Panoptic和ITOP数据集上的实验表明，UniPose达到了与完全监督方法相当的性能。结合大规模未标注数据（如NTU RGB+D 60）提高了其在挑战性条件下的表现，展示了其实用应用的巨大潜力。我们提出的3D提升方法也取得了最新的成果。
URL
https://arxiv.org/abs/2509.23376
