Abstract
Human motion, with its inherent complexities, such as non-rigid deformations, articulated movements, clothing distortions, and frequent occlusions caused by limbs or other individuals, provides a rich and challenging source of supervision that is crucial for training robust and generalizable point trackers. Despite the suitability of human motion, acquiring extensive training data for point tracking remains difficult due to laborious manual annotation. Our proposed pipeline, AnthroTAP, addresses this by proposing an automated pipeline to generate pseudo-labeled training data, leveraging the Skinned Multi-Person Linear (SMPL) model. We first fit the SMPL model to detected humans in video frames, project the resulting 3D mesh vertices onto 2D image planes to generate pseudo-trajectories, handle occlusions using ray-casting, and filter out unreliable tracks based on optical flow consistency. A point tracking model trained on AnthroTAP annotated dataset achieves state-of-the-art performance on the TAP-Vid benchmark, surpassing other models trained on real videos while using 10,000 times less data and only 1 day in 4 GPUs, compared to 256 GPUs used in recent state-of-the-art.
Abstract (translated)
人类运动的固有复杂性，包括非刚体变形、关节活动、衣物扭曲以及由于肢体或其他个体造成的频繁遮挡，为训练稳健且通用的点跟踪器提供了丰富而具有挑战性的监督来源。尽管人体运动非常适合此类任务，但由于手动标注工作量大，获取广泛的训练数据仍然困难重重。 我们提出的AnthroTAP管道通过提出一个自动化流程来生成伪标签训练数据，解决了这个问题，该流程利用了Skinned Multi-Person Linear (SMPL)模型。具体而言，我们首先将SMPL模型拟合到视频帧中检测到的人体上，然后将生成的3D网格顶点投影到2D图像平面上以产生伪轨迹，并使用光线投射技术处理遮挡问题。此外，基于光流一致性原则过滤掉不可靠的跟踪结果。 在AnthroTAP标注的数据集上训练的点跟踪模型，在TAP-Vid基准测试中达到了最先进的性能水平。与那些需要256个GPU进行长时间训练并在真实视频数据上达到最佳表现的模型相比，我们的模型仅使用4个GPU，并且在一个工作日内就完成了10,000倍少的数据量的学习过程。 这一成果表明了在点跟踪任务中利用伪标签生成技术的有效性以及其在提高计算效率和减少标注成本方面的潜力。
URL
https://arxiv.org/abs/2507.06233
