Abstract
Current large-scale video datasets focus on general human activity, but lack depth of coverage on fine-grained activities needed to address physical skill learning. We introduce SportSkills, the first large-scale sports dataset geared towards physical skill learning with in-the-wild video. SportSkills has more than 360k instructional videos containing more than 630k visual demonstrations paired with instructional narrations explaining the know-how behind the actions from 55 varied sports. Through a suite of experiments, we show that SportSkills unlocks the ability to understand fine-grained differences between physical actions. Our representation achieves gains of up to 4x with the same model trained on traditional activity-centric datasets. Crucially, building on SportSkills, we introduce the first large-scale task formulation of mistake-conditioned instructional video retrieval, bridging representation learning and actionable feedback generation (e.g., "here's my execution of a skill; which video clip should I watch to improve it?"). Formal evaluations by professional coaches show our retrieval approach significantly advances the ability of video models to personalize visual instructions for a user query.
Abstract (translated)
当前大规模视频数据集聚焦于通用人类活动，但在物理技能学习所需的细粒度动作覆盖深度上存在不足。我们推出SportSkills，首个面向物理技能学习的大规模体育数据集，包含真实场景视频。该数据集拥有超过36万条教学视频，涵盖55项不同运动，提供超过63万条视觉演示，并配有解释动作背后技巧的教学旁白。通过一系列实验，我们证明SportSkills使模型能够理解物理动作间的细粒度差异。在相同模型下，相较于传统以活动为中心的数据集，我们的表征方法最高可提升4倍性能。关键的是，基于SportSkills，我们首次提出了大规模“基于错误条件的教学视频检索”任务框架，将表征学习与可操作反馈生成相连接（例如：“这是我的技能执行录像；我应该观看哪个视频片段来改进？”）。专业教练的正式评估表明，我们的检索方法显著提升了视频模型对用户查询进行个性化视觉指导的能力。
URL
https://arxiv.org/abs/2603.25163
