Abstract
Feedback is essential for learning a new skill or improving one's current skill-level. However, current methods for skill-assessment from video only provide scores or compare demonstrations, leaving the burden of knowing what to do differently on the user. We introduce a novel method to generate actionable feedback from video of a person doing a physical activity, such as basketball or soccer. Our method takes a video demonstration and its accompanying 3D body pose and generates (1) free-form expert commentary describing what the person is doing well and what they could improve, and (2) a visual expert demonstration that incorporates the required corrections. We show how to leverage Ego-Exo4D's videos of skilled activity and expert commentary together with a strong language model to create a weakly-supervised training dataset for this task, and we devise a multimodal video-language model to infer coaching feedback. Our method is able to reason across multi-modal input combinations to output full-spectrum, actionable coaching -- expert commentary, expert video retrieval, and the first-of-its-kind expert pose generation -- outperforming strong vision-language models on both established metrics and human preference studies.
Abstract (translated)
反馈对于学习新技能或提高现有技能水平至关重要。然而,目前从视频进行技能评估的方法仅提供分数或比较演示,这使得用户不知道应该采取哪些不同的措施。我们提出了一种新颖的方法,可以从一个人做运动(如篮球或足球)的视频中生成具有行动able反馈的方法。我们的方法基于一个视频演示及其相应的3D身体姿势,生成(1)自由形式的专家评论,描述这个人做得好的地方以及可以改进的地方,和(2)视觉专家演示,包括所需的修正。我们展示了如何利用Ego-Exo4D的熟练活动视频和专家评论,以及强大的语言模型,为这项任务创建一个弱监督的训练数据集,并设计了一个多模态视频语言模型以推断教练反馈。我们的方法能够通过多模态输入组合进行推理,输出完整的、行动able教练——专家评论、专家视频检索,以及世界上独一无二的专家姿态生成,——超过在既定指标和人类偏好研究上 strong vision-language 模型。
URL
https://arxiv.org/abs/2408.00672
