Abstract
Many modern video-based human action recognition (HAR) approaches use 2D skeleton as the intermediate representation in their prediction pipelines. Despite overall encouraging results, these approaches still struggle in many common scenes, mainly because the skeleton does not capture critical action-related information pertaining to the depth of the joints, contour of the human body, and interaction between the human and objects. To address this, we propose an effective approach to augment skeleton with a representation capturing action-related information in the pipeline of HAR. The representation, termed Scale-Body-Flow (SBF), consists of three distinct components, namely a scale map volume given by the scale (and hence depth information) of each joint, a body map outlining the human subject, and a flow map indicating human-object interaction given by pixel-wise optical flow values. To predict SBF, we further present SFSNet, a novel segmentation network supervised by the skeleton and optical flow without extra annotation overhead beyond the existing skeleton extraction. Extensive experiments across different datasets demonstrate that our pipeline based on SBF and SFSNet achieves significantly higher HAR accuracy with similar compactness and efficiency as compared with the state-of-the-art skeleton-only approaches.
Abstract (translated)
许多现代基于视频的人体动作识别（HAR）方法在预测流程中使用2D骨架作为中间表示。尽管整体结果令人鼓舞，但这些方法在许多常见场景中仍面临挑战，主要原因在于骨架无法捕捉与关节深度、人体轮廓以及人与物体交互相关的关键动作信息。为解决这一问题，我们提出了一种有效方法，在HAR流程中通过一种能捕捉动作相关信息的表示来增强骨架。该表示称为尺度-体素-流（SBF），包含三个不同组件：由每个关节的尺度（因而包含深度信息）给出的尺度图体积、勾勒人体主体的体素图，以及通过像素级光流值表示的人-物交互流图。为预测SBF，我们进一步提出了SFSNet——一种通过骨架和光流进行监督的新型分割网络，无需在现有骨架提取之外增加额外标注开销。在不同数据集上的大量实验表明，基于SBF和SFSNet的流程相比最先进的纯骨架方法，在保持相似紧凑性和效率的同时，显著提升了HAR准确率。
URL
https://arxiv.org/abs/2604.03590
