Abstract
Procedure step recognition (PSR) aims to identify all correctly completed steps and their sequential order in videos of procedural tasks. The existing state-of-the-art models rely solely on detecting assembly object states in individual video frames. By neglecting temporal features, model robustness and accuracy are limited, especially when objects are partially occluded. To overcome these limitations, we propose Spatio-Temporal Occlusion-Resilient Modeling for Procedure Step Recognition (STORM-PSR), a dual-stream framework for PSR that leverages both spatial and temporal features. The assembly state detection stream operates effectively with unobstructed views of the object, while the spatio-temporal stream captures both spatial and temporal features to recognize step completions even under partial occlusion. This stream includes a spatial encoder, pre-trained using a novel weakly supervised approach to capture meaningful spatial representations, and a transformer-based temporal encoder that learns how these spatial features relate over time. STORM-PSR is evaluated on the MECCANO and IndustReal datasets, reducing the average delay between actual and predicted assembly step completions by 11.2% and 26.1%, respectively, compared to prior methods. We demonstrate that this reduction in delay is driven by the spatio-temporal stream, which does not rely on unobstructed views of the object to infer completed steps. The code for STORM-PSR, along with the newly annotated MECCANO labels, is made publicly available at this https URL .
Abstract (translated)
步骤识别（PSR）的目标是确定程序任务视频中所有正确完成的步骤及其顺序。现有最先进的模型仅依赖于检测单个视频帧中的装配物体状态。由于忽略了时间特征，这些模型在面对部分遮挡情况时的鲁棒性和准确性受到限制。为克服这一局限性，我们提出了时空抗遮挡模型（STORM-PSR），这是一种用于程序步骤识别的双流框架，结合了空间和时间特征的优势。 装配状态检测流能够有效地处理物体未受阻碍的情况，而时空流则捕捉了时间和空间特性，在部分遮挡的情况下也能识别出步骤完成情况。这一流程包括一个通过新型弱监督方法预训练的空间编码器，用于捕获有意义的空间表示；以及一个基于变压器的时间编码器，学习这些空间特征随时间的关系。 STORM-PSR在MECCANO和IndustReal数据集上进行了评估，在预测装配步骤完成的平均延迟方面相比先前的方法分别减少了11.2%和26.1%。我们展示了这种延迟减少是由时空流驱动的，该流程不需要物体未受阻碍的视角来推断已完成的步骤。 STORM-PSR的代码以及新标注的MECCANO标签已在以下URL公开提供：[这里应插入链接]（原文中的具体网址需要根据实际情况填写）。
URL
https://arxiv.org/abs/2510.12385
