Abstract
As extended reality (XR) is redefining how users interact with computing devices, research in human action recognition is gaining prominence. Typically, models deployed on immersive computing devices are static and limited to their default set of classes. The goal of our research is to provide users and developers with the capability to personalize their experience by adding new action classes to their device models continually. Importantly, a user should be able to add new classes in a low-shot and efficient manner, while this process should not require storing or replaying any of user's sensitive training data. We formalize this problem as privacy-aware few-shot continual action recognition. Towards this end, we propose POET: Prompt-Offset Tuning. While existing prompt tuning approaches have shown great promise for continual learning of image, text, and video modalities; they demand access to extensively pretrained transformers. Breaking away from this assumption, POET demonstrates the efficacy of prompt tuning a significantly lightweight backbone, pretrained exclusively on the base class data. We propose a novel spatio-temporal learnable prompt offset tuning approach, and are the first to apply such prompt tuning to Graph Neural Networks. We contribute two new benchmarks for our new problem setting in human action recognition: (i) NTU RGB+D dataset for activity recognition, and (ii) SHREC-2017 dataset for hand gesture recognition. We find that POET consistently outperforms comprehensive benchmarks. Source code at this https URL.
Abstract (translated)
随着扩展现实（XR）重新定义了用户与计算设备的交互方式，人类动作识别领域的研究正变得越来越重要。传统上，在沉浸式计算设备中部署的模型是静态的，并且局限于其默认的一组类。我们的研究目标是为用户提供能力，让他们能够通过向设备模型持续添加新的动作类别来个性化他们的体验。至关重要的是，用户应当能够在低样本和高效的方式下添加新类别，而这一过程不应涉及存储或重放任何用户的敏感训练数据。我们将这个问题形式化为隐私感知的少样本连续动作识别问题。 为此，我们提出了POET：Prompt-Offset Tuning（提示偏移微调）。尽管现有的提示微调方法在图像、文本和视频模式的持续学习中显示出巨大的潜力；但它们要求访问大规模预训练的变压器。POET打破了这一假设，展示了对仅基于基础类数据进行预先训练的轻量级骨干网络进行提示微调的有效性。 我们提出了一种新颖的空间-时间可学习提示偏移微调方法，并且首次将这种提示微调应用于图神经网络（Graph Neural Networks, GNN）中。为了我们的新问题设置，我们在人类动作识别领域贡献了两个新的基准测试：(i) NTU RGB+D 数据集用于活动识别；以及 (ii) SHREC-2017 数据集用于手部姿态识别。 我们发现POET在全面的基准测试中表现一致出色。源代码可在提供的链接地址获取。
URL
https://arxiv.org/abs/2504.18059
