Abstract
With the availability of egocentric 3D hand-object interaction datasets, there is increasing interest in developing unified models for hand-object pose estimation and action recognition. However, existing methods still struggle to recognise seen actions on unseen objects due to the limitations in representing object shape and movement using 3D bounding boxes. Additionally, the reliance on object templates at test time limits their generalisability to unseen objects. To address these challenges, we propose to leverage superquadrics as an alternative 3D object representation to bounding boxes and demonstrate their effectiveness on both template-free object reconstruction and action recognition tasks. Moreover, as we find that pure appearance-based methods can outperform the unified methods, the potential benefits from 3D geometric information remain unclear. Therefore, we study the compositionality of actions by considering a more challenging task where the training combinations of verbs and nouns do not overlap with the testing split. We extend H2O and FPHA datasets with compositional splits and design a novel collaborative learning framework that can explicitly reason about the geometric relations between hands and the manipulated object. Through extensive quantitative and qualitative evaluations, we demonstrate significant improvements over the state-of-the-arts in (compositional) action recognition.
Abstract (translated)
随着自中心3D手部与物体交互数据集的可用性提高，开发用于手部和物体姿态估计及动作识别的统一模型的兴趣日益增长。然而，现有方法仍然难以在未见过的物体上准确识别已见的动作，这是由于使用三维边界框来表示物体形状和运动存在局限性所致。此外，在测试时依赖于对象模板会限制其对未见过的物体的泛化能力。为了解决这些挑战，我们提议采用超二次体作为边界框的替代3D对象表示，并展示了它们在无模板的对象重建和动作识别任务中的有效性。 另外，我们发现基于外观的方法可以优于统一方法，但从三维几何信息中可能带来的潜在收益仍不清楚。因此，通过考虑一个更具挑战性的任务——训练动词和名词组合与测试分组不重叠——我们研究了行动的构成性，并将H2O和FPHA数据集扩展为具有组成性分割的数据集，设计了一种新的协作学习框架，该框架可以明确地推理出手部和被操作物体之间的几何关系。通过广泛的定量和定性评估，我们在（组成型）动作识别方面显著优于现有的最佳方法。
URL
https://arxiv.org/abs/2501.07100
