Abstract
Understanding the dynamic relationship between humans and the built environment is a key challenge in disciplines ranging from environmental psychology to reinforcement learning (RL). A central obstacle in modeling these interactions is the inability to capture human psychological states in a way that is both generalizable and privacy preserving. Traditional methods rely on theoretical models or questionnaires, which are limited in scope, static, and labor intensive. We present a kinesics recognition framework that infers the communicative functions of human activity -- known as kinesics -- directly from 3D skeleton joint data. Combining a spatial-temporal graph convolutional network (ST-GCN) with a convolutional neural network (CNN), the framework leverages transfer learning to bypass the need for manually defined mappings between physical actions and psychological categories. The approach preserves user anonymity while uncovering latent structures in bodily movements that reflect cognitive and emotional states. Our results on the Dyadic User EngagemenT (DUET) dataset demonstrate that this method enables scalable, accurate, and human-centered modeling of behavior, offering a new pathway for enhancing RL-driven simulations of human-environment interaction.
Abstract (translated)
理解人类与人造环境之间的动态关系是包括环境心理学和强化学习（RL）在内的多个学科的关键挑战。在建模这些互动时，一个核心障碍在于无法以既具普适性又保护隐私的方式捕捉人类的心理状态。传统方法依赖于理论模型或问卷调查，这些方式在范围、静态性和劳动强度方面都有局限。 我们提出了一种基于动作识别的框架，该框架直接从三维骨骼关节数据中推断出人类活动的交流功能——即动情学（kinesics）。这一框架结合了空间-时间图卷积网络（ST-GCN）和卷积神经网络（CNN），利用迁移学习来规避手动定义的身体行动与心理类别之间映射的需求。该方法在保持用户匿名性的同时，揭示出反映认知和情绪状态的肢体运动中的潜在结构。 我们在二元用户参与数据集（DUET）上的实验结果表明，这种方法能够实现行为的大规模、准确且以人为中心建模，为强化学习驱动的人类-环境互动仿真提供了新的途径。
URL
https://arxiv.org/abs/2510.04844
