Gesture
-
OMG-Bench: A New Challenging Benchmark for Skeleton-based Online Micro Hand Gesture Recognition
2025-12-18 16:27:31Haochen Chang, Pengfei Ren, Buyuan Zhang, Da Li, Tianhao Han, Haoyang Zhang, Liang Xie, Hongbo Chen, Erwei YinAbstract
Online micro gesture recognition from hand skeletons is critical for VR/AR interaction but faces challenges due to limited public datasets and task-specific algorithms. Micro gestures involve subtle motion patterns, which make constructing datasets with precise skeletons and frame-level annotations difficult. To this end, we develop a multi-view self-supervised pipeline to automatically generate skeleton data, complemented by heuristic rules and expert refinement for semi-automatic annotation. Based on this pipeline, we introduce OMG-Bench, the first large-scale public benchmark for skeleton-based online micro gesture recognition. It features 40 fine-grained gesture classes with 13,948 instances across 1,272 sequences, characterized by subtle motions, rapid dynamics, and continuous execution. To tackle these challenges, we propose Hierarchical Memory-Augmented Transformer (HMATr), an end-to-end framework that unifies gesture detection and classification by leveraging hierarchical memory banks which store frame-level details and window-level semantics to preserve historical context. In addition, it employs learnable position-aware queries initialized from the memory to implicitly encode gesture positions and semantics. Experiments show that HMATr outperforms state-of-the-art methods by 7.6\% in detection rate, establishing a strong baseline for online micro gesture recognition. Project page: this https URL
Abstract (translated)
基于手骨骼的在线微手势识别对于虚拟现实（VR）和增强现实（AR）中的交互至关重要，但受限于公开数据集较少及任务特定算法的发展缓慢而面临挑战。由于微手势涉及细微的动作模式，构建具有精确骨骼信息和帧级注释的数据集变得困难。为此，我们开发了一种多视角自监督流水线来自动生成骨架数据，并辅以启发式规则和专家校准进行半自动注解。基于此流程，我们引入了OMG-Bench，这是第一个用于基于骨架的在线微手势识别的大规模公开基准测试平台。该平台包含40个精细的手势类别，共有13,948个实例及1,272条序列，这些实例和序列以细微动作、快速变化以及连续执行为特点。 为了应对上述挑战，我们提出了分层记忆增强变换器（Hierarchical Memory-Augmented Transformer, HMATr），这是一个统一手势检测与分类的端到端框架。该框架利用存储帧级细节及窗口级语义的信息层级存储库来保持历史上下文，并通过从内存中初始化的学习位置感知查询隐式编码手势的位置和语义。 实验表明，HMATr在检测率上比现有最佳方法高出7.6%，为在线微手势识别建立了一个强大的基准。项目页面：[此URL](https://this-url.com)
URL
https://arxiv.org/abs/2512.16727
PDF
-
Human-like Working Memory from Artificial Intrinsic Plasticity Neurons
2025-12-17 17:24:37Jingli Liu, Huannan Zheng, Bohao Zou, Kezhou YangAbstract
Working memory enables the brain to integrate transient information for rapid decision-making. Artificial networks typically replicate this via recurrent or parallel architectures, yet incur high energy costs and noise sensitivity. Here we report IPNet, a hardware-software co-designed neuromorphic architecture realizing human-like working memory via neuronal intrinsic plasticity. Exploiting Joule-heating dynamics of Magnetic Tunnel Junctions (MTJs), IPNet physically emulates biological memory volatility. The memory behavior of the proposed architecture shows similar trends in n-back, free recall and memory interference tasks to that of reported human subjects. Implemented exclusively with MTJ neurons, the architecture with human-like working memory achieves 99.65% accuracy on 11-class DVS gesture datasets and maintains 99.48% on a novel 22-class time-reversed benchmark, outperforming RNN, LSTM, and 2+1D CNN baselines sharing identical backbones. For autonomous driving (DDD-20), IPNet reduces steering prediction error by 14.4% compared to ResNet-LSTM. Architecturally, we identify a 'Memory-at-the-Frontier' effect where performance is maximized at the sensing interface, validating a bio-plausible near-sensor processing paradigm. Crucially, all results rely on raw parameters from fabricated devices without optimization. Hardware-in-the-loop validation confirms the system's physical realizability. Separately, energy analysis reveals a reduction in memory power of 2,874x compared to LSTMs and 90,920x versus parallel 3D-CNNs. This capacitor-free design enables a compact ~1.5um2 footprint (28 nm CMOS): a >20-fold reduction over standard LIF neurons. Ultimately, we demonstrate that instantiating human-like working memory via intrinsic neuronal plasticity endows neural networks with the dual biological advantages of superior dynamic vision processing and minimal metabolic cost.
Abstract (translated)
工作记忆使大脑能够整合瞬时信息，以实现快速决策。人工网络通常通过递归或并行架构来模拟这一过程，但这样会带来高昂的能量消耗和噪声敏感性。在此，我们报道了一种名为IPNet的神经形态架构，它通过神经元固有可塑性实现了类似于人类的工作记忆。利用磁隧道结（MTJ）中的焦耳热动力学，IPNet物理上模仿了生物记忆的波动特性。该架构的记忆行为在n-back、自由回忆和记忆干扰任务中表现出与报告的人类受试者相似的趋势。 仅使用MTJ神经元实现的具有类似人类工作记忆的架构，在11类DVS手势数据集上实现了99.65%的准确率，并且在22类时间反转基准测试上保持了99.48%的准确率，优于具有相同骨干网络的RNN、LSTM和2+1D CNN基线。对于自动驾驶（DDD-20），IPNet将转向预测误差降低了14.4%，相较于ResNet-LSTM。 从架构上看，我们发现了一个“前沿记忆”效应，在感知接口处性能最大化，验证了生物合理且接近传感器的处理范式。关键的是，所有结果都依赖于未经优化的设备原始参数。硬件在环测试证实系统的物理实现可行性。独立的能量分析揭示与LSTMs相比内存功率降低2,874倍，与并行3D-CNNs相比降低了90,920倍。 这种无电容设计实现了约1.5um²的小尺寸（基于28nm CMOS工艺），比标准的LIF神经元小了超过20倍。最终，我们证明通过固有神经可塑性实现类似于人类的工作记忆赋予神经网络在动态视觉处理和最低代谢成本方面的双重生物学优势。
URL
https://arxiv.org/abs/2512.15829
PDF
-
Adaptive Multimodal Person Recognition: A Robust Framework for Handling Missing Modalities
2025-12-16 22:59:24Aref Farhadipour, Teodora Vukovic, Volker Dellwo, Petr Motlicek, Srikanth MadikeriAbstract
Person recognition systems often rely on audio, visual, or behavioral cues, but real-world conditions frequently result in missing or degraded modalities. To address this challenge, we propose a Trimodal person identification framework that integrates voice, face, and gesture modalities, while remaining robust to modality loss. Our approach leverages multi-task learning to process each modality independently, followed by a cross-attention and gated fusion mechanisms to facilitate interaction across modalities. Moreover, a confidence-weighted fusion strategy dynamically adapts to missing and low-quality data, ensuring optimal classification even in Unimodal or Bimodal scenarios. We evaluate our method on CANDOR, a newly introduced interview-based multimodal dataset, which we benchmark for the first time. Our results demonstrate that the proposed Trimodal system achieves 99.18% Top-1 accuracy on person identification tasks, outperforming conventional Unimodal and late-fusion approaches. In addition, we evaluate our model on the VoxCeleb1 dataset as a benchmark and reach 99.92% accuracy in Bimodal mode. Moreover, we show that our system maintains high accuracy even when one or two modalities are unavailable, making it a robust solution for real-world person recognition applications. The code and data for this work are publicly available.
Abstract (translated)
人员识别系统通常依赖于音频、视觉或行为线索，但在实际环境中，这些模态往往会出现缺失或退化的情况。为了解决这一挑战，我们提出了一种三模态（语音、面部和手势）的人体识别框架，该框架能够在模态丢失的情况下保持稳健性。我们的方法利用多任务学习来独立处理每个模态，并通过跨模态交互的交叉注意力机制和门控融合机制加以增强。此外，采用基于置信度加权的融合策略可以动态适应缺失或低质量的数据，在单模态或多模态场景下确保最佳分类效果。 我们在CANDOR数据集上评估了这种方法，该数据集是新近推出的基于访谈的多模态数据集，并首次对它进行了基准测试。实验结果显示，我们的三模态系统在人员识别任务中达到了99.18%的第一名准确率，优于传统的单模态和晚期融合方法。此外，在VoxCeleb1数据集中，我们通过双模态模式达到了99.92%的精度。 更重要的是，即使当一个或两个模态不可用时，我们的系统仍能保持高准确性，使其成为现实世界中人员识别应用的一种稳健解决方案。本研究的相关代码和数据已公开发布。
URL
https://arxiv.org/abs/2512.14961
PDF
-
ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body
2025-12-16 09:41:21Juze Zhang, Changan Chen, Xin Chen, Heng Yu, Tiange Xiang, Ali Sartaz Khan, Shrinidhi K. Lakshmikanth, Ehsan AdeliAbstract
Human communication is inherently multimodal and social: words, prosody, and body language jointly carry intent. Yet most prior systems model human behavior as a translation task co-speech gesture or text-to-motion that maps a fixed utterance to motion clips-without requiring agentic decision-making about when to move, what to do, or how to adapt across multi-turn dialogue. This leads to brittle timing, weak social grounding, and fragmented stacks where speech, text, and motion are trained or inferred in isolation. We introduce ViBES (Voice in Behavioral Expression and Synchrony), a conversational 3D agent that jointly plans language and movement and executes dialogue-conditioned body actions. Concretely, ViBES is a speech-language-behavior (SLB) model with a mixture-of-modality-experts (MoME) backbone: modality-partitioned transformer experts for speech, facial expression, and body motion. The model processes interleaved multimodal token streams with hard routing by modality (parameters are split per expert), while sharing information through cross-expert attention. By leveraging strong pretrained speech-language models, the agent supports mixed-initiative interaction: users can speak, type, or issue body-action directives mid-conversation, and the system exposes controllable behavior hooks for streaming responses. We further benchmark on multi-turn conversation with automatic metrics of dialogue-motion alignment and behavior quality, and observe consistent gains over strong co-speech and text-to-motion baselines. ViBES goes beyond "speech-conditioned motion generation" toward agentic virtual bodies where language, prosody, and movement are jointly generated, enabling controllable, socially competent 3D interaction. Code and data will be made available at: this http URL
Abstract (translated)
人类交流本质上是多模态和社交性的：语言、音调以及肢体语言共同传递意图。然而，大多数先前的系统将人类行为建模为一种翻译任务——即伴随言语的手势或文本到动作映射，这一过程固定了一个话语与动作片段之间的关系，并不需要有关何时移动、做什么或如何在多轮对话中适应这些方面的问题决策。这导致了刻板的时间处理方式、弱化的社交背景以及碎片化的方法，在这种方法中，语音、文字和动作的训练或推断都是孤立进行的。 我们引入了一种名为ViBES（行为表达与同步中的声音）的会话式3D代理，它能够同时计划语言和行动，并根据对话条件执行身体动作。具体而言，ViBES是一个包含混合模态专家（MoME）骨干架构的语言-语音-行为（SLB）模型：该模型通过针对语音、面部表情以及肢体运动的不同转换器专家进行分模态处理。模型可以依据模态间交错的多模式标记流进行硬路由（每个专家都有独立的参数），并通过跨专家注意力机制共享信息。 利用强大的预训练语言-语音模型，代理支持混合发起互动：用户可以在对话过程中说话、打字或发出身体动作指令，并且系统提供可控制的行为接口用于实时响应。我们进一步在多轮对话上进行基准测试，并使用自动度量标准评估对话-动作对齐及行为质量，观察到与强大的伴随言语和文本到动作基线相比，ViBES有显著改进。 ViBES超越了“语音驱动的动作生成”，朝向能够共同生成语言、音调和运动的代理式虚拟身体发展。这使得可控且社交胜任度高的3D互动成为可能。代码及数据将在以下链接提供：[此链接]（需要将"http URL"替换为实际提供的GitHub或项目页面地址）。
URL
https://arxiv.org/abs/2512.14234
PDF
-
USTM: Unified Spatial and Temporal Modeling for Continuous Sign Language Recognition
2025-12-15 15:05:16Ahmed Abul Hasanaath, Hamzah LuqmanAbstract
Continuous sign language recognition (CSLR) requires precise spatio-temporal modeling to accurately recognize sequences of gestures in videos. Existing frameworks often rely on CNN-based spatial backbones combined with temporal convolution or recurrent modules. These techniques fail in capturing fine-grained hand and facial cues and modeling long-range temporal dependencies. To address these limitations, we propose the Unified Spatio-Temporal Modeling (USTM) framework, a spatio-temporal encoder that effectively models complex patterns using a combination of a Swin Transformer backbone enhanced with lightweight temporal adapter with positional embeddings (TAPE). Our framework captures fine-grained spatial features alongside short and long-term temporal context, enabling robust sign language recognition from RGB videos without relying on multi-stream inputs or auxiliary modalities. Extensive experiments on benchmarked datasets including PHOENIX14, PHOENIX14T, and CSL-Daily demonstrate that USTM achieves state-of-the-art performance against RGB-based as well as multi-modal CSLR approaches, while maintaining competitive performance against multi-stream approaches. These results highlight the strength and efficacy of the USTM framework for CSLR. The code is available at this https URL
Abstract (translated)
连续手语识别（CSLR）需要精确的时空建模，以准确地从视频中识别手势序列。现有的框架通常依赖于基于CNN的空间骨干网络，并结合时间卷积或递归模块。这些技术无法捕捉精细的手部和面部线索，也无法建立长时间的时序依赖关系。为了克服这些限制，我们提出了统一时空模型（USTM）框架，这是一种时空编码器，通过将Swin Transformer骨干网与轻量级的时间适配器结合使用，并添加位置嵌入（TAPE），来有效地建模复杂模式。我们的框架可以捕捉精细的视觉特征以及短期和长期时序上下文，从而能够仅从RGB视频中进行稳健的手语识别，而无需依赖多流输入或辅助模式。在包括PHOENIX14、PHOENIX14T和CSL-Daily在内的基准数据集上的广泛实验表明，USTM框架在基于RGB的方法以及多模态CSLR方法方面取得了最先进的性能，并且与多流方法相比保持了竞争性的表现。这些结果突显了USTM框架在CSLR中的强大有效性和适用性。代码可以在提供的链接中获取。
URL
https://arxiv.org/abs/2512.13415
PDF
-
Towards Unified Co-Speech Gesture Generation via Hierarchical Implicit Periodicity Learning
2025-12-15 09:43:08Xin Guo, Yifan Zhao, Jia LiAbstract
Generating 3D-based body movements from speech shows great potential in extensive downstream applications, while it still suffers challenges in imitating realistic human movements. Predominant research efforts focus on end-to-end generation schemes to generate co-speech gestures, spanning GANs, VQ-VAE, and recent diffusion models. As an ill-posed problem, in this paper, we argue that these prevailing learning schemes fail to model crucial inter- and intra-correlations across different motion units, i.e. head, body, and hands, thus leading to unnatural movements and poor coordination. To delve into these intrinsic correlations, we propose a unified Hierarchical Implicit Periodicity (HIP) learning approach for audio-inspired 3D gesture generation. Different from predominant research, our approach models this multi-modal implicit relationship by two explicit technique insights: i) To disentangle the complicated gesture movements, we first explore the gesture motion phase manifolds with periodic autoencoders to imitate human natures from realistic distributions while incorporating non-period ones from current latent states for instance-level diversities. ii) To model the hierarchical relationship of face motions, body gestures, and hand movements, driving the animation with cascaded guidance during learning. We exhibit our proposed approach on 3D avatars and extensive experiments show our method outperforms the state-of-the-art co-speech gesture generation methods by both quantitative and qualitative evaluations. Code and models will be publicly available.
Abstract (translated)
从语音生成基于3D的身体动作在广泛的下游应用中展现出巨大潜力，然而它仍然面临模仿真实人体运动的挑战。目前的研究主要集中在端到端生成方案上，以生成与言语同步的手势，涵盖了GAN、VQ-VAE以及最近的扩散模型。作为一种病态问题，在本文中我们论证了这些流行的学习方案未能充分建模不同动作单元（如头部、身体和手）之间的重要内在和外在相关性，从而导致不自然的动作和协调性差。 为了深入探究这些内在关联，我们提出了一种统一的分层隐式周期性（HIP）学习方法，用于语音启发式的3D手势生成。与主流研究不同，我们的方法通过两种明确的技术洞察来建模这种多模态的隐含关系：i) 为了解构复杂的动作模式，我们首先使用周期自动编码器探索手势运动相位流形，并从真实分布中模仿人类特性，同时结合非周期性的当前潜在状态以实现实例级别的多样性。ii) 为了模型面部、身体和手部动作之间的层级关系，在学习过程中采用级联引导来驱动动画。 我们在3D化身上演示了我们提出的方法，并通过广泛的实验展示了我们的方法在定量和定性评估中都超越了最先进的与言语同步手势生成方法的性能。代码和模型将公开发布。
URL
https://arxiv.org/abs/2512.13131
PDF
-
InteracTalker: Prompt-Based Human-Object Interaction with Co-Speech Gesture Generation
2025-12-14 12:29:49Sreehari Rajan, Kunal Bhosikar, Charu SharmaAbstract
Generating realistic human motions that naturally respond to both spoken language and physical objects is crucial for interactive digital experiences. Current methods, however, address speech-driven gestures or object interactions independently, limiting real-world applicability due to a lack of integrated, comprehensive datasets. To overcome this, we introduce InteracTalker, a novel framework that seamlessly integrates prompt-based object-aware interactions with co-speech gesture generation. We achieve this by employing a multi-stage training process to learn a unified motion, speech, and prompt embedding space. To support this, we curate a rich human-object interaction dataset, formed by augmenting an existing text-to-motion dataset with detailed object interaction annotations. Our framework utilizes a Generalized Motion Adaptation Module that enables independent training, adapting to the corresponding motion condition, which is then dynamically combined during inference. To address the imbalance between heterogeneous conditioning signals, we propose an adaptive fusion strategy, which dynamically reweights the conditioning signals during diffusion sampling. InteracTalker successfully unifies these previously separate tasks, outperforming prior methods in both co-speech gesture generation and object-interaction synthesis, outperforming gesture-focused diffusion methods, yielding highly realistic, object-aware full-body motions with enhanced realism, flexibility, and control.
Abstract (translated)
生成逼真的人体动作，这些动作能够自然地响应口语和物理对象，对于互动数字体验至关重要。然而，当前的方法分别处理由语音驱动的手势和与物体的交互，这限制了它们在现实世界中的应用能力，因为缺乏综合的数据集。为了解决这一问题，我们引入了一个名为InteracTalker的新框架，该框架能够无缝地将基于提示的、感知到的对象互动与伴随言语的手势生成相结合。通过采用一个多阶段训练过程来学习统一的动作、语音和提示嵌入空间，我们实现了这一点。 为了支持这个框架，我们精心策划了一个丰富的人体-物体交互数据集，它是通过对现有的文本至动作的数据集进行详细的对象交互注释扩充而形成的。我们的框架利用了一个通用的运动适应模块，该模块允许独立训练，并根据相应的运动条件进行调整，然后在推理过程中动态结合。 为了应对异构调节信号之间的不平衡问题，我们提出了一种自适应融合策略，在扩散采样期间动态重新加权调节信号。InteracTalker成功地将这些以前分离的任务统一起来，在伴随言语的手势生成和对象互动合成方面均超过了先前的方法，特别是在手势聚焦的扩散方法上表现更优。通过这种方式，InteracTalker能够生成具有增强现实感、灵活性和控制性的高度逼真且感知到物体的存在全身体动。
URL
https://arxiv.org/abs/2512.12664
PDF
-
Embodied Referring Expression Comprehension in Human-Robot Interaction
2025-12-06 20:10:06Md Mofijul Islam, Alexi Gladstone, Sujan Sarker, Ganesh Nanduru, Md Fahim, Keyan Du, Aman Chadha, Tariq IqbalAbstract
As robots enter human workspaces, there is a crucial need for them to comprehend embodied human instructions, enabling intuitive and fluent human-robot interaction (HRI). However, accurate comprehension is challenging due to a lack of large-scale datasets that capture natural embodied interactions in diverse HRI settings. Existing datasets suffer from perspective bias, single-view collection, inadequate coverage of nonverbal gestures, and a predominant focus on indoor environments. To address these issues, we present the Refer360 dataset, a large-scale dataset of embodied verbal and nonverbal interactions collected across diverse viewpoints in both indoor and outdoor settings. Additionally, we introduce MuRes, a multimodal guided residual module designed to improve embodied referring expression comprehension. MuRes acts as an information bottleneck, extracting salient modality-specific signals and reinforcing them into pre-trained representations to form complementary features for downstream tasks. We conduct extensive experiments on four HRI datasets, including the Refer360 dataset, and demonstrate that current multimodal models fail to capture embodied interactions comprehensively; however, augmenting them with MuRes consistently improves performance. These findings establish Refer360 as a valuable benchmark and exhibit the potential of guided residual learning to advance embodied referring expression comprehension in robots operating within human environments.
Abstract (translated)
随着机器人进入人类工作空间，它们需要理解具体的人类指令以实现直观流畅的人机交互（HRI）。然而，由于缺乏能够捕捉多样环境中的自然具体互动的大规模数据集，准确的理解变得具有挑战性。现有的数据集存在视角偏差、单一视图收集、非言语手势覆盖不足以及主要关注室内环境等问题。为了解决这些问题，我们推出了Refer360数据集，这是一个大规模的数据集，包含了在室内和室外环境中从多个不同角度采集的具体语言和非语言互动。此外，我们还引入了MuRes，一个多模态引导残差模块，旨在提高具体指示表达的理解能力。MuRes作为一个信息瓶颈，提取出每个模式中显著的信号，并将其强化到预训练表示中，形成互补特征以用于下游任务。 我们在四个HRI数据集上进行了广泛的实验，包括Refer360数据集，并展示了当前多模态模型无法全面捕捉具体互动；然而，在这些模型中加入MuRes可以持续提升性能。这一发现确立了Refer360作为宝贵的基准测试的价值，并表明引导残差学习在提高机器人理解人类环境中的具体指示表达方面具有潜力。
URL
https://arxiv.org/abs/2512.06558
PDF
-
WiFi-based Cross-Domain Gesture Recognition Using Attention Mechanism
2025-12-04 07:09:13Ruijing Liu, Cunhua Pan, Jiaming Zeng, Hong Ren, Kezhi Wang, Lei Kong, Jiangzhou WangAbstract
While fulfilling communication tasks, wireless signals can also be used to sense the environment. Among various types of sensing media, WiFi signals offer advantages such as widespread availability, low hardware cost, and strong robustness to environmental conditions like light, temperature, and humidity. By analyzing Wi-Fi signals in the environment, it is possible to capture dynamic changes of the human body and accomplish sensing applications such as gesture recognition. Although many existing gesture sensing solutions perform well in-domain but lack cross-domain capabilities (i.e., recognition performance in untrained environments). To address this, we extract Doppler spectra from the channel state information (CSI) received by all receivers and concatenate each Doppler spectrum along the same time axis to generate fused images with multi-angle information as input features. Furthermore, inspired by the convolutional block attention module (CBAM), we propose a gesture recognition network that integrates a multi-semantic spatial attention mechanism with a self-attention-based channel mechanism. This network constructs attention maps to quantify the spatiotemporal features of gestures in images, enabling the extraction of key domain-independent features. Additionally, ResNet18 is employed as the backbone network to further capture deep-level features. To validate the network performance, we evaluate the proposed network on the public Widar3 dataset, and the results show that it not only maintains high in-domain accuracy of 99.72%, but also achieves high performance in cross-domain recognition of 97.61%, significantly outperforming existing best solutions.
Abstract (translated)
在执行通信任务的同时，无线信号也可以用来感知环境。在各种传感媒介中，WiFi信号具有广泛可用性、硬件成本低以及对光照、温度和湿度等环境条件的强健性等特点。通过分析环境中Wi-Fi信号，可以捕捉人体动态变化并实现手势识别等功能。然而，许多现有的手势感应解决方案虽然在特定领域内表现出色，但缺乏跨域能力（即，在未训练环境中进行识别的能力）。为了解决这一问题，我们从信道状态信息(CSI)中提取多接收器接收到的所有数据中的多普勒频谱，并将每个频谱沿同一时间轴连接起来以生成包含多角度信息的融合图像作为输入特征。此外，受卷积块注意模块（CBAM）启发，我们提出了一种集成了多语义空间注意力机制与基于自注意力的通道机制的手势识别网络。该网络构建了注意力图来量化手势在图像中的时空特性，从而提取出关键的领域无关特征。此外，还使用ResNet18作为骨干网络以进一步捕获深层次特征。为了验证网络性能，我们在公开的Widar3数据集上评估了所提出的网络，并且结果表明该方法不仅保持了99.72%的高度域内准确率，还在跨域识别中取得了97.61%的表现，远远超过了现有的最佳解决方案。
URL
https://arxiv.org/abs/2512.04521
PDF
-
Stable Signer: Hierarchical Sign Language Generative Model
2025-12-03 18:33:40Sen Fang, Yalin Feng, Hongbin Zhong, Yanxin Zhang, Dimitris N. MetaxasAbstract
Sign Language Production (SLP) is the process of converting the complex input text into a real video. Most previous works focused on the Text2Gloss, Gloss2Pose, Pose2Vid stages, and some concentrated on Prompt2Gloss and Text2Avatar stages. However, this field has made slow progress due to the inaccuracy of text conversion, pose generation, and the rendering of poses into real human videos in these stages, resulting in gradually accumulating errors. Therefore, in this paper, we streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer. It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding (Prompt2Gloss, Text2Gloss) and Pose2Vid, and executes text understanding through our proposed new Sign Language Understanding Linker called SLUL, and generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multi-style sign language videos. SLUL is trained using the newly developed Semantic-Aware Gloss Masking Loss (SAGM Loss). Its performance has improved by 48.6% compared to the current SOTA generation methods.
Abstract (translated)
手语生成（SLP）是指将复杂的输入文本转换为真实视频的过程。以往的研究主要集中在Text2Gloss、Gloss2Pose和Pose2Vid等阶段，也有一些研究关注Prompt2Gloss和Text2Avatar阶段。然而，由于这些阶段中文字转换的不准确性、姿态生成以及姿态转化为真人视频时存在的误差逐渐累积，该领域的发展较为缓慢。 因此，在这篇论文中，我们简化了传统的冗余结构，优化并精简了任务目标，并设计了一个新的手语生成模型——Stable Signer。它将SLP任务重新定义为一个包含文本理解和姿势到视频转换的分层端到端任务（即Prompt2Gloss、Text2Gloss和Pose2Vid）。我们通过提出的Sign Language Understanding Linker (SLUL) 来执行文本理解，并利用名为SLP-MoE的手势渲染专家块来生成手势，从而实现高质量且多风格的手语视频的端到端生成。该模型中的SLUL使用新的感知词汇掩码损失（Semantic-Aware Gloss Masking Loss, SAGM Loss）进行训练，在当前最先进的生成方法中表现出48.6%的性能提升。
URL
https://arxiv.org/abs/2512.04048
PDF
-
Head, posture, and full-body gestures in interactive communication
2025-12-03 10:16:34\v{L}ubo\v{s} Hl\'adek, Bernhard U. SeeberAbstract
When face-to-face communication becomes effortful due to background noise or interfering talkers, the role of visual cues becomes increasingly important for communication success. While previous research has selectively examined head or hand movements, here we explore movements of the whole body in acoustically adverse conditions. We hypothesized that increasing background noise in conversations would lead to increased gesture frequency in hand, head, trunk, and leg movements typical of conversation. Increased use of hand movements should support the speaker's role, while increased head and trunk movements may help the listener. We conducted a free dyadic conversation experiment with normal-hearing participants (n=8) in a virtual acoustic environment. Conversational movements were described with a newly developed labeling system for typical conversational actions, and the frequency of individual types was analyzed. In addition, we analyzed gesture quality by assessing hand-speech synchrony, with the hypothesis that higher levels of background noise would lead to a loss of synchrony according to an interactive coupling model. Higher noise levels led to increased hand-gesture complexity during speaking and listening, more pronounced up-down head movements, and contrary to expectations, head movements during listening generally decreased relative to speaking. Synchrony and peak velocity were unaffected by noise, while gesture quality scaled only modestly. The results support previous findings regarding gesturing frequency, but we found only limited evidence for changes in speech-gesture synchrony. This work reveals communication patterns of the whole body and illustrates multimodal adaptation to communication demands.
Abstract (translated)
当面对面交流因背景噪音或干扰说话者而变得困难时，视觉线索在沟通成功中的作用变得更加重要。虽然之前的研究所选择地研究头部或手部动作，但在这里我们探讨了在声学条件不利的情况下全身运动的作用。我们的假设是，在对话中增加背景噪声会导致手势频率的增加，包括手、头、躯干和腿部的动作。手部动作的增加应该支持说话者的角色，而头部和躯干动作的增加可能有助于听者。 我们进行了一项自由双人对话实验，参与者为正常听力人士（n=8），在虚拟声学环境中完成。对话运动用一个新的标签系统来描述典型的对话行为，并分析了个别类型的频率。此外，通过评估手部动作与言语的同步性，我们还分析了手势质量，假设根据交互耦合模型，在较高水平背景噪声的情况下会失去这种同步性。 较高的噪音水平导致在说话和倾听时的手势复杂度增加，头部上下运动更加明显，但出乎意料的是，听者头部位移通常相对于说话而言减少了。同步性和峰值速度不受噪声的影响，而手势质量仅适度变化。结果支持了关于手势频率的先前发现，但我们仅发现了有限的证据表明语音与动作之间的同步性有所改变。 这项工作揭示了整个身体在交流中的模式，并展示了多模态适应于沟通需求的方式。
URL
https://arxiv.org/abs/2512.03636
PDF
-
Co-speech Gesture Video Generation via Motion-Based Graph Retrieval
2025-12-02 09:46:12Yafei Song, Peng Zhang, Bang ZhangAbstract
Synthesizing synchronized and natural co-speech gesture videos remains a formidable challenge. Recent approaches have leveraged motion graphs to harness the potential of existing video data. To retrieve an appropriate trajectory from the graph, previous methods either utilize the distance between features extracted from the input audio and those associated with the motions in the graph or embed both the input audio and motion into a shared feature space. However, these techniques may not be optimal due to the many-to-many mapping nature between audio and gestures, which cannot be adequately addressed by one-to-one mapping. To alleviate this limitation, we propose a novel framework that initially employs a diffusion model to generate gesture motions. The diffusion model implicitly learns the joint distribution of audio and motion, enabling the generation of contextually appropriate gestures from input audio sequences. Furthermore, our method extracts both low-level and high-level features from the input audio to enrich the training process of the diffusion model. Subsequently, a meticulously designed motion-based retrieval algorithm is applied to identify the most suitable path within the graph by assessing both global and local similarities in motion. Given that not all nodes in the retrieved path are sequentially continuous, the final step involves seamlessly stitching together these segments to produce a coherent video output. Experimental results substantiate the efficacy of our proposed method, demonstrating a significant improvement over prior approaches in terms of synchronization accuracy and naturalness of generated gestures.
Abstract (translated)
合成同步且自然的伴随言语的手势视频仍然是一个艰巨的任务。近期的研究方法利用了运动图，以发掘现有视频数据的潜力。为了从图中检索合适的轨迹，先前的方法要么采用输入音频与图中动作特征之间的距离来获取相应的路径，要么将输入音频和动作嵌入到共享的特征空间中。然而，由于声音与手势之间存在多对多映射关系，这两种技术可能并非最优选择，因为这种复杂性不能仅通过一对一映射解决。 为了解决这一限制，我们提出了一种新的框架，首先利用扩散模型生成手势运动。该扩散模型隐式地学习了音频和动作的联合分布，从而能够从输入音频序列中产生语境相关的手势。此外，我们的方法同时提取输入音频中的低级和高级特征，以丰富扩散模型的训练过程。 随后，我们设计了一个精细的动作基础检索算法，通过评估全局和局部运动相似性来识别图中最合适的路径。考虑到在检索到的路径上并非所有节点都连续相连，最后一步需要将这些片段无缝拼接起来，生成一个连贯的视频输出。 实验结果证实了我们方法的有效性，展示了相比之前的方法，在同步准确性和手势自然度方面有显著改进。
URL
https://arxiv.org/abs/2512.02576
PDF
-
SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation
2025-12-01 18:13:40Zisu Li, Hengye Lyu, Jiaxin Shi, Yufeng Zeng, Mingming Fan, Hanwang Zhang, Chen LiangAbstract
Modeling and synthesizing complex hand-object interactions remains a significant challenge, even for state-of-the-art physics engines. Conventional simulation-based approaches rely on explicitly defined rigid object models and pre-scripted hand gestures, making them inadequate for capturing dynamic interactions with non-rigid or articulated entities such as deformable fabrics, elastic materials, hinge-based structures, furry surfaces, or even living creatures. In this paper, we present SpriteHand, an autoregressive video generation framework for real-time synthesis of versatile hand-object interaction videos across a wide range of object types and motion patterns. SpriteHand takes as input a static object image and a video stream in which the hands are imagined to interact with the virtual object embedded in a real-world scene, and generates corresponding hand-object interaction effects in real time. Our model employs a causal inference architecture for autoregressive generation and leverages a hybrid post-training approach to enhance visual realism and temporal coherence. Our 1.3B model supports real-time streaming generation at around 18 FPS and 640x368 resolution, with an approximate 150 ms latency on a single NVIDIA RTX 5090 GPU, and more than a minute of continuous output. Experiments demonstrate superior visual quality, physical plausibility, and interaction fidelity compared to both generative and engine-based baselines.
Abstract (translated)
对给定文本的翻译如下： 复杂的手部与物体之间的交互建模和合成仍然是一个重大挑战，即使是当前最先进的物理引擎也无法完全解决。传统基于仿真的方法依赖于明确定义的刚性对象模型以及预设的手势脚本，这使得它们在捕捉非刚性或关节结构（如可变形织物、弹性材料、铰链结构、毛茸表面甚至活体生物）与手部之间的动态交互时显得力不从心。本文中，我们介绍了SpriteHand——这是一个用于实时生成多种类型物体和运动模式下的手部-物体互动视频的自回归视频生成框架。SpriteHand以一个静态的对象图像和一段假想的手部在现实场景中的虚拟对象上进行互动的视频流作为输入，并实时生成相应的真实感手部与对象之间的交互效果。我们的模型采用因果推理架构来进行自回归生成，并利用混合后训练方法来增强视觉真实性和时间一致性。我们的1.3B参数量的模型支持以约每秒18帧和640x368分辨率进行实时流式生成，在单个NVIDIA RTX 5090 GPU上的延迟约为150毫秒，且可以持续输出超过一分钟的内容。实验结果表明，我们的方法在视觉质量、物理合理性及交互真实度方面均优于基于生成和引擎的基线模型。
URL
https://arxiv.org/abs/2512.01960
PDF
-
fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignment
2025-12-01 02:09:44Chunzheng Zhu, Jialin Shao, Jianxin Lin, Yijun Wang, Jing Wang, Jinhui Tang, Kenli LiAbstract
Understanding how the brain responds to external stimuli and decoding this process has been a significant challenge in neuroscience. While previous studies typically concentrated on brain-to-image and brain-to-language reconstruction, our work strives to reconstruct gestures associated with speech stimuli perceived by brain. Unfortunately, the lack of paired \{brain, speech, gesture\} data hinders the deployment of deep learning models for this purpose. In this paper, we introduce a novel approach, \textbf{fMRI2GES}, that allows training of fMRI-to-gesture reconstruction networks on unpaired data using \textbf{Dual Brain Decoding Alignment}. This method relies on two key components: (i) observed texts that elicit brain responses, and (ii) textual descriptions associated with the gestures. Then, instead of training models in a completely supervised manner to find a mapping relationship among the three modalities, we harness an fMRI-to-text model, a text-to-gesture model with paired data and an fMRI-to-gesture model with unpaired data, establishing dual fMRI-to-gesture reconstruction patterns. Afterward, we explicitly align two outputs and train our model in a self-supervision way. We show that our proposed method can reconstruct expressive gestures directly from fMRI recordings. We also investigate fMRI signals from different ROIs in the cortex and how they affect generation results. Overall, we provide new insights into decoding co-speech gestures, thereby advancing our understanding of neuroscience and cognitive science.
Abstract (translated)
理解大脑对外部刺激的反应并解码这一过程一直是神经科学面临的重大挑战。以往的研究通常集中在脑影像到图像或脑影像到语言的重建上，而我们的工作则致力于重构与由大脑感知的语言刺激相关的手势。遗憾的是，缺乏成对的{\{大脑, 语音, 手势\}}数据阻碍了深度学习模型在这一领域的部署。在这篇论文中，我们提出了一种新方法**fMRI2GES**，它允许使用**双脑解码对齐**技术在非配对的数据上训练fMRI到手势的重构网络。该方法依赖于两个关键组成部分：（i）引发大脑反应的观察文本；和（ii）与手势相关的文字描述。我们不是以完全监督的方式训练模型来寻找三种模式之间的映射关系，而是利用一个fMRI到文本的模型、一个使用配对数据的文字到手势模型以及一个基于非配对数据的fMRI到手势模型，建立双fMRI到手势重构模式。之后，我们将两个输出明确对齐并以自监督方式训练我们的模型。我们展示了所提出的方法可以直接从fMRI记录中重构出富有表现力的手势，并研究了皮层不同脑区（ROI）的fMRI信号及其对手部生成结果的影响。总体而言，我们为解码伴随语言的动作提供了新的见解，从而推进了对神经科学和认知科学的理解。
URL
https://arxiv.org/abs/2512.01189
PDF
-
TalkingPose: Efficient Face and Gesture Animation with Feedback-guided Diffusion Model
2025-11-30 14:26:24Alireza Javanmardi, Pragati Jaiswal, Tewodros Amberbir Habtegebrial, Christen Millerdurai, Shaoxiang Wang, Alain Pagani, Didier StrickerAbstract
Recent advancements in diffusion models have significantly improved the realism and generalizability of character-driven animation, enabling the synthesis of high-quality motion from just a single RGB image and a set of driving poses. Nevertheless, generating temporally coherent long-form content remains challenging. Existing approaches are constrained by computational and memory limitations, as they are typically trained on short video segments, thus performing effectively only over limited frame lengths and hindering their potential for extended coherent generation. To address these constraints, we propose TalkingPose, a novel diffusion-based framework specifically designed for producing long-form, temporally consistent human upper-body animations. TalkingPose leverages driving frames to precisely capture expressive facial and hand movements, transferring these seamlessly to a target actor through a stable diffusion backbone. To ensure continuous motion and enhance temporal coherence, we introduce a feedback-driven mechanism built upon image-based diffusion models. Notably, this mechanism does not incur additional computational costs or require secondary training stages, enabling the generation of animations with unlimited duration. Additionally, we introduce a comprehensive, large-scale dataset to serve as a new benchmark for human upper-body animation.
Abstract (translated)
最近在扩散模型方面的进展显著提升了以角色驱动的动画的真实感和泛化能力，使得仅通过一张RGB图像和一组驱动姿势即可合成高质量的动作成为可能。然而，生成具有时间连贯性的长篇内容仍然颇具挑战性。现有方法受到计算资源和内存限制的影响，因为它们通常在短视频片段上进行训练，因此只能在有限的帧长度内有效运作，并且阻碍了其长时间连贯生成的能力。为了解决这些限制，我们提出了TalkingPose，这是一种专门为生产长期、时间一致的人类上半身动画而设计的新扩散框架。 TalkingPose利用驱动帧精确捕捉表情丰富的面部和手部动作，并通过一个稳定的扩散骨干网络将其无缝转移至目标演员身上。为了确保连续运动并增强时间一致性，我们在基于图像的扩散模型之上引入了一种反馈驱动机制。值得注意的是，这一机制不会增加额外的计算成本或需要二次训练阶段，从而能够生成无限时长的动画。 此外，我们还引入了一个全面且大规模的数据集，作为人类上半身动画的新基准标准。
URL
https://arxiv.org/abs/2512.00909
PDF
-
CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generation
2025-11-28 03:38:08Fengyi Fang, Sicheng Yang, Wenming YangAbstract
Co-speech gesture generation has significantly advanced human-computer interaction, yet speaker movements remain constrained due to the omission of text-driven non-spontaneous gestures (e.g., bowing while talking). Existing methods face two key challenges: 1) the semantic prior gap due to the lack of descriptive text annotations in gesture datasets, and 2) the difficulty in achieving coordinated multimodal control over gesture generation. To address these challenges, this paper introduces CoordSpeaker, a comprehensive framework that enables coordinated caption-empowered co-speech gesture synthesis. Our approach first bridges the semantic prior gap through a novel gesture captioning framework, leveraging a motion-language model to generate descriptive captions at multiple granularities. Building upon this, we propose a conditional latent diffusion model with unified cross-dataset motion representation and a hierarchically controlled denoiser to achieve highly controlled, coordinated gesture generation. CoordSpeaker pioneers the first exploration of gesture understanding and captioning to tackle the semantic gap in gesture generation while offering a novel perspective of bidirectional gesture-text mapping. Extensive experiments demonstrate that our method produces high-quality gestures that are both rhythmically synchronized with speeches and semantically coherent with arbitrary captions, achieving superior performance with higher efficiency compared to existing approaches.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22863
PDF
-
Lips-Jaw and Tongue-Jaw Articulatory Tradeoff in DYNARTmo
Abstract
This paper investigates how the dynamic articulatory model DYNARTmo accounts for articulatory tradeoffs between primary and secondary articulators, with a focus on lips-jaw and tongue-jaw coordination. While DYNARTmo does not implement full task-dynamic second-order biomechanics, it adopts first-order task-space gesture specifications comparable to those used in articulatory phonology and integrates a simplified mechanism for distributing articulatory effort across multiple articulators. We first outline the conceptual relationship between task dynamics and DYNARTmo, emphasizing the distinction between high-level task-space trajectories and their low-level articulatory execution. We then present simulation results for a set of CV syllables that illustrate how jaw displacement varies as a function of both place of articulation (labial, apical, dorsal) and vowel context (/a/, /i/, /u/). The model reproduces empirically attested patterns of articulatory synergy, including jaw-supported apical closures, lower-lip elevation in bilabial stops, tongue-jaw co-movement, and saturation effects in labial constrictions. These results demonstrate that even with computationally simplified assumptions, DYNARTmo can generate realistic spatio-temporal movement patterns that capture key aspects of articulatory tradeoff and synergy across a range of consonant-vowel combinations.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22155
PDF
-
Design of an Adaptive Modular Anthropomorphic Dexterous Hand for Human-like Manipulation
2025-11-27 04:42:44Zelong Zhou, Wenrui Chen, Zeyun Hu, Qiang Diao, Qixin Gao, Yaonan WangAbstract
Biological synergies have emerged as a widely adopted paradigm for dexterous hand design, enabling human-like manipulation with a small number of actuators. Nonetheless, excessive coupling tends to diminish the dexterity of hands. This paper tackles the trade-off between actuation complexity and dexterity by proposing an anthropomorphic finger topology with 4 DoFs driven by 2 actuators, and by developing an adaptive, modular dexterous hand based on this finger topology. We explore the biological basis of hand synergies and human gesture analysis, translating joint-level coordination and structural attributes into a modular finger architecture. Leveraging these biomimetic mappings, we design a five-finger modular hand and establish its kinematic model to analyze adaptive grasping and in-hand manipulation. Finally, we construct a physical prototype and conduct preliminary experiments, which validate the effectiveness of the proposed design and analysis.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22100
PDF
-
Dance Style Classification using Laban-Inspired and Frequency-Domain Motion Features
2025-11-25 16:33:45Ben Hamscher, Arnold Brosch, Nicolas Binninger, Maksymilian Jan Dejna, Kira MaagAbstract
Dance is an essential component of human culture and serves as a tool for conveying emotions and telling stories. Identifying and distinguishing dance genres based on motion data is a complex problem in human activity recognition, as many styles share similar poses, gestures, and temporal motion patterns. This work presents a lightweight framework for classifying dance styles that determines motion characteristics based on pose estimates extracted from videos. We propose temporal-spatial descriptors inspired by Laban Movement Analysis. These features capture local joint dynamics such as velocity, acceleration, and angular movement of the upper body, enabling a structured representation of spatial coordination. To further encode rhythmic and periodic aspects of movement, we integrate Fast Fourier Transform features that characterize movement patterns in the frequency domain. The proposed approach achieves robust classification of different dance styles with low computational effort, as complex model architectures are not required, and shows that interpretable motion representations can effectively capture stylistic nuances.
Abstract (translated)
URL
https://arxiv.org/abs/2511.20469
PDF
-
Scale What Counts, Mask What Matters: Evaluating Foundation Models for Zero-Shot Cross-Domain Wi-Fi Sensing
2025-11-24 05:46:29Cheng Jiang, Yihe Yan, Yanxiang Wang, Chun Tung Chou, Wen HuAbstract
While Wi-Fi sensing offers a compelling, privacy-preserving alternative to cameras, its practical utility has been fundamentally undermined by a lack of robustness across domains. Models trained in one setup fail to generalize to new environments, hardware, or users, a critical "domain shift" problem exacerbated by modest, fragmented public datasets. We shift from this limited paradigm and apply a foundation model approach, leveraging Masked Autoencoding (MAE) style pretraining on the largest and most heterogeneous Wi-Fi CSI datasets collection assembled to date. Our study pretrains and evaluates models on over 1.3 million samples extracted from 14 datasets, collected using 4 distinct devices across the 2.4/5/6 GHz bands and bandwidths from 20 to 160 MHz. Our large-scale evaluation is the first to systematically disentangle the impacts of data diversity versus model capacity on cross-domain performance. The results establish scaling trends on Wi-Fi CSI sensing. First, our experiments show log-linear improvements in unseen domain performance as the amount of pretraining data increases, suggesting that data scale and diversity are key to domain generalization. Second, based on the current data volume, larger model can only provide marginal gains for cross-domain performance, indicating that data, rather than model capacity, is the current bottleneck for Wi-Fi sensing generalization. Finally, we conduct a series of cross-domain evaluations on human activity recognition, human gesture recognition and user identification tasks. The results show that the large-scale pretraining improves cross-domain accuracy ranging from 2.2% to 15.7%, compared to the supervised learning baseline. Overall, our findings provide insightful direction for designing future Wi-Fi sensing systems that can eventually be robust enough for real-world deployment.
Abstract (translated)
URL
https://arxiv.org/abs/2511.18792
PDF
