Gesture
-
Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Framework
2025-11-05 17:11:53Dogucan Yaman, Fevziye Irem Eyiokur, Haz{\i}m Kemal Ekenel, Alexander WaibelAbstract
Inpainting-based talking face generation aims to preserve video details such as pose, lighting, and gestures while modifying only lip motion, often using an identity reference image to maintain speaker consistency. However, this mechanism can introduce lip leaking, where generated lips are influenced by the reference image rather than solely by the driving audio. Such leakage is difficult to detect with standard metrics and conventional test setup. To address this, we propose a systematic evaluation methodology to analyze and quantify lip leakage. Our framework employs three complementary test setups: silent-input generation, mismatched audio-video pairing, and matched audio-video synthesis. We also introduce derived metrics including lip-sync discrepancy and silent-audio-based lip-sync scores. In addition, we study how different identity reference selections affect leakage, providing insights into reference design. The proposed methodology is model-agnostic and establishes a more reliable benchmark for future research in talking face generation.
Abstract (translated)
URL
https://arxiv.org/abs/2511.08613
PDF
-
Gesture Generation Needs Improved Human Evaluation Practices: Insights from a Community-Driven State-of-the-Art Benchmark
2025-11-03 05:17:28Rajmund Nagy (KTH Royal Institute of Technology), Hendric Voss (Bielefeld University), Thanh Hoang-Minh (University of Science -- VNUHCM), Mihail Tsakov (Independent Researcher), Teodor Nikolov (Motorica AB), Zeyi Zhang (Peking University), Tenglong Ao (Peking University), Sicheng Yang (Huawei Technologies Ltd), Shaoli Huang (Astribot), Yongkang Cheng (Astribot), M. Hamza Mughal (Max-Planck Institute for Informatics, SIC), Rishabh Dabral (Max-Planck Institute for Informatics, SIC), Kiran Chhatre (KTH Royal Institute of Technology), Christian Theobalt (Max-Planck Institute for Informatics, SIC), Libin Liu (Peking University), Stefan Kopp (Bielefeld University), Rachel McDonnell (Trinity College Dublin), Michael Neff (University of California, Davis), Taras Kucherenko (SEED -- Electronic Arts), Youngwoo Yoon (Electronics and Telecommunications Research Institute), Gustav Eje Henter (KTH Royal Institute of Technology, Motorica AB)Abstract
We review human evaluation practices in automated, speech-driven 3D gesture generation and find a lack of standardisation and frequent use of flawed experimental setups. This leads to a situation where it is impossible to know how different methods compare, or what the state of the art is. In order to address common shortcomings of evaluation design, and to standardise future user studies in gesture-generation works, we introduce a detailed human evaluation protocol for the widely-used BEAT2 motion-capture dataset. Using this protocol, we conduct large-scale crowdsourced evaluation to rank six recent gesture-generation models -- each trained by its original authors -- across two key evaluation dimensions: motion realism and speech-gesture alignment. Our results provide strong evidence that 1) newer models do not consistently outperform earlier approaches; 2) published claims of high motion realism or speech-gesture alignment may not hold up under rigorous evaluation; and 3) the field must adopt disentangled assessments of motion quality and multimodal alignment for accurate benchmarking in order to make progress. Finally, in order to drive standardisation and enable new evaluation research, we will release five hours of synthetic motion from the benchmarked models; over 750 rendered video stimuli from the user studies -- enabling new evaluations without model reimplementation required -- alongside our open-source rendering script, and the 16,000 pairwise human preference votes collected for our benchmark.
Abstract (translated)
URL
https://arxiv.org/abs/2511.01233
PDF
-
Rhythm in the Air: Vision-based Real-Time Music Generation through Gestures
2025-11-02 04:07:05Barathi Subramanian, Rathinaraja Jeyaraj, Anand Paul, Kapilya GangadharanAbstract
Gesture recognition is an essential component of human-computer interaction (HCI), facilitating seamless interconnectivity between users and computer systems without physical touch. This paper introduces an innovative application of vision-based dynamic gesture recognition (VDGR) for real-time music composition through gestures. To implement this application, we generate a custom gesture dataset that encompasses over 15000 samples across 21 classes, incorporating 7 musical notes each manifesting at three distinct pitch levels. To effectively deal with the modest volume of training data and to accurately discern and prioritize complex gesture sequences for music creation, we develop a multi-layer attention-based gated recurrent unit (MLA-GRU) model, in which gated recurrent unit (GRU) is used to learn temporal patterns from the observed sequence and an attention layer is employed to focus on musically pertinent gesture segments. Our empirical studies demonstrate that MLA-GRU significantly surpasses the classical GRU model, achieving a remarkable accuracy of 96.83% compared to the baseline's 86.7%. Moreover, our approach exhibits superior efficiency and processing speed, which are crucial for interactive applications. Using our proposed system, we believe that people will interact with music in a new and exciting way. It not only advances HCI experiences but also highlights MLA-GRU's effectiveness in scenarios demanding swift and precise gesture recognition.
Abstract (translated)
URL
https://arxiv.org/abs/2511.00793
PDF
-
Spiking Patches: Asynchronous, Sparse, and Efficient Tokens for Event Cameras
2025-10-30 15:40:34Christoffer Koo {\O}hrstr{\o}m, Ronja G\"uldenring, Lazaros NalpantidisAbstract
We propose tokenization of events and present a tokenizer, Spiking Patches, specifically designed for event cameras. Given a stream of asynchronous and spatially sparse events, our goal is to discover an event representation that preserves these properties. Prior works have represented events as frames or as voxels. However, while these representations yield high accuracy, both frames and voxels are synchronous and decrease the spatial sparsity. Spiking Patches gives the means to preserve the unique properties of event cameras and we show in our experiments that this comes without sacrificing accuracy. We evaluate our tokenizer using a GNN, PCN, and a Transformer on gesture recognition and object detection. Tokens from Spiking Patches yield inference times that are up to 3.4x faster than voxel-based tokens and up to 10.4x faster than frames. We achieve this while matching their accuracy and even surpassing in some cases with absolute improvements up to 3.8 for gesture recognition and up to 1.4 for object detection. Thus, tokenization constitutes a novel direction in event-based vision and marks a step towards methods that preserve the properties of event cameras.
Abstract (translated)
URL
https://arxiv.org/abs/2510.26614
PDF
-
IBIS: A Powerful Hybrid Architecture for Human Activity Recognition
2025-10-28 20:06:08Alison M. Fernandes, Hermes I. Del Monego, Bruno S. Chang, Anelise Munaretto, H\'elder M. Fontes, Rui L. CamposAbstract
The increasing interest in Wi-Fi sensing stems from its potential to capture environmental data in a low-cost, non-intrusive way, making it ideal for applications like healthcare, space occupancy analysis, and gesture-based IoT control. However, a major limitation in this field is the common problem of overfitting, where models perform well on training data but fail to generalize to new data. To overcome this, we introduce a novel hybrid architecture that integrates Inception-BiLSTM with a Support Vector Machine (SVM), which we refer to as IBIS. Our IBIS approach is uniquely engineered to improve model generalization and create more robust classification boundaries. By applying this method to Doppler-derived data, we achieve a movement recognition accuracy of nearly 99%. Comprehensive performance metrics and confusion matrices confirm the significant effectiveness of our proposed solution.
Abstract (translated)
URL
https://arxiv.org/abs/2510.24936
PDF
-
Your Microphone Array Retains Your Identity: A Robust Voice Liveness Detection System for Smart Speakers
2025-10-28 13:05:30Yan Meng, Jiachun Li, Matthew Pillari, Arjun Deopujari, Liam Brennan, Hafsah Shamsie, Haojin Zhu, Yuan TianAbstract
Though playing an essential role in smart home systems, smart speakers are vulnerable to voice spoofing attacks. Passive liveness detection, which utilizes only the collected audio rather than the deployed sensors to distinguish between live-human and replayed voices, has drawn increasing attention. However, it faces the challenge of performance degradation under the different environmental factors as well as the strict requirement of the fixed user gestures. In this study, we propose a novel liveness feature, array fingerprint, which utilizes the microphone array inherently adopted by the smart speaker to determine the identity of collected audios. Our theoretical analysis demonstrates that by leveraging the circular layout of microphones, compared with existing schemes, array fingerprint achieves a more robust performance under the environmental change and user's movement. Then, to leverage such a fingerprint, we propose ARRAYID, a lightweight passive detection scheme, and elaborate a series of features working together with array fingerprint. Our evaluation on the dataset containing 32,780 audio samples and 14 spoofing devices shows that ARRAYID achieves an accuracy of 99.84%, which is superior to existing passive liveness detection schemes.
Abstract (translated)
URL
https://arxiv.org/abs/2510.24393
PDF
-
emg2speech: synthesizing speech from electromyography using self-supervised speech models
2025-10-28 00:50:15Harshavardhana T. Gowda, Lee M. MillerAbstract
We present a neuromuscular speech interface that translates electromyographic (EMG) signals collected from orofacial muscles during speech articulation directly into audio. We show that self-supervised speech (SS) representations exhibit a strong linear relationship with the electrical power of muscle action potentials: SS features can be linearly mapped to EMG power with a correlation of $r = 0.85$. Moreover, EMG power vectors corresponding to different articulatory gestures form structured and separable clusters in feature space. This relationship: $\text{SS features}$ $\xrightarrow{\texttt{linear mapping}}$ $\text{EMG power}$ $\xrightarrow{\texttt{gesture-specific clustering}}$ $\text{articulatory movements}$, highlights that SS models implicitly encode articulatory mechanisms. Leveraging this property, we directly map EMG signals to SS feature space and synthesize speech, enabling end-to-end EMG-to-speech generation without explicit articulatory models and vocoder training.
Abstract (translated)
URL
https://arxiv.org/abs/2510.23969
PDF
-
Enabling Vibration-Based Gesture Recognition on Everyday Furniture via Energy-Efficient FPGA Implementation of 1D Convolutional Networks
2025-10-27 09:30:36Koki Shibata, Tianheng Ling, Chao Qian, Tomokazu Matsui, Hirohiko Suwa, Keiichi Yasumoto, Gregor SchieleAbstract
The growing demand for smart home interfaces has increased interest in non-intrusive sensing methods like vibration-based gesture recognition. While prior studies demonstrated feasibility, they often rely on complex preprocessing and large Neural Networks (NNs) requiring costly high-performance hardware, resulting in high energy usage and limited real-world deployability. This study proposes an energy-efficient solution deploying compact NNs on low-power Field-Programmable Gate Arrays (FPGAs) to enable real-time gesture recognition with competitive accuracy. We adopt a series of optimizations: (1) We replace complex spectral preprocessing with raw waveform input, eliminating complex on-board preprocessing while reducing input size by 21x without sacrificing accuracy. (2) We design two lightweight architectures (1D-CNN and 1D-SepCNN) tailored for embedded FPGAs, reducing parameters from 369 million to as few as 216 while maintaining comparable accuracy. (3) With integer-only quantization and automated RTL generation, we achieve seamless FPGA deployment. A ping-pong buffering mechanism in 1D-SepCNN further improves deployability under tight memory constraints. (4) We extend a hardware-aware search framework to support constraint-driven model configuration selection, considering accuracy, deployability, latency, and energy consumption. Evaluated on two swipe-direction datasets with multiple users and ordinary tables, our approach achieves low-latency, energy-efficient inference on the AMD Spartan-7 XC7S25 FPGA. Under the PS data splitting setting, the selected 6-bit 1D-CNN reaches 0.970 average accuracy across users with 9.22 ms latency. The chosen 8-bit 1D-SepCNN further reduces latency to 6.83 ms (over 53x CPU speedup) with slightly lower accuracy (0.949). Both consume under 1.2 mJ per inference, demonstrating suitability for long-term edge operation.
Abstract (translated)
URL
https://arxiv.org/abs/2510.23156
PDF
-
Reconnaissance Automatique des Langues des Signes : Une Approche Hybrid'ee CNN-LSTM Bas'ee sur Mediapipe
2025-10-24 20:25:25Fraisse Sacr\'e Takouchouang, Ho Tuong VinhAbstract
Sign languages play a crucial role in the communication of deaf communities, but they are often marginalized, limiting access to essential services such as healthcare and education. This study proposes an automatic sign language recognition system based on a hybrid CNN-LSTM architecture, using Mediapipe for gesture keypoint extraction. Developed with Python, TensorFlow and Streamlit, the system provides real-time gesture translation. The results show an average accuracy of 92\%, with very good performance for distinct gestures such as ``Hello'' and ``Thank you''. However, some confusions remain for visually similar gestures, such as ``Call'' and ``Yes''. This work opens up interesting perspectives for applications in various fields such as healthcare, education and public services.
Abstract (translated)
URL
https://arxiv.org/abs/2510.22011
PDF
-
SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generation
2025-10-24 03:19:48Alec Helbling, Shruti Palaskar, Kundan Krishna, Polo Chau, Leon Gatys, Joseph Yitan ChengAbstract
What exactly makes a particular image unsafe? Systematically differentiating between benign and problematic images is a challenging problem, as subtle changes to an image, such as an insulting gesture or symbol, can drastically alter its safety implications. However, existing image safety datasets are coarse and ambiguous, offering only broad safety labels without isolating the specific features that drive these differences. We introduce SafetyPairs, a scalable framework for generating counterfactual pairs of images, that differ only in the features relevant to the given safety policy, thus flipping their safety label. By leveraging image editing models, we make targeted changes to images that alter their safety labels while leaving safety-irrelevant details unchanged. Using SafetyPairs, we construct a new safety benchmark, which serves as a powerful source of evaluation data that highlights weaknesses in vision-language models' abilities to distinguish between subtly different images. Beyond evaluation, we find our pipeline serves as an effective data augmentation strategy that improves the sample efficiency of training lightweight guard models. We release a benchmark containing over 3,020 SafetyPair images spanning a diverse taxonomy of 9 safety categories, providing the first systematic resource for studying fine-grained image safety distinctions.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21120
PDF
-
OmniMotion-X: Versatile Multimodal Whole-Body Motion Generation
2025-10-22 17:25:33Guowei Xu, Yuxuan Bian, Ailing Zeng, Mingyi Shi, Shaoli Huang, Wen Li, Lixin Duan, Qiang XuAbstract
This paper introduces OmniMotion-X, a versatile multimodal framework for whole-body human motion generation, leveraging an autoregressive diffusion transformer in a unified sequence-to-sequence manner. OmniMotion-X efficiently supports diverse multimodal tasks, including text-to-motion, music-to-dance, speech-to-gesture, and global spatial-temporal control scenarios (e.g., motion prediction, in-betweening, completion, and joint/trajectory-guided synthesis), as well as flexible combinations of these tasks. Specifically, we propose the use of reference motion as a novel conditioning signal, substantially enhancing the consistency of generated content, style, and temporal dynamics crucial for realistic animations. To handle multimodal conflicts, we introduce a progressive weak-to-strong mixed-condition training strategy. To enable high-quality multimodal training, we construct OmniMoCap-X, the largest unified multimodal motion dataset to date, integrating 28 publicly available MoCap sources across 10 distinct tasks, standardized to the SMPL-X format at 30 fps. To ensure detailed and consistent annotations, we render sequences into videos and use GPT-4o to automatically generate structured and hierarchical captions, capturing both low-level actions and high-level semantics. Extensive experimental evaluations confirm that OmniMotion-X significantly surpasses existing methods, demonstrating state-of-the-art performance across multiple multimodal tasks and enabling the interactive generation of realistic, coherent, and controllable long-duration motions.
Abstract (translated)
本文介绍了OmniMotion-X，这是一个用于全身人类动作生成的多功能多模态框架，在统一的序列到序列方式中利用自回归扩散变压器。OmniMotion-X高效支持各种多模态任务，包括文本到动作、音乐到舞蹈、语音到手势以及全局空间时间控制场景（例如运动预测、插值、完成和关节/轨迹引导合成），并且还支持这些任务之间的灵活组合。 具体而言，我们提出将参考动作作为新颖的条件信号使用，这在很大程度上增强了生成内容的一致性、风格和关键的时间动态特性，这对于现实主义动画至关重要。为了处理多模态冲突，我们引入了一种逐步从弱到强混合条件训练策略。为了实现高质量的多模态训练，我们构建了OmniMoCap-X，这是迄今为止最大的统一多模态运动数据集，整合了28个公开的Motion Capture (MoCap) 数据源，涵盖10项不同的任务，并标准化为SMPL-X格式，以每秒30帧的速度进行。 为了确保详细且一致的注释，我们渲染序列成视频，并使用GPT-4o自动生成结构化和分层标题，捕捉低级动作和高级语义。广泛的实验评估确认OmniMotion-X在多个多模态任务上显著优于现有方法，展示了最先进的性能，并能够生成真实、连贯且可控的长持续时间运动。
URL
https://arxiv.org/abs/2510.19789
PDF
-
Modeling Turn-Taking with Semantically Informed Gestures
2025-10-22 08:17:54Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera DembergAbstract
In conversation, humans use multimodal cues, such as speech, gestures, and gaze, to manage turn-taking. While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions. To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types. Using this dataset, we model turn-taking prediction through a Mixture-of-Experts framework integrating text, audio, and gestures. Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in multimodal turn-taking.
Abstract (translated)
在对话中，人类使用多种模态的线索（如言语、手势和目光交流）来管理轮流发言。虽然语言和声学特征具有信息性，但手势提供了互补性的线索，有助于建模这些转换过程。为了研究这一问题，我们引入了DnD Gesture++，这是一个扩展版的多方对话DnD Gesture语料库，其中新增了2,663个语义手势标注，涵盖了象形、隐喻、指示和话语类型的手势。利用这个数据集，我们通过混合专家框架整合文本、音频和手势来建模轮流发言预测。实验表明，引入由语义指导的手势能持续地优于基线模型，在多模态轮流发言中展示了它们的互补作用。
URL
https://arxiv.org/abs/2510.19350
PDF
-
Gestura: A LVLM-Powered System Bridging Motion and Semantics for Real-Time Free-Form Gesture Understanding
2025-10-21 14:46:48Zhuoming Li, Aitong Liu, Mengxi Jia, Tengxiang Zhang, Dell Zhang, Xuelong LiAbstract
Free-form gesture understanding is highly appealing for human-computer interaction, as it liberates users from the constraints of predefined gesture categories. However, the sole existing solution GestureGPT suffers from limited recognition accuracy and slow response times. In this paper, we propose Gestura, an end-to-end system for free-form gesture understanding. Gestura harnesses a pre-trained Large Vision-Language Model (LVLM) to align the highly dynamic and diverse patterns of free-form gestures with high-level semantic concepts. To better capture subtle hand movements across different styles, we introduce a Landmark Processing Module that compensate for LVLMs' lack of fine-grained domain knowledge by embedding anatomical hand priors. Further, a Chain-of-Thought (CoT) reasoning strategy enables step-by-step semantic inference, transforming shallow knowledge into deep semantic understanding and significantly enhancing the model's ability to interpret ambiguous or unconventional gestures. Together, these components allow Gestura to achieve robust and adaptable free-form gesture comprehension. Additionally, we have developed the first open-source dataset for free-form gesture intention reasoning and understanding with over 300,000 annotated QA pairs.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21814
PDF
-
ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Input
2025-10-20 15:01:56Hendric Voss, Stefan KoppAbstract
Human communication combines speech with expressive nonverbal cues such as hand gestures that serve manifold communicative functions. Yet, current generative gesture generation approaches are restricted to simple, repetitive beat gestures that accompany the rhythm of speaking but do not contribute to communicating semantic meaning. This paper tackles a core challenge in co-speech gesture synthesis: generating iconic or deictic gestures that are semantically coherent with a verbal utterance. Such gestures cannot be derived from language input alone, which inherently lacks the visual meaning that is often carried autonomously by gestures. We therefore introduce a zero-shot system that generates gestures from a given language input and additionally is informed by imagistic input, without manual annotation or human intervention. Our method integrates an image analysis pipeline that extracts key object properties such as shape, symmetry, and alignment, together with a semantic matching module that links these visual details to spoken text. An inverse kinematics engine then synthesizes iconic and deictic gestures and combines them with co-generated natural beat gestures for coherent multimodal communication. A comprehensive user study demonstrates the effectiveness of our approach. In scenarios where speech alone was ambiguous, gestures generated by our system significantly improved participants' ability to identify object properties, confirming their interpretability and communicative value. While challenges remain in representing complex shapes, our results highlight the importance of context-aware semantic gestures for creating expressive and collaborative virtual agents or avatars, marking a substantial step forward towards efficient and robust, embodied human-agent interaction. More information and example videos are available here: this https URL
Abstract (translated)
人类交流结合了言语与表达性非语言线索，如手势，这些手势在沟通中具有多种功能。然而，当前的生成式手势生成方法仅限于伴随说话节奏出现的简单、重复的手势，并且不参与传递语义意义。本文解决了一个核心挑战：如何根据口头表述合成具象化或指示性的手势，这些手势与语言输入本身所缺乏的视觉含义密切相关。 我们因此引入了一种零样本系统，该系统从给定的语言输入中生成手势，并通过图象输入进一步完善其功能，而无需人工注释或人为干预。我们的方法整合了一个图像分析管道，提取形状、对称性和对齐等关键物体属性，以及一个语义匹配模块将这些视觉细节与口语文本连接起来。随后，逆向动力学引擎会合成具象化和指示性的手势，并将其与自动生成的自然节奏手势结合，以实现连贯的多模态沟通。 一项全面的用户研究证明了我们方法的有效性。在仅凭言语无法明确传达信息的情况下，系统生成的手势显著提升了参与者识别物体属性的能力，从而证实了其可解释性和交流价值。虽然代表复杂形状仍有挑战，但我们的成果强调了语境感知手势对于创造具有表现力和协作性的虚拟代理或化身的重要性，这标志着朝向高效且稳健的人机互动迈出了重要一步。 更多信息及示例视频请参见：[提供的链接]
URL
https://arxiv.org/abs/2510.17617
PDF
-
Conveying Meaning through Gestures: An Investigation into Semantic Co-Speech Gesture Generation
2025-10-20 14:47:56Hendric Voss, Lisa Michelle Bohnenkamp, Stefan KoppAbstract
This study explores two frameworks for co-speech gesture generation, AQ-GT and its semantically-augmented variant AQ-GT-a, to evaluate their ability to convey meaning through gestures and how humans perceive the resulting movements. Using sentences from the SAGA spatial communication corpus, contextually similar sentences, and novel movement-focused sentences, we conducted a user-centered evaluation of concept recognition and human-likeness. Results revealed a nuanced relationship between semantic annotations and performance. The original AQ-GT framework, lacking explicit semantic input, was surprisingly more effective at conveying concepts within its training domain. Conversely, the AQ-GT-a framework demonstrated better generalization, particularly for representing shape and size in novel contexts. While participants rated gestures from AQ-GT-a as more expressive and helpful, they did not perceive them as more human-like. These findings suggest that explicit semantic enrichment does not guarantee improved gesture generation and that its effectiveness is highly dependent on the context, indicating a potential trade-off between specialization and generalization.
Abstract (translated)
这项研究探讨了两种共话语势生成框架，AQ-GT及其语义增强变体AQ-GT-a，以评估它们通过手势传达意义的能力以及人类如何感知这些动作。利用SAGA空间交流语料库中的句子、上下文相似的句子和新型运动焦点句子，我们进行了一项以人为中心的概念识别和人性化程度评估。结果揭示了语义标注与性能之间存在复杂的关系。 原始AQ-GT框架由于缺乏显式语义输入，在其训练领域内传达概念的效果反而更佳。相反，AQ-GT-a框架在表示形状和大小的新上下文中表现出更好的泛化能力。尽管参与者认为来自AQ-GT-a的手势更具表现力且更有帮助，但他们并没有觉得这些手势更加人性化。 这些发现表明，显式的语义丰富并不保证能提高手势生成的效果，并且其效果高度依赖于具体的情境，暗示了专业化与泛化之间可能存在权衡。
URL
https://arxiv.org/abs/2510.17599
PDF
-
Embody 3D: A Large-scale Multimodal Motion and Behavior Dataset
2025-10-17 23:06:36Claire McLean, Makenzie Meendering, Tristan Swartz, Orri Gabbay, Alexandra Olsen, Rachel Jacobs, Nicholas Rosen, Philippe de Bree, Tony Garcia, Gadsden Merrill, Jake Sandakly, Julia Buffalini, Neham Jain, Steven Krenn, Moneish Kumar, Dejan Markovic, Evonne Ng, Fabian Prada, Andrew Saba, Siwei Zhang, Vasu Agrawal, Tim Godisart, Alexander Richard, Michael ZollhoeferAbstract
The Codec Avatars Lab at Meta introduces Embody 3D, a multimodal dataset of 500 individual hours of 3D motion data from 439 participants collected in a multi-camera collection stage, amounting to over 54 million frames of tracked 3D motion. The dataset features a wide range of single-person motion data, including prompted motions, hand gestures, and locomotion; as well as multi-person behavioral and conversational data like discussions, conversations in different emotional states, collaborative activities, and co-living scenarios in an apartment-like space. We provide tracked human motion including hand tracking and body shape, text annotations, and a separate audio track for each participant.
Abstract (translated)
Meta的Codec Avatars实验室推出了Embody 3D，这是一个包含来自439名参与者的500小时多模态三维动作数据的数据集。这些数据是通过一个多摄像头采集阶段收集而来的，累计追踪了超过5400万帧的三维动作。该数据集涵盖了广泛的一人动作数据，包括引导的动作、手势和行走；以及多人行为与对话数据，如讨论、不同情绪状态下的对话、协作活动及公寓内的共同生活场景。我们还提供了每个参与者的跟踪人体运动（包括手部追踪和体型）、文本注释以及单独的音频轨道。
URL
https://arxiv.org/abs/2510.16258
PDF
-
OmniMotion: Multimodal Motion Generation with Continuous Masked Autoregression
2025-10-16 17:57:53Zhe Li, Weihao Yuan, Weichao Shen, Siyu Zhu, Zilong Dong, Chang XuAbstract
Whole-body multi-modal human motion generation poses two primary challenges: creating an effective motion generation mechanism and integrating various modalities, such as text, speech, and music, into a cohesive framework. Unlike previous methods that usually employ discrete masked modeling or autoregressive modeling, we develop a continuous masked autoregressive motion transformer, where a causal attention is performed considering the sequential nature within the human motion. Within this transformer, we introduce a gated linear attention and an RMSNorm module, which drive the transformer to pay attention to the key actions and suppress the instability caused by either the abnormal movements or the heterogeneous distributions within multi-modalities. To further enhance both the motion generation and the multimodal generalization, we employ the DiT structure to diffuse the conditions from the transformer towards the targets. To fuse different modalities, AdaLN and cross-attention are leveraged to inject the text, speech, and music signals. Experimental results demonstrate that our framework outperforms previous methods across all modalities, including text-to-motion, speech-to-gesture, and music-to-dance. The code of our method will be made public.
Abstract (translated)
全身多模态人体动作生成面临两个主要挑战：创建有效的运动生成机制以及将各种模式（如文本、语音和音乐）整合到一个连贯的框架中。与以往通常采用离散掩码建模或自回归建模的方法不同，我们开发了一种连续的掩码自回归运动变换器，在该模型中考虑到人体动作中的序列特性而执行因果注意力操作。在这个变换器内，我们引入了门控线性注意机制和RMSNorm模块，这些设计促使变换器关注关键的动作，并抑制由于异常移动或多模态分布差异所导致的不稳定性。 为了进一步增强运动生成和多模式泛化能力，我们采用了DiT结构来从变换器向目标扩散条件。在融合不同模式时，我们利用AdaLN和交叉注意力机制将文本、语音和音乐信号注入到模型中。实验结果表明，我们的框架在所有模态（包括文本到动作、语音到手势以及音乐到舞蹈）上都优于现有方法。 我们的方法代码将在适当的时候公开发布。
URL
https://arxiv.org/abs/2510.14954
PDF
-
MOTION: ML-Assisted On-Device Low-Latency Motion Recognition
2025-10-14 01:15:47Veeramani Pugazhenthi, Wei-Hsiang Chu, Junwei Lu, Jadyn N. Miyahira, Soheil SalehiAbstract
The use of tiny devices capable of low-latency gesture recognition is gaining momentum in everyday human-computer interaction and especially in medical monitoring fields. Embedded solutions such as fall detection, rehabilitation tracking, and patient supervision require fast and efficient tracking of movements while avoiding unwanted false alarms. This study presents an efficient solution on how to build very efficient motion-based models only using triaxial accelerometer sensors. We explore the capability of the AutoML pipelines to extract the most important features from the data segments. This approach also involves training multiple lightweight machine learning algorithms using the extracted features. We use WeBe Band, a multi-sensor wearable device that is equipped with a powerful enough MCU to effectively perform gesture recognition entirely on the device. Of the models explored, we found that the neural network provided the best balance between accuracy, latency, and memory use. Our results also demonstrate that reliable real-time gesture recognition can be achieved in WeBe Band, with great potential for real-time medical monitoring solutions that require a secure and fast response time.
Abstract (translated)
使用能够进行低延迟手势识别的微型设备在日常人机交互中，尤其是在医疗监测领域中正变得越来越受欢迎。嵌入式解决方案如跌倒检测、康复跟踪和患者监护需要快速且高效的运动追踪，并同时避免不必要的误报。本研究提出了一种有效的方法来构建仅利用三轴加速度计传感器的高度高效的动作模型。我们探索了自动机器学习（AutoML）管道从数据段中提取最重要的特征的能力。该方法还涉及使用提取的特征训练多个轻量级机器学习算法。 我们采用WeBe Band，这是一种配备了足够强大的MCU以完全在设备上执行手势识别的多传感器可穿戴设备。在研究的各种模型中，我们发现神经网络提供了准确度、延迟和内存使用的最佳平衡。我们的结果还表明，在WeBe Band中可以实现可靠且实时的手势识别，这为需要安全快速响应时间的实时医疗监测解决方案带来了巨大潜力。
URL
https://arxiv.org/abs/2512.00008
PDF
-
Nepali Sign Language Characters Recognition: Dataset Development and Deep Learning Approaches
2025-10-13 10:29:08Birat Poudel, Satyam Ghimire, Sijan Bhattarai, Saurav Bhandari, Suramya Sharma DahalAbstract
Sign languages serve as essential communication systems for individuals with hearing and speech impairments. However, digital linguistic dataset resources for underrepresented sign languages, such as Nepali Sign Language (NSL), remain scarce. This study introduces the first benchmark dataset for NSL, consisting of 36 gesture classes with 1,500 samples per class, designed to capture the structural and visual features of the language. To evaluate recognition performance, we fine-tuned MobileNetV2 and ResNet50 architectures on the dataset, achieving classification accuracies of 90.45% and 88.78%, respectively. These findings demonstrate the effectiveness of convolutional neural networks in sign recognition tasks, particularly within low-resource settings. To the best of our knowledge, this work represents the first systematic effort to construct a benchmark dataset and assess deep learning approaches for NSL recognition, highlighting the potential of transfer learning and fine-tuning for advancing research in underexplored sign languages.
Abstract (translated)
手语作为听力和言语障碍人士的重要沟通方式，发挥着不可或缺的作用。然而，对于如尼泊尔手语（NSL）等代表性不足的手语来说，数字语言数据集资源仍然非常稀缺。本研究首次介绍了针对NSL的基准数据集，该数据集包含36个手势类别，每个类别有1500个样本，旨在捕捉该语言的结构和视觉特征。 为了评估识别性能，我们在MobileNetV2和ResNet50架构上对数据集进行了微调，分别实现了90.45%和88.78%的分类准确率。这些发现证明了卷积神经网络在手语识别任务中的有效性，尤其是在资源匮乏的情况下。 据我们所知，这项工作代表了首次系统性地为NSL构建基准数据集并评估深度学习方法的研究尝试，突显了迁移学习和微调技术对于探索未被充分研究的手语领域的潜力。
URL
https://arxiv.org/abs/2510.11243
PDF
-
Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Prediction
2025-10-13 09:51:26T\'eo Guichoux, Th\'eodor Lemerle, Shivam Mehta, Jonas Beskow, Gustave Eje Henter, Laure Soulier, Catherine Pelachaud, Nicolas ObinAbstract
Human communication is multimodal, with speech and gestures tightly coupled, yet most computational methods for generating speech and gestures synthesize them sequentially, weakening synchrony and prosody alignment. We introduce Gelina, a unified framework that jointly synthesizes speech and co-speech gestures from text using interleaved token sequences in a discrete autoregressive backbone, with modality-specific decoders. Gelina supports multi-speaker and multi-style cloning and enables gesture-only synthesis from speech inputs. Subjective and objective evaluations demonstrate competitive speech quality and improved gesture generation over unimodal baselines.
Abstract (translated)
人类的交流是多模态的，言语和手势紧密相连，然而大多数用于生成言语和手势的计算方法都是顺序合成的，这会削弱同步性和语调的一致性。我们介绍了 Gelina，这是一个统一框架，它使用离散自回归主干中的交错令牌序列从文本中联合合成语音和伴随语言的手势，并且配有特定于每种模式的解码器。Gelina 支持多说话人和多种风格的克隆，并能够仅通过语音输入生成手势。主观和客观评估表明，与单模态基线相比，该框架在言语质量方面具有竞争力，在手势生成方面有所改进。
URL
https://arxiv.org/abs/2510.12834
PDF
