Gesture
-
A novel gesture interaction control method for rehabilitation lower extremity exoskeleton
2025-04-02 16:46:01Shuang Qiu, Zhongcai Pei, Chen Wang, Jing Zhang, Zhiyong TangAbstract
With the rapid development of Rehabilitation Lower Extremity Robotic Exoskeletons (RLEEX) technology, significant advancements have been made in Human-Robot Interaction (HRI) methods. These include traditional physical HRI methods that are easily recognizable and various bio-electrical signal-based HRI methods that can visualize and predict actions. However, most of these HRI methods are contact-based, facing challenges such as operational complexity, sensitivity to interference, risks associated with implantable devices, and, most importantly, limitations in comfort. These challenges render the interaction less intuitive and natural, which can negatively impact patient motivation for rehabilitation. To address these issues, this paper proposes a novel non-contact gesture interaction control method for RLEEX, based on RGB monocular camera depth estimation. This method integrates three key steps: detecting keypoints, recognizing gestures, and assessing distance, thereby applying gesture information and augmented reality triggering technology to control gait movements of RLEEX. Results indicate that this approach provides a feasible solution to the problems of poor comfort, low reliability, and high latency in HRI for RLEEX platforms. Specifically, it achieves a gesture-controlled exoskeleton motion accuracy of 94.11\% and an average system response time of 0.615 seconds through non-contact HRI. The proposed non-contact HRI method represents a pioneering advancement in control interactions for RLEEX, paving the way for further exploration and development in this field.
Abstract (translated)
随着康复下肢外骨骼机器人（RLEEX）技术的迅速发展，人机交互（HRI）方法取得了显著进展。这些方法包括易于识别的传统物理人机交互方式和基于生物电信号的人机交互方式，后者可以实现动作的可视化与预测。然而，大多数现有的人机交互方法都是接触式的，面临操作复杂、易受干扰、植入式设备风险以及最重要的是舒适度差等问题。这些问题使互动显得不够直观自然，从而可能影响患者的康复积极性。为了解决这些问题，本文提出了一种基于RGB单目摄像头深度估计的非接触手势控制RLEEX的新方法。该方法整合了三个关键步骤：检测关键点、识别手势和评估距离，利用手势信息与增强现实触发技术来控制RLEEX的姿态动作。 实验结果表明，这种方法为解决RLEEX平台人机交互中舒适度差、可靠性低以及延迟高的问题提供了可行的解决方案。具体而言，通过非接触式的人机互动方式，该方法实现了94.11%的手势控制外骨骼运动精度，并达到了平均系统响应时间为0.615秒的效果。 所提出的非接触人机交互方法代表了RLEEX控制互动领域的开创性进展，为这一领域进一步的探索和开发铺平了道路。
URL
https://arxiv.org/abs/2504.01888
PDF
-
TransforMerger: Transformer-based Voice-Gesture Fusion for Robust Human-Robot Communication
2025-04-02 13:15:59Petr Vanc, Karla StepanovaAbstract
As human-robot collaboration advances, natural and flexible communication methods are essential for effective robot control. Traditional methods relying on a single modality or rigid rules struggle with noisy or misaligned data as well as with object descriptions that do not perfectly fit the predefined object names (e.g. 'Pick that red object'). We introduce TransforMerger, a transformer-based reasoning model that infers a structured action command for robotic manipulation based on fused voice and gesture inputs. Our approach merges multimodal data into a single unified sentence, which is then processed by the language model. We employ probabilistic embeddings to handle uncertainty and we integrate contextual scene understanding to resolve ambiguous references (e.g., gestures pointing to multiple objects or vague verbal cues like "this"). We evaluate TransforMerger in simulated and real-world experiments, demonstrating its robustness to noise, misalignment, and missing information. Our results show that TransforMerger outperforms deterministic baselines, especially in scenarios requiring more contextual knowledge, enabling more robust and flexible human-robot communication. Code and datasets are available at: this http URL.
Abstract (translated)
随着人机协作的不断发展，自然且灵活的通信方法对于有效的机器人控制至关重要。传统的方法依赖单一模态或刚性规则，在处理嘈杂或对齐不良的数据时以及面对不完全符合预定义对象名称的对象描述（例如“拿起那个红色物品”）时会遇到困难。我们引入了TransforMerger，这是一种基于变压器的推理模型，它能够根据融合的声音和手势输入推断出机器人操作的结构化动作命令。我们的方法将多模态数据合并为一个统一的句子，然后由语言模型进行处理。我们采用概率嵌入来处理不确定性，并整合上下文场景理解以解决模糊引用问题（例如，指向多个对象的手势或含糊不清的语言提示如“这个”）。我们在模拟和现实世界实验中评估了TransforMerger的表现，展示了其在面对噪音、不对齐以及信息缺失时的鲁棒性。我们的结果显示，TransforMerger超越了确定性的基线模型，在需要更多上下文知识的情况下表现尤为出色，从而实现了更加强大且灵活的人机通信。代码和数据集可在以下链接获取：[此网址]（请将[此网址]替换为实际提供的URL）。
URL
https://arxiv.org/abs/2504.01708
PDF
-
CLIP-SLA: Parameter-Efficient CLIP Adaptation for Continuous Sign Language Recognition
2025-04-02 12:15:33Sarah Alyami, Hamzah LuqmanAbstract
Continuous sign language recognition (CSLR) focuses on interpreting and transcribing sequences of sign language gestures in videos. In this work, we propose CLIP sign language adaptation (CLIP-SLA), a novel CSLR framework that leverages the powerful pre-trained visual encoder from the CLIP model to sign language tasks through parameter-efficient fine-tuning (PEFT). We introduce two variants, SLA-Adapter and SLA-LoRA, which integrate PEFT modules into the CLIP visual encoder, enabling fine-tuning with minimal trainable parameters. The effectiveness of the proposed frameworks is validated on four datasets: Phoenix2014, Phoenix2014-T, CSL-Daily, and Isharah-500, where both CLIP-SLA variants outperformed several SOTA models with fewer trainable parameters. Extensive ablation studies emphasize the effectiveness and flexibility of the proposed methods with different vision-language models for CSLR. These findings showcase the potential of adapting large-scale pre-trained models for scalable and efficient CSLR, which pave the way for future advancements in sign language understanding.
Abstract (translated)
连续手语识别（CSLR）专注于解读和转录视频中的一系列手语手势。在这项工作中，我们提出了一种新颖的CSLR框架——CLIP手语适应（CLIP-SLA），该框架利用了从CLIP模型中预训练的强大视觉编码器，并通过参数高效微调（PEFT）将其应用于手语任务。我们引入了两种变体：SLA-Adapter和SLA-LoRA，它们将PEFT模块整合到CLIP视觉编码器中，从而可以在保持极少量可训练参数的情况下进行精细调整。 该框架的有效性在四个数据集上进行了验证：Phoenix2014、Phoenix2014-T、CSL-Daily 和 Isharah-500，在这些数据集中，两个 CLIP-SLA 变体均以更少的可训练参数优于多个最先进（SOTA）模型。广泛的消融研究表明了所提出的方法在不同的视觉语言模型中用于 CSLR 时的有效性和灵活性。 这些发现展示了大规模预训练模型适应于具有规模和效率优势的手语识别系统的潜力，为未来手语理解领域的进一步发展铺平了道路。
URL
https://arxiv.org/abs/2504.01666
PDF
-
MRHaD: Mixed Reality-based Hand-Drawn Map Editing Interface for Mobile Robot Navigation
2025-04-01 09:34:02Takumi Taki, Masato Kobayashi, Eduardo Iglesius, Naoya Chiba, Shizuka Shirai, Yuki UranishiAbstract
Mobile robot navigation systems are increasingly relied upon in dynamic and complex environments, yet they often struggle with map inaccuracies and the resulting inefficient path planning. This paper presents MRHaD, a Mixed Reality-based Hand-drawn Map Editing Interface that enables intuitive, real-time map modifications through natural hand gestures. By integrating the MR head-mounted display with the robotic navigation system, operators can directly create hand-drawn restricted zones (HRZ), thereby bridging the gap between 2D map representations and the real-world environment. Comparative experiments against conventional 2D editing methods demonstrate that MRHaD significantly improves editing efficiency, map accuracy, and overall usability, contributing to safer and more efficient mobile robot operations. The proposed approach provides a robust technical foundation for advancing human-robot collaboration and establishing innovative interaction models that enhance the hybrid future of robotics and human society. For additional material, please check: this https URL
Abstract (translated)
移动机器人导航系统在动态和复杂环境中越来越受到重视，但它们通常会因地图不准确而导致路径规划效率低下。本文提出了一种基于混合现实的手绘地图编辑界面MRHaD（Mixed Reality-based Hand-drawn Map Editing Interface），该界面允许用户通过自然手势进行直观、实时的地图修改。通过将混合现实头戴式显示器与机器人导航系统集成，操作员可以直接创建手绘限制区域（HRZ），从而弥合二维地图表示与真实世界环境之间的差距。 对比实验表明，MRHaD相比于传统的2D编辑方法，在提高编辑效率、地图准确性以及整体易用性方面表现显著。这些改进有助于实现更安全和高效的移动机器人作业。所提出的这种方法为推进人机协作提供了一个坚实的技术基础，并建立了一种创新的交互模型，以增强机器人技术与人类社会融合的未来。 欲了解更多信息，请访问：此链接 [请将“this https URL”替换为您实际提供的链接地址]
URL
https://arxiv.org/abs/2504.00580
PDF
-
Understanding Co-speech Gestures in-the-wild
2025-03-28 17:55:52Sindhu B Hegde, K R Prajwal, Taein Kwon, Andrew ZissermanAbstract
Co-speech gestures play a vital role in non-verbal communication. In this paper, we introduce a new framework for co-speech gesture understanding in the wild. Specifically, we propose three new tasks and benchmarks to evaluate a model's capability to comprehend gesture-text-speech associations: (i) gesture-based retrieval, (ii) gestured word spotting, and (iii) active speaker detection using gestures. We present a new approach that learns a tri-modal speech-text-video-gesture representation to solve these tasks. By leveraging a combination of global phrase contrastive loss and local gesture-word coupling loss, we demonstrate that a strong gesture representation can be learned in a weakly supervised manner from videos in the wild. Our learned representations outperform previous methods, including large vision-language models (VLMs), across all three tasks. Further analysis reveals that speech and text modalities capture distinct gesture-related signals, underscoring the advantages of learning a shared tri-modal embedding space. The dataset, model, and code are available at: this https URL
Abstract (translated)
共同言语手势在非语言交流中扮演着重要角色。本文介绍了一种新的框架，用于理解自然环境中的共言语手势。具体而言，我们提出了三个新任务和基准测试来评估模型对手势-文本-语音关联的理解能力：(i) 基于手势的检索；(ii) 手势词检测；以及 (iii) 使用手势进行活跃说话人识别。我们提出了一种新的方法，通过学习一种三模态（语音、文本和视频）的手势表示来解决这些任务。通过结合全局短语对比损失和局部手势-单词耦合损失的组合，我们展示了如何在弱监督下从野外视频中以有效的方式学习到强大的手势表示。我们的所学表现在所有三项任务上均超越了之前的模型方法，包括大规模视觉语言模型（VLMs）。进一步分析显示，语音和文本模态捕获了不同的与手势相关的信号，这强调了在共享三模态嵌入空间中进行学习的优势。数据集、模型和代码可在以下链接获取：[此网址]。
URL
https://arxiv.org/abs/2503.22668
PDF
-
ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer
2025-03-27 16:39:40Yong Xie, Yunlian Sun, Hongwen Zhang, Yebin Liu, Jinhui TangAbstract
We present ReCoM, an efficient framework for generating high-fidelity and generalizable human body motions synchronized with speech. The core innovation lies in the Recurrent Embedded Transformer (RET), which integrates Dynamic Embedding Regularization (DER) into a Vision Transformer (ViT) core architecture to explicitly model co-speech motion dynamics. This architecture enables joint spatial-temporal dependency modeling, thereby enhancing gesture naturalness and fidelity through coherent motion synthesis. To enhance model robustness, we incorporate the proposed DER strategy, which equips the model with dual capabilities of noise resistance and cross-domain generalization, thereby improving the naturalness and fluency of zero-shot motion generation for unseen speech inputs. To mitigate inherent limitations of autoregressive inference, including error accumulation and limited self-correction, we propose an iterative reconstruction inference (IRI) strategy. IRI refines motion sequences via cyclic pose reconstruction, driven by two key components: (1) classifier-free guidance improves distribution alignment between generated and real gestures without auxiliary supervision, and (2) a temporal smoothing process eliminates abrupt inter-frame transitions while ensuring kinematic continuity. Extensive experiments on benchmark datasets validate ReCoM's effectiveness, achieving state-of-the-art performance across metrics. Notably, it reduces the Fréchet Gesture Distance (FGD) from 18.70 to 2.48, demonstrating an 86.7% improvement in motion realism. Our project page is this https URL.
Abstract (translated)
我们提出了一种名为ReCoM的高效框架，用于生成与语音同步的高度真实和通用性的人体运动。该框架的核心创新在于引入了循环嵌入变换器（Recurrent Embedded Transformer, RET），它将动态嵌入正则化（Dynamic Embedding Regularization, DER）整合到视觉变换器（Vision Transformer, ViT）的架构中，以显式地建模与言语同步的身体运动动力学。这种架构通过联合空间-时间依赖性建模来增强手势的真实性和连贯性。 为了提升模型的鲁棒性，我们引入了DER策略，使模型同时具备抗噪能力和跨域泛化能力，从而改善零样本运动生成（即对于未见过的语音输入）的真实感和流畅度。为了解决自回归推理固有的问题，如误差积累和自我修正能力有限的问题，我们提出了一种迭代重构推理（Iterative Reconstruction Inference, IRI）策略。IRI通过周期性姿势重建来细化运动序列，并由两个关键组件驱动：1) 无分类器引导提高了生成手势与真实手势之间的分布一致性，无需辅助监督；2) 时间平滑过程消除了帧间突然的过渡变化，同时确保了动力学连续性。 在基准数据集上的广泛实验验证了ReCoM的有效性，并实现了跨多个指标的最佳性能。尤其值得注意的是，它将Fréchet动作距离（FGD）从18.70降低到2.48，运动真实性提高了86.7%。我们的项目页面在此网址：[此URL]。
URL
https://arxiv.org/abs/2503.21847
PDF
-
LightSNN: Lightweight Architecture Search for Sparse and Accurate Spiking Neural Networks
2025-03-27 16:38:13Yesmine Abdennadher, Giovanni Perin, Riccardo Mazzieri, Jacopo Pegoraro, Michele RossiAbstract
Spiking Neural Networks (SNNs) are highly regarded for their energy efficiency, inherent activation sparsity, and suitability for real-time processing in edge devices. However, most current SNN methods adopt architectures resembling traditional artificial neural networks (ANNs), leading to suboptimal performance when applied to SNNs. While SNNs excel in energy efficiency, they have been associated with lower accuracy levels than traditional ANNs when utilizing conventional architectures. In response, in this work we present LightSNN, a rapid and efficient Neural Network Architecture Search (NAS) technique specifically tailored for SNNs that autonomously leverages the most suitable architecture, striking a good balance between accuracy and efficiency by enforcing sparsity. Based on the spiking NAS network (SNASNet) framework, a cell-based search space including backward connections is utilized to build our training-free pruning-based NAS mechanism. Our technique assesses diverse spike activation patterns across different data samples using a sparsity-aware Hamming distance fitness evaluation. Thorough experiments are conducted on both static (CIFAR10 and CIFAR100) and neuromorphic datasets (DVS128-Gesture). Our LightSNN model achieves state-of-the-art results on CIFAR10 and CIFAR100, improves performance on DVS128Gesture by 4.49%, and significantly reduces search time, most notably offering a 98x speedup over SNASNet and running 30% faster than the best existing method on DVS128Gesture.
Abstract (translated)
脉冲神经网络（SNNs）因其高能效、固有的激活稀疏性和边缘设备实时处理的适用性而备受推崇。然而，目前大多数SNN方法采用类似于传统人工神经网络（ANNs）的架构，在应用于SNN时导致性能不佳。尽管SNN在能耗方面表现出色，但在使用常规架构的情况下，其准确性往往低于传统的ANNS。为解决这一问题，我们在此提出LightSNN，这是一种专门针对SNN设计的快速高效的神经网络架构搜索（NAS）技术，能够自主选择最合适的架构，在保持高准确性的基础上提高效率，并通过强制稀疏性来实现最佳平衡。基于脉冲NAS网络（SNASNet）框架，我们在包括反向连接的单元搜索空间上构建了一个无需训练的剪枝基NAS机制。我们的方法使用稀疏感知汉明距离评估不同数据样本中的多种脉冲激活模式。我们在静态（CIFAR10和CIFAR100）和神经形态数据集（DVS128-Gesture）上进行了详尽的实验。我们的LightSNN模型在CIFAR10和CIFAR100数据集上取得了最佳结果，在DVS128Gesture数据集上的性能提升了4.49%，并且显著减少了搜索时间，相比SNASNet速度提升高达98倍，并且在DVS128Gesture上比现有最优方法快30%。
URL
https://arxiv.org/abs/2503.21846
PDF
-
Audio-driven Gesture Generation via Deviation Feature in the Latent Space
2025-03-27 15:37:16Jiahui Chen, Yang Huan, Runhua Shi, Chanfan Ding, Xiaoqi Mo, Siyu Xiong, Yinong HeAbstract
Gestures are essential for enhancing co-speech communication, offering visual emphasis and complementing verbal interactions. While prior work has concentrated on point-level motion or fully supervised data-driven methods, we focus on co-speech gestures, advocating for weakly supervised learning and pixel-level motion deviations. We introduce a weakly supervised framework that learns latent representation deviations, tailored for co-speech gesture video generation. Our approach employs a diffusion model to integrate latent motion features, enabling more precise and nuanced gesture representation. By leveraging weakly supervised deviations in latent space, we effectively generate hand gestures and mouth movements, crucial for realistic video production. Experiments show our method significantly improves video quality, surpassing current state-of-the-art techniques.
Abstract (translated)
手势对于增强伴随言语的交流至关重要，它提供了视觉上的强调，并补充了口头互动。尽管之前的工作主要集中在点级别的运动或完全监督的数据驱动方法上，但我们专注于伴随言语的手势，并倡导使用弱监督学习和像素级别运动偏差的方法。我们介绍了一种弱监督框架，该框架能够学习潜在表示偏差，专门用于生成伴随言语手势的视频。 我们的方法采用扩散模型来整合潜在的运动特征，从而实现更精确、细腻的手势表现。通过在潜在空间中利用弱监督偏差，我们有效生成了手部动作和嘴部运动，这对于真实的视频制作至关重要。 实验结果表明，与当前最先进的技术相比，我们的方法显著提升了视频的质量。
URL
https://arxiv.org/abs/2503.21616
PDF
-
ChatAnyone: Stylized Real-time Portrait Video Generation with Hierarchical Motion Diffusion Model
2025-03-27 04:18:53Jinwei Qi, Chaonan Ji, Sheng Xu, Peng Zhang, Bang Zhang, Liefeng BoAbstract
Real-time interactive video-chat portraits have been increasingly recognized as the future trend, particularly due to the remarkable progress made in text and voice chat technologies. However, existing methods primarily focus on real-time generation of head movements, but struggle to produce synchronized body motions that match these head actions. Additionally, achieving fine-grained control over the speaking style and nuances of facial expressions remains a challenge. To address these limitations, we introduce a novel framework for stylized real-time portrait video generation, enabling expressive and flexible video chat that extends from talking head to upper-body interaction. Our approach consists of the following two stages. The first stage involves efficient hierarchical motion diffusion models, that take both explicit and implicit motion representations into account based on audio inputs, which can generate a diverse range of facial expressions with stylistic control and synchronization between head and body movements. The second stage aims to generate portrait video featuring upper-body movements, including hand gestures. We inject explicit hand control signals into the generator to produce more detailed hand movements, and further perform face refinement to enhance the overall realism and expressiveness of the portrait video. Additionally, our approach supports efficient and continuous generation of upper-body portrait video in maximum 512 * 768 resolution at up to 30fps on 4090 GPU, supporting interactive video-chat in real-time. Experimental results demonstrate the capability of our approach to produce portrait videos with rich expressiveness and natural upper-body movements.
Abstract (translated)
实时互动视频聊天肖像已经被视为未来的发展趋势，这主要是由于文字和语音聊天技术取得了显著的进步。然而，现有的方法主要集中在头部动作的实时生成上，但难以产生与这些头部动作相匹配的身体运动同步。此外，在说话风格和面部表情细微之处实现精细控制仍然是一项挑战。为了克服这些限制，我们提出了一种新的框架，用于生成具有风格化的实时肖像视频，该框架扩展了从脸部互动到上半身互动的表达性和灵活性。 我们的方法包括以下两个阶段： **第一阶段：** 这个阶段涉及高效的分层运动扩散模型，考虑到基于音频输入的显式和隐式的运动表示。这些模型可以产生一系列具有风格化控制并且头部与身体动作同步的面部表情变化。 **第二阶段：** 该阶段的目标是生成包含上半身动作（包括手势）的肖像视频。我们通过向生成器注入明确的手部控制信号来产生更详细的手部动作，并进一步进行脸部优化以增强整个视频的真实感和表现力。此外，我们的方法支持在最大512 * 768分辨率下以每秒30帧的速度实时生成上半身肖像视频，在4090 GPU上的性能表现出色。 实验结果表明，该方法能够生成具有丰富表情和自然上半身动作的肖像视频。
URL
https://arxiv.org/abs/2503.21144
PDF
-
One Snapshot is All You Need: A Generalized Method for mmWave Signal Generation
2025-03-27 03:24:10Teng Huang, Han Ding, Wenxin Sun, Cui Zhao, Ge Wang, Fei Wang, Kun Zhao, Zhi Wang, Wei XiAbstract
Wireless sensing systems, particularly those using mmWave technology, offer distinct advantages over traditional vision-based approaches, such as enhanced privacy and effectiveness in poor lighting conditions. These systems, leveraging FMCW signals, have shown success in human-centric applications like localization, gesture recognition, and so on. However, comprehensive mmWave datasets for diverse applications are scarce, often constrained by pre-processed signatures (e.g., point clouds or RA heatmaps) and inconsistent annotation formats. To overcome these limitations, we propose mmGen, a novel and generalized framework tailored for full-scene mmWave signal generation. By constructing physical signal transmission models, mmGen synthesizes human-reflected and environment-reflected mmWave signals from the constructed 3D meshes. Additionally, we incorporate methods to account for material properties, antenna gains, and multipath reflections, enhancing the realism of the synthesized signals. We conduct extensive experiments using a prototype system with commercial mmWave devices and Kinect sensors. The results show that the average similarity of Range-Angle and micro-Doppler signatures between the synthesized and real-captured signals across three different environments exceeds 0.91 and 0.89, respectively, demonstrating the effectiveness and practical applicability of mmGen.
Abstract (translated)
无线传感系统，尤其是采用毫米波（mmWave）技术的系统，在隐私保护和低光照条件下表现优异，相较于传统的视觉方法具有明显的优势。这些系统利用调频连续波（FMCW）信号，在人体定位、手势识别等人机交互应用中表现出色。然而，针对多样应用场景的全面毫米波数据集依然稀缺，受限于预处理签名（如点云或反射幅度热图）及不一致的注释格式。为克服这些限制，我们提出了mmGen这一新颖且通用化的框架，专用于全场景毫米波信号生成。 通过构建物理信号传输模型，mmGen能够从构造出的3D网格中合成人体和环境反射的毫米波信号。此外，我们引入了方法来考虑材料属性、天线增益及多径反射的影响，以增强所生成信号的真实感。在原型系统上进行了广泛的实验，该系统结合了商用毫米波设备与Kinect传感器。结果显示，在三种不同环境中合成信号和实际捕获信号的范围-角度（Range-Angle）和微多普勒签名之间的平均相似度分别超过0.91和0.89，这证明了mmGen的有效性和实用价值。 这项研究为未来的毫米波技术开发奠定了坚实的基础，并展示了其在各种环境条件下的广泛应用潜力。
URL
https://arxiv.org/abs/2503.21122
PDF
-
Siformer: Feature-isolated Transformer for Efficient Skeleton-based Sign Language Recognition
2025-03-26 11:10:29Muxin Pu, Mei Kuan Lim, Chun Yong ChongAbstract
Sign language recognition (SLR) refers to interpreting sign language glosses from given videos automatically. This research area presents a complex challenge in computer vision because of the rapid and intricate movements inherent in sign languages, which encompass hand gestures, body postures, and even facial expressions. Recently, skeleton-based action recognition has attracted increasing attention due to its ability to handle variations in subjects and backgrounds independently. However, current skeleton-based SLR methods exhibit three limitations: 1) they often neglect the importance of realistic hand poses, where most studies train SLR models on non-realistic skeletal representations; 2) they tend to assume complete data availability in both training or inference phases, and capture intricate relationships among different body parts collectively; 3) these methods treat all sign glosses uniformly, failing to account for differences in complexity levels regarding skeletal representations. To enhance the realism of hand skeletal representations, we present a kinematic hand pose rectification method for enforcing constraints. Mitigating the impact of missing data, we propose a feature-isolated mechanism to focus on capturing local spatial-temporal context. This method captures the context concurrently and independently from individual features, thus enhancing the robustness of the SLR model. Additionally, to adapt to varying complexity levels of sign glosses, we develop an input-adaptive inference approach to optimise computational efficiency and accuracy. Experimental results demonstrate the effectiveness of our approach, as evidenced by achieving a new state-of-the-art (SOTA) performance on WLASL100 and LSA64. For WLASL100, we achieve a top-1 accuracy of 86.50\%, marking a relative improvement of 2.39% over the previous SOTA. For LSA64, we achieve a top-1 accuracy of 99.84%.
Abstract (translated)
手语识别（SLR）是指自动解析给定视频中的手势语言简写。这一研究领域在计算机视觉中构成了一个复杂挑战，原因在于手势语言包含了快速且复杂的动作，涉及手部姿势、身体姿态及面部表情等多方面因素。近年来，基于骨架的动作识别因其能够独立处理主体和背景的变化而吸引了越来越多的关注。然而，目前的基于骨架的手语识别方法存在三个局限性：1）它们通常忽视了真实手部姿势的重要性，在大多数研究中训练SLR模型使用的是非真实的骨骼表示；2）这些方法倾向于假设在训练或推断阶段数据是完整的，并且能够集体捕捉不同身体部位之间的复杂关系；3）这些方法对待所有的手势简写都采用统一处理的方式，未能考虑到关于骨架表示的复杂程度差异。 为了增强手部骨骼表示的真实性，我们提出了一种动力学手部姿势矫正方法，用于施加约束。为减轻数据缺失的影响，我们提出了一个特征隔离机制来专注于捕捉局部时空上下文，这种方法能够同时独立地从个别特征中捕获上下文信息，从而增强了SLR模型的鲁棒性。此外，为了适应手势简写复杂程度的不同，我们开发了一种输入自适应推断方法以优化计算效率和准确性。 实验结果表明了我们方法的有效性，具体表现为在WLASL100和LSA64数据集上实现了新的最先进（SOTA）性能表现。对于WLASL100数据集，我们的模型达到了86.50%的Top-1准确率，相较于之前的最先进水平提高了2.39%；而对于LSA64数据集，我们则实现了高达99.84%的Top-1准确率。
URL
https://arxiv.org/abs/2503.20436
PDF
-
Video Motion Graphs
2025-03-26 04:20:14Haiyang Liu, Zhan Xu, Fa-Ting Hong, Hsin-Ping Huang, Yi Zhou, Yang ZhouAbstract
We present Video Motion Graphs, a system designed to generate realistic human motion videos. Using a reference video and conditional signals such as music or motion tags, the system synthesizes new videos by first retrieving video clips with gestures matching the conditions and then generating interpolation frames to seamlessly connect clip boundaries. The core of our approach is HMInterp, a robust Video Frame Interpolation (VFI) model that enables seamless interpolation of discontinuous frames, even for complex motion scenarios like dancing. HMInterp i) employs a dual-branch interpolation approach, combining a Motion Diffusion Model for human skeleton motion interpolation with a diffusion-based video frame interpolation model for final frame generation. ii) adopts condition progressive training to effectively leverage identity strong and weak conditions, such as images and pose. These designs ensure both high video texture quality and accurate motion trajectory. Results show that our Video Motion Graphs outperforms existing generative- and retrieval-based methods for multi-modal conditioned human motion video generation. Project page can be found at this https URL
Abstract (translated)
我们介绍了Video Motion Graphs（VMG）系统，这是一个旨在生成逼真的人体动作视频的创新解决方案。通过使用参考视频以及音乐或动作标签等条件信号，该系统能够合成新的视频：首先检索与给定条件匹配的动作片段，然后生成插值帧以无缝连接这些片段之间的边界。我们方法的核心是HMInterp，这是一种强大的视频帧插值（VFI）模型，它能够在包括舞蹈在内的复杂运动场景中实现不连续帧的平滑插值。 HMInterp具有以下两个关键特性： i) 采用了双分支插值方法，结合了用于人体骨骼动作插值的Motion Diffusion Model和基于扩散的方法进行最终帧生成。 ii) 采用逐步条件训练策略，有效利用身份强弱条件（如图像和姿态），以确保视频纹理质量和准确的动作轨迹。 实验结果表明，在多模态条件下的人体运动视频生成方面，我们的Video Motion Graphs系统优于现有的生成式及检索式的解决方案。有关项目的更多信息可以在以下链接找到：[项目页面](https://this https URL) （请将URL中的"this https URL"替换为实际的项目网址）。
URL
https://arxiv.org/abs/2503.20218
PDF
-
SARGes: Semantically Aligned Reliable Gesture Generation via Intent Chain
2025-03-26 03:55:41Nan Gao, Yihua Bao, Dongdong Weng, Jiayi Zhao, Jia Li, Yan Zhou, Pengfei Wan, Di ZhangAbstract
Co-speech gesture generation enhances human-computer interaction realism through speech-synchronized gesture synthesis. However, generating semantically meaningful gestures remains a challenging problem. We propose SARGes, a novel framework that leverages large language models (LLMs) to parse speech content and generate reliable semantic gesture labels, which subsequently guide the synthesis of meaningful co-speech this http URL, we constructed a comprehensive co-speech gesture ethogram and developed an LLM-based intent chain reasoning mechanism that systematically parses and decomposes gesture semantics into structured inference steps following ethogram criteria, effectively guiding LLMs to generate context-aware gesture labels. Subsequently, we constructed an intent chain-annotated text-to-gesture label dataset and trained a lightweight gesture label generation model, which then guides the generation of credible and semantically coherent co-speech gestures. Experimental results demonstrate that SARGes achieves highly semantically-aligned gesture labeling (50.2% accuracy) with efficient single-pass inference (0.4 seconds). The proposed method provides an interpretable intent reasoning pathway for semantic gesture synthesis.
Abstract (translated)
协同语音手势生成通过同步语音的手势合成增强了人机交互的逼真度，但生成语义有意义的手势仍然是一个具有挑战性的问题。我们提出了SARGes（Speech-Associated Reliable Gesture Synthesis）这一新颖框架，该框架利用大型语言模型（LLMs）解析语音内容，并生成可靠的语义手势标签，随后指导生成意义明确的协同语音手势。 具体而言，我们构建了一个全面的手势图谱（co-speech gesture ethogram），并开发了一种基于LLM的意图链推理机制。这一机制系统地按照手势图谱的标准解析和拆分手势语义为结构化的推理步骤，有效引导LLMs生成情境感知的手势标签。随后，我们建立了一个带有意图链标注的文本到手势标签的数据集，并训练了轻量级的手势标签生成模型，该模型之后指导生成可信且语义连贯的协同语音手势。 实验结果表明，SARGes在手势标记上的语义对齐度达到了50.2%的准确率，并实现了高效的单次推理（0.4秒）。所提出的方法提供了一条可解释的意图推理路径，用于语义手势合成。
URL
https://arxiv.org/abs/2503.20202
PDF
-
Towards Online Multi-Modal Social Interaction Understanding
2025-03-25 17:17:19Xinpeng Li, Shijian Deng, Bolin Lai, Weiguo Pian, James M. Rehg, Yapeng TianAbstract
Multimodal social interaction understanding (MMSI) is critical in human-robot interaction systems. In real-world scenarios, AI agents are required to provide real-time feedback. However, existing models often depend on both past and future contexts, which hinders them from applying to real-world problems. To bridge this gap, we propose an online MMSI setting, where the model must resolve MMSI tasks using only historical information, such as recorded dialogues and video streams. To address the challenges of missing the useful future context, we develop a novel framework, named Online-MMSI-VLM, that leverages two complementary strategies: multi-party conversation forecasting and social-aware visual prompting with multi-modal large language models. First, to enrich linguistic context, the multi-party conversation forecasting simulates potential future utterances in a coarse-to-fine manner, anticipating upcoming speaker turns and then generating fine-grained conversational details. Second, to effectively incorporate visual social cues like gaze and gesture, social-aware visual prompting highlights the social dynamics in video with bounding boxes and body keypoints for each person and frame. Extensive experiments on three tasks and two datasets demonstrate that our method achieves state-of-the-art performance and significantly outperforms baseline models, indicating its effectiveness on Online-MMSI. The code and pre-trained models will be publicly released at: this https URL.
Abstract (translated)
多模态社会交互理解（MMSI）在人机交互系统中至关重要。在现实场景下，AI代理需要提供实时反馈。然而，现有模型往往依赖于过去的和将来的上下文信息，这阻碍了它们应用于实际问题的能力。为了弥合这一差距，我们提出了一种在线多模态社会交互理解（Online-MMSI）设置，在这种设置中，模型必须仅利用历史信息（如记录的对话和视频流）来解决MMSI任务。 为了解决缺乏有用未来上下文的问题，我们开发了一个新颖的框架，命名为Online-MMSI-VLM，该框架采用了两种互补策略：多党会议预测和基于多人模态大型语言模型的社会感知视觉提示。首先，为了丰富语言背景，多党会议预测以从粗到细的方式模拟潜在的未来的言语行为，在预测即将到来的说话者回合后生成详细的对话细节。其次，为有效整合诸如凝视和手势等视觉社会线索，社会感知视觉提示通过为人像中的每一帧标注边界框和身体关键点来突出视频中的社交动态。 在三个任务和两个数据集上的广泛实验表明，我们的方法达到了最先进的性能，并显著优于基线模型，这证明了它在线MMSI的有效性。代码及预训练模型将在以下网址公开发布：[此链接](this https URL)。
URL
https://arxiv.org/abs/2503.19851
PDF
-
AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers
2025-03-25 16:38:23Jiazhi Guan, Kaisiyuan Wang, Zhiliang Xu, Quanwei Yang, Yasheng Sun, Shengyi He, Borong Liang, Yukang Cao, Yingying Li, Haocheng Feng, Errui Ding, Jingdong Wang, Youjian Zhao, Hang Zhou, Ziwei LiuAbstract
Despite the recent progress of audio-driven video generation, existing methods mostly focus on driving facial movements, leading to non-coherent head and body dynamics. Moving forward, it is desirable yet challenging to generate holistic human videos with both accurate lip-sync and delicate co-speech gestures w.r.t. given audio. In this work, we propose AudCast, a generalized audio-driven human video generation framework adopting a cascade Diffusion-Transformers (DiTs) paradigm, which synthesizes holistic human videos based on a reference image and a given audio. 1) Firstly, an audio-conditioned Holistic Human DiT architecture is proposed to directly drive the movements of any human body with vivid gesture dynamics. 2) Then to enhance hand and face details that are well-knownly difficult to handle, a Regional Refinement DiT leverages regional 3D fitting as the bridge to reform the signals, producing the final results. Extensive experiments demonstrate that our framework generates high-fidelity audio-driven holistic human videos with temporal coherence and fine facial and hand details. Resources can be found at this https URL.
Abstract (translated)
尽管最近在音频驱动的视频生成方面取得了进展，现有的方法大多集中在驱动面部动作上，导致头部和身体动态不一致。向前发展，根据给定的音频同时生成准确的唇部同步和细腻的伴随手势的完整人体视频是令人向往但具有挑战性的目标。在这项工作中，我们提出了AudCast，这是一个通用的音频驱动的人体视频生成框架，采用级联扩散-变压器（DiTs）范式，在参考图像和给定音频的基础上合成完整的身体动作。 1) 首先，提出了一种基于音频条件的整体人体DiT架构，可以直接控制任何人类的身体动作，并产生生动的手势动态。 2) 为了增强手部和面部的细节处理，众所周知这些是难以处理的部分，区域细化DiT利用局部3D拟合作为信号重制的桥梁，生成最终的结果。 广泛的实验表明，我们的框架能够生成高保真度的音频驱动的整体人体视频，并具有时间一致性以及细致的手部和面部特征。更多资源可以在这里找到：[此URL]（请将“this https URL”替换为实际链接）。
URL
https://arxiv.org/abs/2503.19824
PDF
-
A multitask transformer to sign language translation using motion gesture primitives
2025-03-25 13:53:25Fredy Alejandro Mendoza L\'opez, Jefferson Rodriguez, Fabio Mart\'inezAbstract
The absence of effective communication the deaf population represents the main social gap in this community. Furthermore, the sign language, main deaf communication tool, is unlettered, i.e., there is no formal written representation. In consequence, main challenge today is the automatic translation among spatiotemporal sign representation and natural text language. Recent approaches are based on encoder-decoder architectures, where the most relevant strategies integrate attention modules to enhance non-linear correspondences, besides, many of these approximations require complex training and architectural schemes to achieve reasonable predictions, because of the absence of intermediate text projections. However, they are still limited by the redundant background information of the video sequences. This work introduces a multitask transformer architecture that includes a gloss learning representation to achieve a more suitable translation. The proposed approach also includes a dense motion representation that enhances gestures and includes kinematic information, a key component in sign language. From this representation it is possible to avoid background information and exploit the geometry of the signs, in addition, it includes spatiotemporal representations that facilitate the alignment between gestures and glosses as an intermediate textual representation. The proposed approach outperforms the state-of-the-art evaluated on the CoL-SLTD dataset, achieving a BLEU-4 of 72,64% in split 1, and a BLEU-4 of 14,64% in split 2. Additionally, the strategy was validated on the RWTH-PHOENIX-Weather 2014 T dataset, achieving a competitive BLEU-4 of 11,58%.
Abstract (translated)
聋人群体中有效沟通的缺乏代表了这一社区的主要社会差距。此外，手语作为主要的交流工具是没有文字形式的，即没有正式的文字表示方式。因此，当前的主要挑战在于在空间时间手势表示与自然语言文本之间实现自动翻译。 最近的方法基于编码器-解码器架构，其中最相关的策略是集成注意力模块来增强非线性对应关系。然而，由于缺乏中间文本投影，许多这些方法需要复杂的训练和体系结构方案才能获得合理的预测结果。尽管如此，它们仍然受到视频序列中冗余背景信息的限制。 这项工作介绍了一种多任务变压器架构，该架构包括手势词学习表示以实现更合适的翻译。所提出的方法还包括密集运动表示，增强了手势，并包含了关键点动力学信息，在手语中这是一个关键组成部分。从这种表示法中可以避免背景信息并利用手势的几何形状，此外它还包含空间时间表示形式，有助于在手势和手势词之间进行对齐作为中间文本表示。 所提出的方法在CoL-SLTD数据集上进行了评估，并超越了现有技术，在split 1中实现了72.64%的BLEU-4评分，在split 2中实现了14.64%的BLEU-4评分。此外，该策略还在RWTH-PHOENIX-Weather 2014 T数据集上进行了验证，并取得了具有竞争力的11.58%的BLEU-4得分。
URL
https://arxiv.org/abs/2503.19668
PDF
-
A-MESS: Anchor based Multimodal Embedding with Semantic Synchronization for Multimodal Intent Recognition
2025-03-25 09:09:30Yaomin Shen, Xiaojian Lin, Wei FanAbstract
In the domain of multimodal intent recognition (MIR), the objective is to recognize human intent by integrating a variety of modalities, such as language text, body gestures, and tones. However, existing approaches face difficulties adequately capturing the intrinsic connections between the modalities and overlooking the corresponding semantic representations of intent. To address these limitations, we present the Anchor-based Mul- timodal Embedding with Semantic Synchronization (A-MESS) framework. We first design an Anchor-based Multimodal Embed- ding (A-ME) module that employs an anchor-based embedding fusion mechanism to integrate multimodal inputs. Furthermore, we develop a Semantic Synchronization (SS) strategy with the Triplet Contrastive Learning pipeline, which optimizes the pro- cess by synchronizing multimodal representation with label de- scriptions produced by the large language model. Comprehensive experiments indicate that our A-MESS achieves state-of-the-art and provides substantial insight into multimodal representation and downstream tasks.
Abstract (translated)
在多模态意图识别（MIR）领域，目标是通过整合各种模态（如语言文本、肢体动作和语调）来识别人类的意图。然而，现有方法难以充分捕捉这些模态之间的内在联系，并且忽视了相应的语义表示。为了解决这些问题，我们提出了带有语义同步的锚点基多模态嵌入框架（Anchor-based Multimodal Embedding with Semantic Synchronization, A-MESS）。首先，我们设计了一个基于锚点的多模态嵌入模块（Anchor-based Multimodal Embedding, A-ME），该模块采用了一种基于锚点的嵌入融合机制来整合多模态输入。此外，我们还开发了语义同步策略，并结合三元组对比学习管道，通过将多模态表示与大型语言模型生成的标签描述进行同步优化，进一步改进过程。全面的实验表明，我们的A-MESS框架达到了最先进的水平，并为多模态表征和下游任务提供了重要的见解。
URL
https://arxiv.org/abs/2503.19474
PDF
-
Beyond Object Categories: Multi-Attribute Reference Understanding for Visual Grounding
2025-03-25 00:59:58Hao Guo, Jianfei Zhu, Wei Fan, Chunzhi Yi, Feng JiangAbstract
Referring expression comprehension (REC) aims at achieving object localization based on natural language descriptions. However, existing REC approaches are constrained by object category descriptions and single-attribute intention descriptions, hindering their application in real-world scenarios. In natural human-robot interactions, users often express their desires through individual states and intentions, accompanied by guiding gestures, rather than detailed object descriptions. To address this challenge, we propose Multi-ref EC, a novel task framework that integrates state descriptions, derived intentions, and embodied gestures to locate target objects. We introduce the State-Intention-Gesture Attributes Reference (SIGAR) dataset, which combines state and intention expressions with embodied references. Through extensive experiments with various baseline models on SIGAR, we demonstrate that properly ordered multi-attribute references contribute to improved localization performance, revealing that single-attribute reference is insufficient for natural human-robot interaction scenarios. Our findings underscore the importance of multi-attribute reference expressions in advancing visual-language understanding.
Abstract (translated)
参考表达理解（REC）旨在根据自然语言描述实现对象定位。然而，现有的REC方法受到物体类别描述和单一属性意图描述的限制，这阻碍了它们在现实场景中的应用。在自然的人机交互中，用户通常通过个体状态和意图来表达他们的需求，并伴有引导手势，而不是详细的物体描述。为了解决这一挑战，我们提出了一种新的任务框架——Multi-ref EC，该框架将状态描述、推导出的意图以及身体动作整合起来以定位目标对象。我们引入了State-Intention-Gesture Attributes Reference（SIGAR）数据集，结合了状态和意图表达与身体引用。通过在SIGAR上进行广泛的基线模型实验，我们证明了适当排序的多属性参考有助于提高定位性能，并揭示了单一属性参考对于自然人机交互场景是不足的。我们的研究结果强调了多属性参考表达在推进视觉-语言理解中的重要性。
URL
https://arxiv.org/abs/2503.19240
PDF
-
Smart Ankleband for Plug-and-Play Hand-Prosthetic Control
2025-03-22 19:45:12Dean Zadok, Oren Salzman, Alon Wolf, Alex M. BronsteinAbstract
Building robotic prostheses requires the creation of a sensor-based interface designed to provide the robotic hand with the control required to perform hand gestures. Traditional Electromyography (EMG) based prosthetics and emerging alternatives often face limitations such as muscle-activation limitations, high cost, and complex-calibration procedures. In this paper, we present a low-cost robotic system composed of a smart ankleband for intuitive, calibration-free control of a robotic hand, and a robotic prosthetic hand that executes actions corresponding to leg gestures. The ankleband integrates an Inertial Measurement Unit (IMU) sensor with a lightweight temporal neural network to infer user-intended leg gestures from motion data. Our system represents a significant step towards higher adoption rates of robotic prostheses among arm amputees, as it enables one to operate a prosthetic hand using a low-cost, low-power, and calibration-free solution. To evaluate our work, we collected data from 10 subjects and tested our prototype ankleband with a robotic hand on an individual with upper-limb amputations. Our results demonstrate that this system empowers users to perform daily tasks more efficiently, requiring few compensatory movements.
Abstract (translated)
构建机器人假肢需要设计一个基于传感器的接口，以便为机器手提供执行手势所需的控制。传统的肌电图（EMG）假肢和新兴替代方案常常面临肌肉激活限制、成本高昂以及复杂校准程序等问题。在这篇论文中，我们提出了一种低成本的机器人系统，该系统包括一个智能踝带，用于直观且无需校准地控制机器手，并配有一个执行与腿部动作对应的机器假手。此踝带集成了惯性测量单元（IMU）传感器和轻量级时间神经网络，可以从运动数据中推断出用户意图的腿部动作。我们的系统代表了在手臂截肢者中间增加机器人假肢使用率的一个重要步骤，因为它使人们能够通过低成本、低功耗且无需校准的方法来操作假手。 为了评估我们的研究成果，我们从10名受试者那里收集了数据，并对配备有原型踝带的机器手进行了一位上肢截肢者的测试。结果表明，该系统能够让用户更高效地完成日常任务，减少补偿性动作的需求。
URL
https://arxiv.org/abs/2503.17846
PDF
-
DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speech
2025-03-21 11:23:39Yongkang Cheng, Shaoli Huang, Xuelin Chen, Jifeng Ning, Mingming GongAbstract
Diffusion models have demonstrated remarkable synthesis quality and diversity in generating co-speech gestures. However, the computationally intensive sampling steps associated with diffusion models hinder their practicality in real-world applications. Hence, we present DIDiffGes, for a Decoupled Semi-Implicit Diffusion model-based framework, that can synthesize high-quality, expressive gestures from speech using only a few sampling steps. Our approach leverages Generative Adversarial Networks (GANs) to enable large-step sampling for diffusion model. We decouple gesture data into body and hands distributions and further decompose them into marginal and conditional distributions. GANs model the marginal distribution implicitly, while L2 reconstruction loss learns the conditional distributions exciplictly. This strategy enhances GAN training stability and ensures expressiveness of generated full-body gestures. Our framework also learns to denoise root noise conditioned on local body representation, guaranteeing stability and realism. DIDiffGes can generate gestures from speech with just 10 sampling steps, without compromising quality and expressiveness, reducing the number of sampling steps by a factor of 100 compared to existing methods. Our user study reveals that our method outperforms state-of-the-art approaches in human likeness, appropriateness, and style correctness. Project is this https URL.
Abstract (translated)
扩散模型在生成与语音同步的手势时展示了卓越的合成质量和多样性。然而，与扩散模型相关的计算密集型采样步骤阻碍了它们在现实世界应用中的实用性。为此，我们提出了 DIDiffGes——一个基于解耦半隐式扩散模型框架的方法，能够仅通过少量采样步骤就从语音中生成高质量且富有表现力的手势。我们的方法利用生成对抗网络（GANs）来实现大步长采样的扩散模型。我们将手势数据分解为主体和手部分布，并进一步将其拆分为边缘分布和条件分布。GAN 模型隐式地模拟边缘分布，而 L2 重构损失则显式学习条件分布。这一策略增强了 GAN 训练的稳定性并确保生成的整体身体手势具有表现力。我们的框架还学会了根据局部身体表示对根噪声进行去噪，以保证稳定性和现实性。DIDiffGes 能够仅用10个采样步骤就从语音中生成手势，并且不会牺牲质量和表现力，相比现有方法将采样步骤减少了100倍。我们的用户研究表明，我们提出的方法在人类相似度、适当性和风格正确性方面超过了现有的先进方法。 项目详情请参见：[此链接](https://this.is.the.url.of.project)（这里需要替换为实际的项目网址）。
URL
https://arxiv.org/abs/2503.17059
PDF
