Gesture
-
Grounded Gesture Generation: Language, Motion, and Space
2025-07-06 20:19:34Anna Deichler, Jim O'Regan, Teo Guichoux, David Johansson, Jonas BeskowAbstract
Human motion generation has advanced rapidly in recent years, yet the critical problem of creating spatially grounded, context-aware gestures has been largely overlooked. Existing models typically specialize either in descriptive motion generation, such as locomotion and object interaction, or in isolated co-speech gesture synthesis aligned with utterance semantics. However, both lines of work often treat motion and environmental grounding separately, limiting advances toward embodied, communicative agents. To address this gap, our work introduces a multimodal dataset and framework for grounded gesture generation, combining two key resources: (1) a synthetic dataset of spatially grounded referential gestures, and (2) MM-Conv, a VR-based dataset capturing two-party dialogues. Together, they provide over 7.7 hours of synchronized motion, speech, and 3D scene information, standardized in the HumanML3D format. Our framework further connects to a physics-based simulator, enabling synthetic data generation and situated evaluation. By bridging gesture modeling and spatial grounding, our contribution establishes a foundation for advancing research in situated gesture generation and grounded multimodal interaction. Project page: this https URL
Abstract (translated)
近年来，人类动作生成技术迅速发展，然而创建空间定位且具有情境感知能力的手势问题却一直被忽视。现有的模型通常专注于描述性运动生成（如行走和物体互动）或与话语语义相匹配的孤立共言语手势合成。但是，这两种方法往往将运动和环境联系分开处理，从而限制了在实体交互代理方面的进展。为了解决这一差距，我们的工作引入了一个多模态数据集和框架，用于生成基于情境的手势，结合两个关键资源：（1）一个包含空间定位指代手势的合成数据集；（2）MM-Conv，这是一个基于VR的数据集，捕捉了双方对话的情景。这两个资源一起提供了超过7.7小时同步的动作、语音以及3D场景信息，并且这些数据被标准化为HumanML3D格式。我们的框架进一步与物理仿真器连接起来，使合成数据生成和情境评估成为可能。通过将手势建模与空间定位相结合，我们建立了研究基于情景的手势生成及多模态交互的基础。 项目页面：[此网址](this https URL)
URL
https://arxiv.org/abs/2507.04522
PDF
-
Visual Hand Gesture Recognition with Deep Learning: A Comprehensive Review of Methods, Datasets, Challenges and Future Research Directions
2025-07-06 17:03:01Konstantinos Foteinos, Jorgen Cani, Manousos Linardakis, Panagiotis Radoglou-Grammatikis, Vasileios Argyriou, Panagiotis Sarigiannidis, Iraklis Varlamis, Georgios Th. PapadopoulosAbstract
The rapid evolution of deep learning (DL) models and the ever-increasing size of available datasets have raised the interest of the research community in the always important field of vision-based hand gesture recognition (VHGR), and delivered a wide range of applications, such as sign language understanding and human-computer interaction using cameras. Despite the large volume of research works in the field, a structured and complete survey on VHGR is still missing, leaving researchers to navigate through hundreds of papers in order to find the right combination of data, model, and approach for each task. The current survey aims to fill this gap by presenting a comprehensive overview of this aspect of computer vision. With a systematic research methodology that identifies the state-of-the-art works and a structured presentation of the various methods, datasets, and evaluation metrics, this review aims to constitute a useful guideline for researchers, helping them to choose the right strategy for delving into a certain VHGR task. Starting with the methodology used for study selection, literature retrieval, and the analytical framing, the survey identifies and organizes key VHGR approaches using a taxonomy-based format in various dimensions such as input modality and application domain. The core of the survey provides an in-depth analysis of state-of-the-art techniques across three primary VHGR tasks: static gesture recognition, isolated dynamic gestures and continuous gesture recognition. For each task, the architectural trends and learning strategies are listed. Additionally, the study reviews commonly used datasets - emphasizing on annotation schemes - and evaluates standard performance metrics. It concludes by identifying major challenges in VHGR, including both general computer vision issues and domain-specific obstacles, and outlines promising directions for future research.
Abstract (translated)
深度学习（DL）模型的快速演进和可用数据集规模的不断增大，使得基于视觉的手势识别（VHGR）这一长久以来重要的研究领域重新引起了科研社区的兴趣，并且催生了许多应用，例如手语理解和使用摄像头的人机交互。尽管该领域的研究成果众多，但至今仍缺乏系统性和全面性的综述文献，这导致研究人员需要在大量论文中寻找适用于特定任务的数据、模型和方法组合。当前的综述旨在填补这一空白，通过呈现计算机视觉领域中的这一方面进行全面概述。采用系统性研究方法识别最新成果，并以结构化的方式介绍各种方法、数据集及评估指标，该回顾力求成为科研人员的重要指南，帮助他们为探索特定的VHGR任务选择合适的战略。 本次综述从用于研究筛选、文献检索和分析框架的方法论开始，使用基于分类法的形式来识别并组织关键的VHGR方法，在输入模式和应用领域等多个维度进行分类。该综述的核心部分提供了对三个主要的VHGR任务——静态手势识别、孤立动态手势及连续手势识别的最新技术进行了深入分析，并列出了每项任务中的架构趋势和学习策略。此外，本研究还回顾了常用的数据库（特别强调标注方案），并评估标准性能指标。综述最后指出了VHGR领域的重大挑战，包括一般计算机视觉问题以及特定领域障碍，并概述了未来研究的潜在方向。
URL
https://arxiv.org/abs/2507.04465
PDF
-
'Hi AirStar, Guide Me to the Badminton Court.'
2025-07-06 15:29:07Ziqin Wang, Jinyu Chen, Xiangyi Zheng, Qinan Liao, Linjiang Huang, Si LiuAbstract
Unmanned Aerial Vehicles, operating in environments with relatively few obstacles, offer high maneuverability and full three-dimensional mobility. This allows them to rapidly approach objects and perform a wide range of tasks often challenging for ground robots, making them ideal for exploration, inspection, aerial imaging, and everyday assistance. In this paper, we introduce AirStar, a UAV-centric embodied platform that turns a UAV into an intelligent aerial assistant: a large language model acts as the cognitive core for environmental understanding, contextual reasoning, and task planning. AirStar accepts natural interaction through voice commands and gestures, removing the need for a remote controller and significantly broadening its user base. It combines geospatial knowledge-driven long-distance navigation with contextual reasoning for fine-grained short-range control, resulting in an efficient and accurate vision-and-language navigation (VLN) this http URL, the system also offers built-in capabilities such as cross-modal question answering, intelligent filming, and target tracking. With a highly extensible framework, it supports seamless integration of new functionalities, paving the way toward a general-purpose, instruction-driven intelligent UAV agent. The supplementary PPT is available at \href{this https URL}{this https URL}.
Abstract (translated)
无人员驾驶飞行器（无人机）在障碍物相对较少的环境中运行时，具有很高的机动性和全方位三维移动能力。这使得它们能够迅速接近物体并执行地面机器人难以完成的一系列任务，如探索、检查、空中成像和日常辅助工作。本文介绍了AirStar，这是一个以无人机为中心的身体化平台，将无人机转化为智能空中小助手：大型语言模型充当环境理解、情境推理和任务规划的认知核心。AirStar通过语音命令和手势接受自然交互方式，从而免去了遥控器的需要，并大大拓宽了用户基础。它结合了基于地理空间知识的远程导航与情景推理驱动的精细近距离控制，实现了高效且准确的视觉-语言导航（VLN）。此外，该系统还内置了一些功能，如跨模式问答、智能拍摄和目标追踪。凭借高度可扩展的框架，它可以无缝集成新功能，为通用指令驱动型智能无人机代理的发展铺平道路。补充材料PPT可在\href{this https URL}{此链接}获取。
URL
https://arxiv.org/abs/2507.04430
PDF
-
Image-driven Robot Drawing with Rapid Lognormal Movements
2025-07-03 20:51:27Daniel Berio, Guillaume Clivaz, Michael Stroh, Oliver Deussen, R\'ejean Plamondon, Sylvain Calinon, Frederic Fol LeymarieAbstract
Large image generation and vision models, combined with differentiable rendering technologies, have become powerful tools for generating paths that can be drawn or painted by a robot. However, these tools often overlook the intrinsic physicality of the human drawing/writing act, which is usually executed with skillful hand/arm gestures. Taking this into account is important for the visual aesthetics of the results and for the development of closer and more intuitive artist-robot collaboration scenarios. We present a method that bridges this gap by enabling gradient-based optimization of natural human-like motions guided by cost functions defined in image space. To this end, we use the sigma-lognormal model of human hand/arm movements, with an adaptation that enables its use in conjunction with a differentiable vector graphics (DiffVG) renderer. We demonstrate how this pipeline can be used to generate feasible trajectories for a robot by combining image-driven objectives with a minimum-time smoothing criterion. We demonstrate applications with generation and robotic reproduction of synthetic graffiti as well as image abstraction.
Abstract (translated)
大型图像生成和视觉模型，结合可微渲染技术，已经成为生成机器人可以绘制或绘画路径的强大工具。然而，这些工具往往忽略了人类绘图/书写行为内在的物理特性，这种行为通常通过熟练的手臂手势来执行。考虑到这一点对于结果的视觉美学以及更接近、更具直观性的艺术家与机器人的协作场景的发展非常重要。我们提出了一种方法，该方法通过在图像空间中定义成本函数的基础上进行基于梯度的优化，使得自然的人类手/臂动作得以实现，从而弥合了这一差距。为此，我们使用了人类手/臂运动的sigma-对数正态模型，并做了一些适应性修改，使其能够与可微向量图形（DiffVG）渲染器结合使用。我们演示了如何通过将图像驱动的目标与最小时间平滑标准相结合来利用这条管道生成机器人的可行轨迹。我们展示了合成涂鸦的生成和机器人复制以及图像抽象的应用实例。
URL
https://arxiv.org/abs/2507.03166
PDF
-
DeepGesture: A conversational gesture synthesis system based on emotions and semantics
2025-07-03 20:04:04Thanh Hoang-MinhAbstract
Along with the explosion of large language models, improvements in speech synthesis, advancements in hardware, and the evolution of computer graphics, the current bottleneck in creating digital humans lies in generating character movements that correspond naturally to text or speech inputs. In this work, we present DeepGesture, a diffusion-based gesture synthesis framework for generating expressive co-speech gestures conditioned on multimodal signals-text, speech, emotion, and seed motion. Built upon the DiffuseStyleGesture model, DeepGesture introduces novel architectural enhancements that improve semantic alignment and emotional expressiveness in generated gestures. Specifically, we integrate fast text transcriptions as semantic conditioning and implement emotion-guided classifier-free diffusion to support controllable gesture generation across affective states. A lightweight Transformer backbone combines full self-attention and cross-local attention for effective feature fusion of heterogeneous modalities. To visualize results, we implement a full rendering pipeline in Unity based on BVH output from the model. Evaluation on the ZeroEGGS dataset shows that DeepGesture produces gestures with improved human-likeness and contextual appropriateness, outperforming baselines on Mean Opinion Score and Frechet Gesture Distance metrics. Our system supports interpolation between emotional states and demonstrates generalization to out-of-distribution speech, including synthetic voices-marking a step forward toward fully multimodal, emotionally aware digital humans.
Abstract (translated)
随着大型语言模型的爆炸式发展、语音合成技术的进步、硬件设备的提升以及计算机图形学的发展，目前在创建数字人类时遇到的一个瓶颈是如何生成与文本或语音输入自然对应的肢体动作。在此研究中，我们介绍了DeepGesture，这是一种基于扩散模型的手势合成框架，能够根据多模态信号（包括文本、语音、情感和初始运动）生成具有表现力的伴随言语手势。 DeepGesture 是在 DiffuseStyleGesture 模型的基础上构建的，并引入了新颖的架构改进以提高生成的手势在语义对齐和情绪表达方面的性能。具体来说，我们整合了快速文字转录作为语义条件输入，并实现了情感引导的无分类器扩散方法来支持跨情绪状态的情感可控手势生成。一个轻量级的Transformer骨干网络结合了完整的自注意力机制与局部交叉注意力，以有效地融合异构模态特征。 为了展示结果，我们在 Unity 中实现了一个基于模型输出 BVH 文件的完整渲染管线。在 ZeroEGGS 数据集上的评估显示，DeepGesture 能够生成更加逼真且符合语境的手势，在平均意见评分和Frechet手势距离等指标上超越了基准方法。我们的系统支持在情绪状态之间进行插值，并展示了对分布外语音（包括合成声音）的泛化能力——这标志着向完全多模态、情感感知数字人类迈进了一步。
URL
https://arxiv.org/abs/2507.03147
PDF
-
Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocation
2025-06-30 10:45:16Xingting Yao, Qinghao Hu, Fei Zhou, Tielong Liu, Gang Li, Peisong Wang, Jian ChengAbstract
Multi-bit spiking neural networks (SNNs) have recently become a heated research spot, pursuing energy-efficient and high-accurate AI. However, with more bits involved, the associated memory and computation demands escalate to the point where the performance improvements become disproportionate. Based on the insight that different layers demonstrate different importance and extra bits could be wasted and interfering, this paper presents an adaptive bit allocation strategy for direct-trained SNNs, achieving fine-grained layer-wise allocation of memory and computation resources. Thus, SNN's efficiency and accuracy can be improved. Specifically, we parametrize the temporal lengths and the bit widths of weights and spikes, and make them learnable and controllable through gradients. To address the challenges caused by changeable bit widths and temporal lengths, we propose the refined spiking neuron, which can handle different temporal lengths, enable the derivation of gradients for temporal lengths, and suit spike quantization better. In addition, we theoretically formulate the step-size mismatch problem of learnable bit widths, which may incur severe quantization errors to SNN, and accordingly propose the step-size renewal mechanism to alleviate this issue. Experiments on various datasets, including the static CIFAR and ImageNet and the dynamic CIFAR-DVS and DVS-GESTURE, demonstrate that our methods can reduce the overall memory and computation cost while achieving higher accuracy. Particularly, our SEWResNet-34 can achieve a 2.69\% accuracy gain and 4.16$\times$ lower bit budgets over the advanced baseline work on ImageNet. This work will be fully open-sourced.
Abstract (translated)
最近，多比特脉冲神经网络（SNNs）已成为研究热点，旨在追求能源效率高且准确度高的AI。然而，随着涉及更多位数，相关的内存和计算需求上升到一定程度，性能改进变得不成比例。基于不同层表现出不同的重要性以及多余的位可能会浪费和相互干扰这一见解，本文提出了一种用于直接训练SNN的自适应位分配策略，实现了细粒度、逐层的内存和计算资源分配。因此，可以提高SNN的效率和准确性。 具体来说，我们参数化了权重和脉冲的时间长度以及位宽，并通过梯度使其可学习且可控。为了解决由变化的位宽和时间长度带来的挑战，我们提出了精炼的脉冲神经元模型，它可以处理不同的时间长度、支持时间长度的梯度导出，并更好地适应脉冲量化。此外，我们从理论上对可学习位宽度的步长不匹配问题进行了建模，这种问题可能会导致严重的量化误差影响SNN，因此提出了一种步长更新机制来缓解这一问题。 在包括静态CIFAR和ImageNet以及动态CIFAR-DVS和DVS-GESTURE等不同数据集上的实验表明，我们的方法可以在减少整体内存和计算成本的同时实现更高的准确度。特别是，在ImageNet上，我们的SEWResNet-34模型相比最先进的基线工作，获得了2.69%的精度提升，并且位预算减少了4.16倍。本研究将完全开源。
URL
https://arxiv.org/abs/2506.23717
PDF
-
Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset
2025-06-27 18:09:49Vasu Agrawal, Akinniyi Akinyemi, Kathryn Alvero, Morteza Behrooz, Julia Buffalini, Fabio Maria Carlucci, Joy Chen, Junming Chen, Zhang Chen, Shiyang Cheng, Praveen Chowdary, Joe Chuang, Antony D'Avirro, Jon Daly, Ning Dong, Mark Duppenthaler, Cynthia Gao, Jeff Girard, Martin Gleize, Sahir Gomez, Hongyu Gong, Srivathsan Govindarajan, Brandon Han, Sen He, Denise Hernandez, Yordan Hristov, Rongjie Huang, Hirofumi Inaguma, Somya Jain, Raj Janardhan, Qingyao Jia, Christopher Klaiber, Dejan Kovachev, Moneish Kumar, Hang Li, Yilei Li, Pavel Litvin, Wei Liu, Guangyao Ma, Jing Ma, Martin Ma, Xutai Ma, Lucas Mantovani, Sagar Miglani, Sreyas Mohan, Louis-Philippe Morency, Evonne Ng, Kam-Woh Ng, Tu Anh Nguyen, Amia Oberai, Benjamin Peloquin, Juan Pino, Jovan Popovic, Omid Poursaeed, Fabian Prada, Alice Rakotoarison, Alexander Richard, Christophe Ropers, Safiyyah Saleem, Vasu Sharma, Alex Shcherbyna, Jia Shen, Jie Shen, Anastasis Stathopoulos, Anna Sun, Paden Tomasello, Tuan Tran, Arina Turkatenko, Bo Wan, Chao Wang, Jeff Wang, Mary Williamson, Carleigh Wood, Tao Xiang, Yilin Yang, Julien Yao, Chen Zhang, Jiemin Zhang, Xinyue Zhang, Jason Zheng, Pavlo Zhyzheria, Jan Zikes, Michael ZollhoeferAbstract
Human communication involves a complex interplay of verbal and nonverbal signals, essential for conveying meaning and achieving interpersonal goals. To develop socially intelligent AI technologies, it is crucial to develop models that can both comprehend and generate dyadic behavioral dynamics. To this end, we introduce the Seamless Interaction Dataset, a large-scale collection of over 4,000 hours of face-to-face interaction footage from over 4,000 participants in diverse contexts. This dataset enables the development of AI technologies that understand dyadic embodied dynamics, unlocking breakthroughs in virtual agents, telepresence experiences, and multimodal content analysis tools. We also develop a suite of models that utilize the dataset to generate dyadic motion gestures and facial expressions aligned with human speech. These models can take as input both the speech and visual behavior of their interlocutors. We present a variant with speech from an LLM model and integrations with 2D and 3D rendering methods, bringing us closer to interactive virtual agents. Additionally, we describe controllable variants of our motion models that can adapt emotional responses and expressivity levels, as well as generating more semantically-relevant gestures. Finally, we discuss methods for assessing the quality of these dyadic motion models, which are demonstrating the potential for more intuitive and responsive human-AI interactions.
Abstract (translated)
人类沟通涉及复杂的言语和非言语信号的互动，这对传达意义和实现人际目标至关重要。为了开发具有社会智能的人工智能技术，需要建立既能理解又能生成二元行为动态（即两人之间的互动）的模型。为此，我们引入了“无缝交互数据集”，这是一个大规模的数据集合，包含了超过4,000名参与者在各种情境下面对面交流的超过4,000小时的视频记录。该数据集有助于开发能够理解二元身体互动的人工智能技术，从而推动虚拟代理、远程呈现体验和多模态内容分析工具的发展。 我们还开发了一系列模型，这些模型利用数据集生成与人类语言相匹配的一对一动作手势和面部表情。这些模型可以接受对话伙伴的言语和视觉行为作为输入。我们提出了一种变体，其中使用大型语言模型（LLM）产生的语音，并结合了2D和3D渲染方法，这使我们更接近于实现交互式的虚拟代理。 此外，我们还描述了我们的动作模型可控制的变体版本，这些版本可以适应情感反应的不同水平并调整表情强度，同时生成更具语义相关性的手势。最后，我们讨论了评估这些二元动作模型质量的方法，展示了人与AI之间更直观和响应互动的巨大潜力。
URL
https://arxiv.org/abs/2506.22554
PDF
-
Evaluating Pointing Gestures for Target Selection in Human-Robot Collaboration
2025-06-27 10:51:31Noora Sassali, Roel PietersAbstract
Pointing gestures are a common interaction method used in Human-Robot Collaboration for various tasks, ranging from selecting targets to guiding industrial processes. This study introduces a method for localizing pointed targets within a planar workspace. The approach employs pose estimation, and a simple geometric model based on shoulder-wrist extension to extract gesturing data from an RGB-D stream. The study proposes a rigorous methodology and comprehensive analysis for evaluating pointing gestures and target selection in typical robotic tasks. In addition to evaluating tool accuracy, the tool is integrated into a proof-of-concept robotic system, which includes object detection, speech transcription, and speech synthesis to demonstrate the integration of multiple modalities in a collaborative application. Finally, a discussion over tool limitations and performance is provided to understand its role in multimodal robotic systems. All developments are available at: this https URL.
Abstract (translated)
指向手势是人机协作中用于各种任务的一种常见交互方式，从选择目标到指导工业流程都适用。这项研究提出了一种在平面工作空间内定位指向目标的方法。该方法采用姿态估计，并基于肩腕延伸的简单几何模型，从RGB-D流中提取手势数据。本研究提出了评估指向手势和目标选择的严格方法及全面分析，在典型的机器人任务中进行验证。除了评价工具的准确性外，还将该工具集成到一个概念验证型机器人系统中，包括物体检测、语音转录和语音合成功能，以展示在协作应用中多模态数据的融合。最后，讨论了工具的局限性和性能，以便理解它在多模态机器人系统中的角色。所有开发成果都可在以下网址获取：this https URL。
URL
https://arxiv.org/abs/2506.22116
PDF
-
MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation
2025-06-27 09:57:23Dechao Meng, Steven Xiao, Xindi Zhang, Guangyuan Wang, Peng Zhang, Qi Wang, Bang Zhang, Liefeng BoAbstract
Audio-driven portrait animation, which synthesizes realistic videos from reference images using audio signals, faces significant challenges in real-time generation of high-fidelity, temporally coherent animations. While recent diffusion-based methods improve generation quality by integrating audio into denoising processes, their reliance on frame-by-frame UNet architectures introduces prohibitive latency and struggles with temporal consistency. This paper introduces MirrorMe, a real-time, controllable framework built on the LTX video model, a diffusion transformer that compresses video spatially and temporally for efficient latent space denoising. To address LTX's trade-offs between compression and semantic fidelity, we propose three innovations: 1. A reference identity injection mechanism via VAE-encoded image concatenation and self-attention, ensuring identity consistency; 2. A causal audio encoder and adapter tailored to LTX's temporal structure, enabling precise audio-expression synchronization; and 3. A progressive training strategy combining close-up facial training, half-body synthesis with facial masking, and hand pose integration for enhanced gesture control. Extensive experiments on the EMTD Benchmark demonstrate MirrorMe's state-of-the-art performance in fidelity, lip-sync accuracy, and temporal stability.
Abstract (translated)
音频驱动的肖像动画技术通过使用音频信号从参考图像合成逼真的视频，但面临着实时生成高质量、时间连贯动画的重大挑战。虽然最近基于扩散的方法通过将音频集成到去噪过程中提高了生成质量，但是它们依赖于帧对帧UNet架构，这引入了高昂的延迟，并且难以实现时间一致性。本文介绍了一种名为MirrorMe的框架，它建立在LTX视频模型之上，这是一种扩散变换器，该变换器可以在空间和时间上压缩视频以进行高效的潜在空间去噪处理。为了应对LTX在压缩与语义保真度之间的权衡问题，我们提出了三项创新： 1. 通过VAE编码图像的拼接和自我注意机制引入参考身份注入机制，确保身份一致性。 2. 设计了一种因果音频编码器和适配器，以适应LTX的时间结构，并实现精确的声音表情同步。 3. 采用一种渐进式训练策略，结合面部特写训练、半身合成（带有面部遮罩）以及手部姿态整合，进一步提高手势控制能力。 在EMTD基准测试上的大量实验表明，MirrorMe在保真度、唇形同步准确性及时间稳定性方面均表现出最先进的性能。
URL
https://arxiv.org/abs/2506.22065
PDF
-
3Description: An Intuitive Human-AI Collaborative 3D Modeling Approach
Abstract
This paper presents 3Description, an experimental human-AI collaborative approach for intuitive 3D modeling. 3Description aims to address accessibility and usability challenges in traditional 3D modeling by enabling non-professional individuals to co-create 3D models using verbal and gesture descriptions. Through a combination of qualitative research, product analysis, and user testing, 3Description integrates AI technologies such as Natural Language Processing and Computer Vision, powered by OpenAI and MediaPipe. Recognizing the web has wide cross-platform capabilities, 3Description is web-based, allowing users to describe the desired model and subsequently adjust its components using verbal and gestural inputs. In the era of AI and emerging media, 3Description not only contributes to a more inclusive and user-friendly design process, empowering more people to participate in the construction of the future 3D world, but also strives to increase human engagement in co-creation with AI, thereby avoiding undue surrender to technology and preserving human creativity.
Abstract (translated)
本文介绍了3Description，这是一种实验性的人机协作方法，旨在实现直观的三维建模。3Description致力于解决传统三维建模中的可访问性和易用性问题，让非专业人士能够通过口头和手势描述共同创建三维模型。通过定性研究、产品分析和用户测试相结合的方式，3Description整合了自然语言处理和计算机视觉等AI技术，这些技术由OpenAI和MediaPipe提供支持。鉴于网络具有广泛的跨平台能力，3Description是基于网页的，允许用户使用口头和手势输入描述所需模型并随后调整其组件。 在人工智能和新兴媒体的时代，3Description不仅有助于设计过程更加包容且易于用户操作，使更多的人能够参与到未来三维世界的建设中来，而且还努力增加人在与AI协同创作中的参与度，从而避免对技术的过度依赖，并保持人类创造力。
URL
https://arxiv.org/abs/2506.21845
PDF
-
Hierarchical Sub-action Tree for Continuous Sign Language Recognition
2025-06-26 02:27:50Dejie Yang, Zhu Xu, Xinjie Gao, Yang LiuAbstract
Continuous sign language recognition (CSLR) aims to transcribe untrimmed videos into glosses, which are typically textual words. Recent studies indicate that the lack of large datasets and precise annotations has become a bottleneck for CSLR due to insufficient training data. To address this, some works have developed cross-modal solutions to align visual and textual modalities. However, they typically extract textual features from glosses without fully utilizing their knowledge. In this paper, we propose the Hierarchical Sub-action Tree (HST), termed HST-CSLR, to efficiently combine gloss knowledge with visual representation learning. By incorporating gloss-specific knowledge from large language models, our approach leverages textual information more effectively. Specifically, we construct an HST for textual information representation, aligning visual and textual modalities step-by-step and benefiting from the tree structure to reduce computational complexity. Additionally, we impose a contrastive alignment enhancement to bridge the gap between the two modalities. Experiments on four datasets (PHOENIX-2014, PHOENIX-2014T, CSL-Daily, and Sign Language Gesture) demonstrate the effectiveness of our HST-CSLR.
Abstract (translated)
连续手语识别（CSLR）旨在将未修剪的视频转换为文字形式的手语词汇，即“gloss”。最近的研究表明，由于缺乏大规模数据集和精准标注，CSLR的发展受到了训练数据不足的瓶颈限制。为了应对这一挑战，一些研究工作开发了跨模态解决方案来对齐视觉与文本模态。然而，这些方法通常只从gloss中提取文本特征，而没有充分利用其知识。 本文提出了一种层次子动作树（Hierarchical Sub-action Tree, HST），命名为HST-CSLR，旨在高效地结合gloss知识和视觉表示学习。通过整合大型语言模型中的手语特定知识，我们的方法更有效地利用了文字信息。具体而言，我们构建了一个用于文本信息表达的HST，逐步对齐视觉与文本模态，并从树结构中受益以降低计算复杂性。此外，我们引入了一种对比式对齐增强机制来弥合两个模态之间的差距。 在四个数据集（PHOENIX-2014、PHOENIX-2014T、CSL-Daily和Sign Language Gesture）上的实验表明了HST-CSLR的有效性。
URL
https://arxiv.org/abs/2506.20947
PDF
-
How do Foundation Models Compare to Skeleton-Based Approaches for Gesture Recognition in Human-Robot Interaction?
2025-06-25 19:36:45Stephanie K\"as, Anton Burenko, Louis Markert, Onur Alp Culha, Dennis Mack, Timm Linder, Bastian LeibeAbstract
Gestures enable non-verbal human-robot communication, especially in noisy environments like agile production. Traditional deep learning-based gesture recognition relies on task-specific architectures using images, videos, or skeletal pose estimates as input. Meanwhile, Vision Foundation Models (VFMs) and Vision Language Models (VLMs) with their strong generalization abilities offer potential to reduce system complexity by replacing dedicated task-specific modules. This study investigates adapting such models for dynamic, full-body gesture recognition, comparing V-JEPA (a state-of-the-art VFM), Gemini Flash 2.0 (a multimodal VLM), and HD-GCN (a top-performing skeleton-based approach). We introduce NUGGET, a dataset tailored for human-robot communication in intralogistics environments, to evaluate the different gesture recognition approaches. In our experiments, HD-GCN achieves best performance, but V-JEPA comes close with a simple, task-specific classification head - thus paving a possible way towards reducing system complexity, by using it as a shared multi-task model. In contrast, Gemini struggles to differentiate gestures based solely on textual descriptions in the zero-shot setting, highlighting the need of further research on suitable input representations for gestures.
Abstract (translated)
手势使非言语的人机通信成为可能，特别是在像敏捷生产这样的嘈杂环境中。传统的基于深度学习的手势识别方法依赖于特定任务的架构，这些架构使用图像、视频或骨骼姿态估计作为输入。同时，视觉基础模型（VFMs）和视觉语言模型（VLMs），凭借其强大的泛化能力，提供了减少系统复杂性的潜力，通过替换专用的任务特定模块来实现这一点。这项研究调查了将此类模型适应于动态全身手势识别的方法，并比较了最先进的视觉基础模型V-JEPA、多模态视觉语言模型Gemini Flash 2.0和基于骨骼的顶级方法HD-GCN。我们引入了NUGGET数据集，该数据集专为物流环境中的人机通信而设计，用于评估不同的手势识别方法。在我们的实验中，HD-GCN取得了最佳性能，但V-JEPA通过使用简单的任务特定分类头也表现出色，这表明向更简单、多功能模型发展的可能性，以减少系统复杂性。相比之下，在零样本设置下仅根据文本描述区分手势时，Gemini面临困难，这强调了进一步研究适合手势的输入表示形式的重要性。
URL
https://arxiv.org/abs/2506.20795
PDF
-
SPARK: Graph-Based Online Semantic Integration System for Robot Task Planning
2025-06-25 13:02:59Mimo Shirasaka, Yuya Ikeda, Tatsuya Matsushima, Yutaka Matsuo, Yusuke IwasawaAbstract
The ability to update information acquired through various means online during task execution is crucial for a general-purpose service robot. This information includes geometric and semantic data. While SLAM handles geometric updates on 2D maps or 3D point clouds, online updates of semantic information remain unexplored. We attribute the challenge to the online scene graph representation, for its utility and scalability. Building on previous works regarding offline scene graph representations, we study online graph representations of semantic information in this work. We introduce SPARK: Spatial Perception and Robot Knowledge Integration. This framework extracts semantic information from environment-embedded cues and updates the scene graph accordingly, which is then used for subsequent task planning. We demonstrate that graph representations of spatial relationships enhance the robot system's ability to perform tasks in dynamic environments and adapt to unconventional spatial cues, like gestures.
Abstract (translated)
在线更新通过各种手段获取的信息对于通用服务机器人在任务执行过程中的信息处理至关重要，这些信息包括几何和语义数据。虽然SLAM（同步定位与地图构建）可以处理二维地图或三维点云的几何更新问题，但关于语义信息的在线更新仍然未被充分探索。我们把挑战归因于在线场景图表示的有效性和可扩展性。在前人对离线场景图表示工作的基础上，我们在本工作中研究了语义信息的在线图形表示方法。 为此，我们提出了SPARK框架：空间感知与机器人知识集成。该框架从环境中嵌入的线索中提取语义信息，并相应地更新场景图，然后将此用于后续的任务规划。我们证明，通过使用空间关系的图表示方式可以增强机器人的系统能力，在动态环境中的任务执行表现和适应非常规的空间提示（如手势）方面的能力得到提升。
URL
https://arxiv.org/abs/2506.20394
PDF
-
ReactEMG: Zero-Shot, Low-Latency Intent Detection via sEMG
2025-06-24 17:28:43Runsheng Wang, Xinyue Zhu, Ava Chen, Jingxi Xu, Lauren Winterbottom, Dawn M. Nilsen, Joel Stein, Matei CiocarlieAbstract
Surface electromyography (sEMG) signals show promise for effective human-computer interfaces, particularly in rehabilitation and prosthetics. However, challenges remain in developing systems that respond quickly and reliably to user intent, across different subjects and without requiring time-consuming calibration. In this work, we propose a framework for EMG-based intent detection that addresses these challenges. Unlike traditional gesture recognition models that wait until a gesture is completed before classifying it, our approach uses a segmentation strategy to assign intent labels at every timestep as the gesture unfolds. We introduce a novel masked modeling strategy that aligns muscle activations with their corresponding user intents, enabling rapid onset detection and stable tracking of ongoing gestures. In evaluations against baseline methods, considering both accuracy and stability for device control, our approach surpasses state-of-the-art performance in zero-shot transfer conditions, demonstrating its potential for wearable robotics and next-generation prosthetic systems. Our project page is available at: this https URL
Abstract (translated)
表面肌电图（sEMG）信号在人机接口方面显示出巨大的潜力，尤其是在康复和假肢领域。然而，在不同用户间快速且可靠地响应用户意图的系统开发仍面临挑战，特别是在不需要耗时校准的情况下。为此，我们提出了一种基于EMG的意图检测框架，旨在解决这些问题。 与传统的手势识别模型等待完整的手势完成后再进行分类不同，我们的方法采用了一种分段策略，在每个时间点上随着手势的发展分配意图标签。我们引入了一种新颖的掩码建模策略，该策略将肌肉激活与相应的用户意图对齐，从而实现快速起始检测和正在进行的手势的稳定跟踪。 在与基线方法进行评估时，我们的方法在零样本迁移条件下（即无需特定于每个用户的校准）实现了设备控制的准确性和稳定性方面的性能超越。这表明了其在可穿戴机器人技术和下一代假肢系统中的巨大潜力。 项目页面在此网址: [this https URL] (请将方括号内的内容替换为实际链接)
URL
https://arxiv.org/abs/2506.19815
PDF
-
Robot Tactile Gesture Recognition Based on Full-body Modular E-skin
2025-06-23 02:59:05Shuo Jiang, Boce Hu, Linfeng Zhao, Lawson L. S. WongAbstract
With the development of robot electronic skin technology, various tactile sensors, enhanced by AI, are unlocking a new dimension of perception for robots. In this work, we explore how robots equipped with electronic skin can recognize tactile gestures and interpret them as human commands. We developed a modular robot E-skin, composed of multiple irregularly shaped skin patches, which can be assembled to cover the robot's body while capturing real-time pressure and pose data from thousands of sensing points. To process this information, we propose an equivariant graph neural network-based recognizer that efficiently and accurately classifies diverse tactile gestures, including poke, grab, stroke, and double-pat. By mapping the recognized gestures to predefined robot actions, we enable intuitive human-robot interaction purely through tactile input.
Abstract (translated)
随着机器人电子皮肤技术的发展，各种由人工智能增强的触觉传感器为机器人开启了感知的新维度。在这项工作中，我们探索了配备有电子皮肤的机器人如何识别触觉手势并将其解释为人机命令。我们开发了一种模块化的机器人E-皮肤，它由多个不规则形状的皮肤补丁组成，可以组装覆盖机器人的身体，并从数千个感测点实时捕捉压力和姿态数据。为了处理这些信息，我们提出了一种基于等变图神经网络的识别器，能够高效且准确地分类包括戳、抓取、抚摸和双拍在内的各种触觉手势。通过将识别出的手势映射到预定义的机器人动作上，我们实现了仅通过触觉输入进行直观的人机交互。
URL
https://arxiv.org/abs/2506.18256
PDF
-
Imitation Learning for Active Neck Motion Enabling Robot Manipulation beyond the Field of View
2025-06-21 07:37:49Koki Nakagawa, Yoshiyuki Ohmura, Yasuo KuniyoshiAbstract
Most prior research in deep imitation learning has predominantly utilized fixed cameras for image input, which constrains task performance to the predefined field of view. However, enabling a robot to actively maneuver its neck can significantly expand the scope of imitation learning to encompass a wider variety of tasks and expressive actions such as neck gestures. To facilitate imitation learning in robots capable of neck movement while simultaneously performing object manipulation, we propose a teaching system that systematically collects datasets incorporating neck movements while minimizing discomfort caused by dynamic viewpoints during teleoperation. In addition, we present a novel network model for learning manipulation tasks including active neck motion. Experimental results showed that our model can achieve a high success rate of around 90\%, regardless of the distraction from the viewpoint variations by active neck motion. Moreover, the proposed model proved particularly effective in challenging scenarios, such as when objects were situated at the periphery or beyond the standard field of view, where traditional models struggled. The proposed approach contributes to the efficiency of dataset collection and extends the applicability of imitation learning to more complex and dynamic scenarios.
Abstract (translated)
大多数关于深度模仿学习的早期研究主要采用固定摄像头输入图像，这限制了任务性能仅限于预定义的视野范围内。然而，让机器人能够主动操纵其颈部可以显著扩展模仿学习的应用范围，包括更多样化的任务和表达性动作，如颈部手势。为了促进能够在进行物体操作的同时执行颈部运动的机器人的模仿学习，我们提出了一种教学系统，该系统有系统地收集包含颈部运动的数据集，并在远程操控时尽量减少因动态视角变化带来的不适感。此外，我们还提出了一个新颖的网络模型来学习包括主动颈部动作在内的操纵任务。实验结果显示，无论是在主动颈部运动导致的视点变化中，我们的模型都能实现大约90%的成功率。此外，在一些具有挑战性的场景（如物体位于视野边缘或超出标准视野范围）下，所提出的模型表现出特别的有效性，而传统模型在这些情况下则表现不佳。我们提出的方法不仅提高了数据集收集的效率，还扩展了模仿学习在更复杂和动态场景中的应用范围。
URL
https://arxiv.org/abs/2506.17624
PDF
-
IsoNet: Causal Analysis of Multimodal Transformers for Neuromuscular Gesture Classification
2025-06-20 04:31:32Eion Tyacke, Kunal Gupta, Jay Patel, Rui LiAbstract
Hand gestures are a primary output of the human motor system, yet the decoding of their neuromuscular signatures remains a bottleneck for basic neuroscience and assistive technologies such as prosthetics. Traditional human-machine interface pipelines rely on a single biosignal modality, but multimodal fusion can exploit complementary information from sensors. We systematically compare linear and attention-based fusion strategies across three architectures: a Multimodal MLP, a Multimodal Transformer, and a Hierarchical Transformer, evaluating performance on scenarios with unimodal and multimodal inputs. Experiments use two publicly available datasets: NinaPro DB2 (sEMG and accelerometer) and HD-sEMG 65-Gesture (high-density sEMG and force). Across both datasets, the Hierarchical Transformer with attention-based fusion consistently achieved the highest accuracy, surpassing the multimodal and best single-modality linear-fusion MLP baseline by over 10% on NinaPro DB2 and 3.7% on HD-sEMG. To investigate how modalities interact, we introduce an Isolation Network that selectively silences unimodal or cross-modal attention pathways, quantifying each group of token interactions' contribution to downstream decisions. Ablations reveal that cross-modal interactions contribute approximately 30% of the decision signal across transformer layers, highlighting the importance of attention-driven fusion in harnessing complementary modality information. Together, these findings reveal when and how multimodal fusion would enhance biosignal classification and also provides mechanistic insights of human muscle activities. The study would be beneficial in the design of sensor arrays for neurorobotic systems.
Abstract (translated)
手势是人类运动系统的主要输出之一，但对其神经肌肉特征的解码仍然是基础神经科学和假肢等辅助技术的一个瓶颈。传统的“人机接口”管道依赖单一类型的生物信号模式，而多模态融合可以通过利用来自不同传感器互补的信息来改进这一点。我们在三种架构（一个多模态MLP、一个多模态Transformer以及一个层级式Transformer）中系统地比较了线性与基于注意力的融合策略在单模态和多模态输入场景下的性能表现。实验使用两个公开数据集：NinaPro DB2（表面肌电信号sEMG和加速度计）和HD-sEMG 65-Gesture（高密度sEMG和力）。在这两个数据集中，基于注意力的融合层级式Transformer在所有情况下都达到了最高的准确度，在NinaPro DB2上超过了多模态和最佳单模态线性融合MLP基准超过10%，在HD-sEMG上则提高了3.7%。 为了探讨模式如何相互作用，我们引入了一个隔离网络，可以有选择地抑制单模态或跨模态注意力路径，并量化每组令牌互动对下游决策的贡献。消融实验显示，跨模态交互在整个Transformer层中为决策信号贡献了约30%，突显了基于注意机制融合在利用互补模式信息中的重要性。 这些发现揭示了多模态融合何时以及如何增强生物信号分类，并提供了有关人类肌肉活动的机理见解。该研究对于神经机器系统传感器阵列的设计具有指导意义。
URL
https://arxiv.org/abs/2506.16744
PDF
-
CLIP-MG: Guiding Semantic Attention with Skeletal Pose Features and RGB Data for Micro-Gesture Recognition on the iMiGUE Dataset
2025-06-19 15:16:06Santosh Patapati, Trisanth Srinivasan, Amith AdirajuAbstract
Micro-gesture recognition is a challenging task in affective computing due to the subtle, involuntary nature of the gestures and their low movement amplitude. In this paper, we introduce a Pose-Guided Semantics-Aware CLIP-based architecture, or CLIP for Micro-Gesture recognition (CLIP-MG), a modified CLIP model tailored for micro-gesture classification on the iMiGUE dataset. CLIP-MG integrates human pose (skeleton) information into the CLIP-based recognition pipeline through pose-guided semantic query generation and a gated multi-modal fusion mechanism. The proposed model achieves a Top-1 accuracy of 61.82%. These results demonstrate both the potential of our approach and the remaining difficulty in fully adapting vision-language models like CLIP for micro-gesture recognition.
Abstract (translated)
微手势识别是情感计算中的一个挑战性任务，由于这些细微、不自觉的手势动作幅度较低。本文介绍了Pose-Guided Semantics-Aware CLIP-based架构（简称CLIP-MG），这是一种为iMiGUE数据集上的微手势分类而修改的CLIP模型。CLIP-MG通过姿势引导的语义查询生成和门控多模态融合机制，将人体姿态（骨骼）信息整合到基于CLIP的识别管道中。所提出的模型达到了61.82%的Top-1准确率。这些结果既展示了我们方法的潜力，也反映了完全适应视觉-语言模型如CLIP进行微手势识别的难度仍然存在。
URL
https://arxiv.org/abs/2506.16385
PDF
-
Co-Speech Gesture and Facial Expression Generation for Non-Photorealistic 3D Characters
2025-06-19 09:14:50Taisei Omine (Sony Group Corporation), Naoyuki Kawabata (Sony Group Corporation), Fuminori Homma (Sony Group Corporation)Abstract
With the advancement of conversational AI, research on bodily expressions, including gestures and facial expressions, has also progressed. However, many existing studies focus on photorealistic avatars, making them unsuitable for non-photorealistic characters, such as those found in anime. This study proposes methods for expressing emotions, including exaggerated expressions unique to non-photorealistic characters, by utilizing expression data extracted from comics and dialogue-specific semantic gestures. A user study demonstrated significant improvements across multiple aspects when compared to existing research.
Abstract (translated)
随着对话人工智能的进步，关于身体语言的研究，包括手势和面部表情，也得到了发展。然而，许多现有的研究主要集中在拟真的虚拟角色上，这使得这些方法不适用于非写实风格的角色，比如动画中的角色。本研究提出了一种利用从漫画中提取的表情数据以及特定于对话的语义手势来表达情感的方法，包括非写实角色特有的夸张表情。一项用户研究表明，在多个方面相比现有研究成果都有显著改进。
URL
https://arxiv.org/abs/2506.16159
PDF
-
Accessible Gesture-Driven Augmented Reality Interaction System
2025-06-18 07:10:48Yikan WangAbstract
Augmented reality (AR) offers immersive interaction but remains inaccessible for users with motor impairments or limited dexterity due to reliance on precise input methods. This study proposes a gesture-based interaction system for AR environments, leveraging deep learning to recognize hand and body gestures from wearable sensors and cameras, adapting interfaces to user capabilities. The system employs vision transformers (ViTs), temporal convolutional networks (TCNs), and graph attention networks (GATs) for gesture processing, with federated learning ensuring privacy-preserving model training across diverse users. Reinforcement learning optimizes interface elements like menu layouts and interaction modes. Experiments demonstrate a 20% improvement in task completion efficiency and a 25% increase in user satisfaction for motor-impaired users compared to baseline AR systems. This approach enhances AR accessibility and scalability. Keywords: Deep learning, Federated learning, Gesture recognition, Augmented reality, Accessibility, Human-computer interaction
Abstract (translated)
增强现实（AR）提供了沉浸式的互动体验，但由于依赖于精确的输入方法，对于运动障碍或灵巧度受限的用户来说仍然难以使用。本研究提出了一种基于手势的交互系统，用于增强现实环境，该系统利用深度学习从穿戴式传感器和摄像头中识别手部和身体动作，并根据用户的实际能力调整界面。 该系统采用视觉变换器（ViT）、时间卷积网络（TCN）以及图注意力网络（GAT）处理手势数据。通过联邦学习在不同用户之间进行隐私保护的模型训练，确保了系统的可扩展性和安全性。此外，强化学习用于优化诸如菜单布局和互动模式等界面元素。 实验结果显示，在任务完成效率上，与基准增强现实系统相比，该系统使运动障碍用户的性能提高了20%，并且其满意度也增加了25%。这一方法显著增强了AR的可访问性及扩展性。 关键词：深度学习、联邦学习、手势识别、增强现实、无障碍设计、人机交互
URL
https://arxiv.org/abs/2506.15189
PDF
