Gesture
-
Real-Time Sign Language Gestures to Speech Transcription using Deep Learning
Abstract
Communication barriers pose significant challenges for individuals with hearing and speech impairments, often limiting their ability to effectively interact in everyday environments. This project introduces a real-time assistive technology solution that leverages advanced deep learning techniques to translate sign language gestures into textual and audible speech. By employing convolution neural networks (CNN) trained on the Sign Language MNIST dataset, the system accurately classifies hand gestures captured live via webcam. Detected gestures are instantaneously translated into their corresponding meanings and transcribed into spoken language using text-to-speech synthesis, thus facilitating seamless communication. Comprehensive experiments demonstrate high model accuracy and robust real-time performance with some latency, highlighting the system's practical applicability as an accessible, reliable, and user-friendly tool for enhancing the autonomy and integration of sign language users in diverse social settings.
Abstract (translated)
沟通障碍对听障和言语障碍人士构成了重大挑战，往往限制了他们在日常环境中有效互动的能力。本项目介绍了一种实时辅助技术解决方案，该方案利用先进的深度学习技术将手语手势翻译成文本和语音输出。通过使用在Sign Language MNIST数据集上训练的卷积神经网络（CNN），系统能够准确地识别并通过摄像头捕获的手势进行分类。检测到的手势会即时转换为其对应的含义，并通过文字转语音合成技术转化为口语，从而实现无缝交流。 全面的实验表明，该模型具有高精度和强大的实时性能，尽管存在一些延迟问题，但仍突显了作为无障碍、可靠且用户友好的工具在各种社交环境中提升手语使用者自主性和融入度的应用潜力。
URL
https://arxiv.org/abs/2508.12713
PDF
-
HOMI: Ultra-Fast EdgeAI platform for Event Cameras
2025-08-18 05:47:48Shankaranarayanan H, Satyapreet Singh Yadav, Adithya Krishna, Ajay Vikram P, Mahesh Mehendale, Chetan Singh ThakurAbstract
Event cameras offer significant advantages for edge robotics applications due to their asynchronous operation and sparse, event-driven output, making them well-suited for tasks requiring fast and efficient closed-loop control, such as gesture-based human-robot interaction. Despite this potential, existing event processing solutions remain limited, often lacking complete end-to-end implementations, exhibiting high latency, and insufficiently exploiting event data sparsity. In this paper, we present HOMI, an ultra-low latency, end-to-end edge AI platform comprising a Prophesee IMX636 event sensor chip with an Xilinx Zynq UltraScale+MPSoC FPGA chip, deploying an in-house developed AI accelerator. We have developed hardware-optimized pre-processing pipelines supporting both constant-time and constant-event modes for histogram accumulation, linear and exponential time surfaces. Our general-purpose implementation caters to both accuracy-driven and low-latency applications. HOMI achieves 94% accuracy on the DVS Gesture dataset as a use case when configured for high accuracy operation and provides a throughput of 1000 fps for low-latency configuration. The hardware-optimised pipeline maintains a compact memory footprint and utilises only 33% of the available LUT resources on the FPGA, leaving ample headroom for further latency reduction, model parallelisation, multi-task deployments, or integration of more complex architectures.
Abstract (translated)
事件相机在边缘机器人应用中提供了显著的优势，这是因为它们的异步操作和稀疏、事件驱动的输出特性非常适合需要快速且高效闭环控制的任务，如基于手势的人机交互。尽管存在这些潜力，现有的事件处理解决方案仍然有限，通常缺乏完整的端到端实现，表现出高延迟，并未能充分挖掘事件数据的稀疏性。 在这篇论文中，我们介绍了HOMI，这是一个超低延时、端到端边缘AI平台，集成了Prophesee IMX636事件传感器芯片与Xilinx Zynq UltraScale+ MPSoC FPGA芯片，并部署了一个内部开发的人工智能加速器。我们开发了硬件优化的预处理流水线，支持常数时间模式和常数事件模式下的直方图累积、以及线性和指数时间表面。我们的通用实现同时适用于以准确性驱动的应用程序和低延迟应用程序。 当配置为高精度操作时，HOMI在DVS手势数据集上实现了94%的准确率，并且在低延迟配置下提供了每秒1000帧的吞吐量。硬件优化的流水线保持了紧凑的记忆占用，并仅使用FPGA上可用LUT资源的33%，为未来的延时减少、模型并行化、多任务部署或更复杂架构的集成留出了充足的空间。
URL
https://arxiv.org/abs/2508.12637
PDF
-
Tactile Gesture Recognition with Built-in Joint Sensors for Industrial Robots
2025-08-17 17:04:58Deqing Song, Weimin Yang, Maryam Rezayati, Hans Wernher van de VennAbstract
While gesture recognition using vision or robot skins is an active research area in Human-Robot Collaboration (HRC), this paper explores deep learning methods relying solely on a robot's built-in joint sensors, eliminating the need for external sensors. We evaluated various convolutional neural network (CNN) architectures and collected two datasets to study the impact of data representation and model architecture on the recognition accuracy. Our results show that spectrogram-based representations significantly improve accuracy, while model architecture plays a smaller role. We also tested generalization to new robot poses, where spectrogram-based models performed better. Implemented on a Franka Emika Research robot, two of our methods, STFT2DCNN and STT3DCNN, achieved over 95% accuracy in contact detection and gesture classification. These findings demonstrate the feasibility of external-sensor-free tactile recognition and promote further research toward cost-effective, scalable solutions for HRC.
Abstract (translated)
尽管基于视觉或机器人皮肤的姿势识别在人机协作（HRC）领域中是一个活跃的研究方向，本文探索了仅依赖于机器人内置关节传感器的深度学习方法，从而无需使用外部传感器。我们评估了几种卷积神经网络（CNN）架构，并收集了两个数据集以研究数据表示和模型架构对识别准确率的影响。我们的结果显示基于频谱图的表示显著提高了准确性，而模型架构的作用较小。我们在新机器人姿态下的泛化测试中发现基于频谱图的模型表现更好。 在Franka Emika Research机器人的实施中，我们提出的STFT2DCNN和STT3DCNN两种方法，在接触检测和姿势分类任务上均达到了超过95%的准确率。这些研究结果表明了无需外部传感器的手感识别的可行性，并促进了向成本效益高、可扩展的人机协作解决方案的研究发展。
URL
https://arxiv.org/abs/2508.12435
PDF
-
iTrace: Click-Based Gaze Visualization on the Apple Vision Pro
2025-08-17 07:34:37Esra Mehmedova, Santiago Berrezueta-Guzman, Stefan WagnerAbstract
The Apple Vision Pro is equipped with accurate eye-tracking capabilities, yet the privacy restrictions on the device prevent direct access to continuous user gaze data. This study introduces iTrace, a novel application that overcomes these limitations through click-based gaze extraction techniques, including manual methods like a pinch gesture, and automatic approaches utilizing dwell control or a gaming controller. We developed a system with a client-server architecture that captures the gaze coordinates and transforms them into dynamic heatmaps for video and spatial eye tracking. The system can generate individual and averaged heatmaps, enabling analysis of personal and collective attention patterns. To demonstrate its effectiveness and evaluate the usability and performance, a study was conducted with two groups of 10 participants, each testing different clicking methods. The 8BitDo controller achieved higher average data collection rates at 14.22 clicks/s compared to 0.45 clicks/s with dwell control, enabling significantly denser heatmap visualizations. The resulting heatmaps reveal distinct attention patterns, including concentrated focus in lecture videos and broader scanning during problem-solving tasks. By allowing dynamic attention visualization while maintaining a high gaze precision of 91 %, iTrace demonstrates strong potential for a wide range of applications in educational content engagement, environmental design evaluation, marketing analysis, and clinical cognitive assessment. Despite the current gaze data restrictions on the Apple Vision Pro, we encourage developers to use iTrace only in research settings.
Abstract (translated)
苹果Vision Pro配备了精确的眼动追踪功能，但由于设备的隐私限制，无法直接访问连续的用户注视数据。本研究引入了一种名为iTrace的新应用，该应用通过基于点击的眼动提取技术克服了这些限制，包括手动方法（如捏合手势）和自动方法（利用停留控制或游戏手柄）。我们开发了一个采用客户端-服务器架构的系统，可以捕获注视坐标并将其转换为动态热图，用于视频和空间眼动追踪。该系统能够生成个人和平均化的热图，从而分析个人和群体的关注模式。 为了展示其有效性，并评估其可用性和性能，本研究对两组各10名参与者进行了测试，每组使用不同的点击方法。8BitDo控制器在数据收集率方面表现更佳，平均每秒为14.22次点击（相比之下，停留控制仅为0.45次/秒），这使得生成的热图可视化更为密集。 生成的热图揭示了明显不同的注意力模式：在讲座视频中表现为集中关注，在问题解决任务中则表现为广泛的扫描。通过允许动态注意力可视化，并保持91%的眼动精确度，iTrace显示出在教育内容参与、环境设计评估、市场分析和临床认知评估等广泛领域的强大潜力。 尽管当前的苹果Vision Pro对注视数据有限制，我们鼓励开发者仅在研究环境中使用iTrace。
URL
https://arxiv.org/abs/2508.12268
PDF
-
Interpretable Robot Control via Structured Behavior Trees and Large Language Models
2025-08-13 08:53:13Ingrid Ma\'eva Chekam, Ines Pastor-Martinez, Ali Tourani, Jose Andres Millan-Romera, Laura Ribeiro, Pedro Miguel Bastos Soares, Holger Voos, Jose Luis Sanchez-LopezAbstract
As intelligent robots become more integrated into human environments, there is a growing need for intuitive and reliable Human-Robot Interaction (HRI) interfaces that are adaptable and more natural to interact with. Traditional robot control methods often require users to adapt to interfaces or memorize predefined commands, limiting usability in dynamic, unstructured environments. This paper presents a novel framework that bridges natural language understanding and robotic execution by combining Large Language Models (LLMs) with Behavior Trees. This integration enables robots to interpret natural language instructions given by users and translate them into executable actions by activating domain-specific plugins. The system supports scalable and modular integration, with a primary focus on perception-based functionalities, such as person tracking and hand gesture recognition. To evaluate the system, a series of real-world experiments was conducted across diverse environments. Experimental results demonstrate that the proposed approach is practical in real-world scenarios, with an average cognition-to-execution accuracy of approximately 94%, making a significant contribution to HRI systems and robots. The complete source code of the framework is publicly available at this https URL.
Abstract (translated)
随着智能机器人在人类环境中的集成越来越紧密，对于直观且可靠的机器人与人交互（HRI）界面的需求也在增长。这些界面需要具备适应性和自然性以更好地进行互动。传统机器人的控制方法通常要求用户去适应特定的接口或记忆预定义的命令，这限制了其在动态、非结构化环境中的可用性。本文提出了一种新颖的框架，该框架结合大型语言模型（LLMs）与行为树，旨在将自然语言理解和机器人执行相结合。这种集成使机器人能够解读用户给出的自然语言指令，并通过激活特定领域的插件将其转化为可执行的动作。系统支持模块化和可扩展的整合，主要聚焦于感知功能，例如人员跟踪和手势识别。 为了评估该系统的性能，在各种环境进行了系列现实世界实验。实验证明了所提出的这种方法在实际场景中的实用性，平均认知到执行准确性达到约94%，这为HRI系统及机器人技术做出了重要贡献。框架的完整源代码可在以下网址公开获取：[此URL]（请将"[此URL]"替换为具体的URL）。
URL
https://arxiv.org/abs/2508.09621
PDF
-
Generation of Indian Sign Language Letters, Numbers, and Words
2025-08-13 06:10:20Ajeet Kumar Yadav, Nishant Kumar, Rathna G NAbstract
Sign language, which contains hand movements, facial expressions and bodily gestures, is a significant medium for communicating with hard-of-hearing people. A well-trained sign language community communicates easily, but those who don't know sign language face significant challenges. Recognition and generation are basic communication methods between hearing and hard-of-hearing individuals. Despite progress in recognition, sign language generation still needs to be explored. The Progressive Growing of Generative Adversarial Network (ProGAN) excels at producing high-quality images, while the Self-Attention Generative Adversarial Network (SAGAN) generates feature-rich images at medium resolutions. Balancing resolution and detail is crucial for sign language image generation. We are developing a Generative Adversarial Network (GAN) variant that combines both models to generate feature-rich, high-resolution, and class-conditional sign language images. Our modified Attention-based model generates high-quality images of Indian Sign Language letters, numbers, and words, outperforming the traditional ProGAN in Inception Score (IS) and Fréchet Inception Distance (FID), with improvements of 3.2 and 30.12, respectively. Additionally, we are publishing a large dataset incorporating high-quality images of Indian Sign Language alphabets, numbers, and 129 words.
Abstract (translated)
手语包含手势、面部表情和身体动作，是与听力障碍人士沟通的重要媒介。经过良好训练的手语社区能够轻松交流，但对于不懂手语的人来说则面临巨大挑战。识别和生成是听觉健全者和听力障碍者之间基本的交流方式。尽管在识别方面取得了进展，但手语生成仍需进一步探索。 渐进式生长生成对抗网络（ProGAN）擅长生成高质量图像，而自注意力生成对抗网络（SAGAN）则擅长在中等分辨率下生成细节丰富的图像。对于手语图像生成而言，在平衡分辨率和细节度上至关重要。我们正在开发一种结合这两种模型的生成对抗网络（GAN）变体，以生成特征丰富、高分辨率且具有类别条件的手语图像。 我们的基于注意力机制的改良模型能够生成高质量的印度手语字母、数字及词汇图像，在Inception分数（IS）和Fréchet Inception距离（FID）指标上超越了传统的ProGAN，分别提高了3.2和30.12。此外，我们还计划发布一个包含高分辨率印度手语字母、数字以及129个词语的大规模数据集。
URL
https://arxiv.org/abs/2508.09522
PDF
-
X-UniMotion: Animating Human Images with Expressive, Unified and Identity-Agnostic Motion Latents
2025-08-12 22:47:20Guoxian Song, Hongyi Xu, Xiaochen Zhao, You Xie, Tianpei Gu, Zenan Li, Chenxu Zhang, Linjie LuoAbstract
We present X-UniMotion, a unified and expressive implicit latent representation for whole-body human motion, encompassing facial expressions, body poses, and hand gestures. Unlike prior motion transfer methods that rely on explicit skeletal poses and heuristic cross-identity adjustments, our approach encodes multi-granular motion directly from a single image into a compact set of four disentangled latent tokens -- one for facial expression, one for body pose, and one for each hand. These motion latents are both highly expressive and identity-agnostic, enabling high-fidelity, detailed cross-identity motion transfer across subjects with diverse identities, poses, and spatial configurations. To achieve this, we introduce a self-supervised, end-to-end framework that jointly learns the motion encoder and latent representation alongside a DiT-based video generative model, trained on large-scale, diverse human motion datasets. Motion--identity disentanglement is enforced via 2D spatial and color augmentations, as well as synthetic 3D renderings of cross-identity subject pairs under shared poses. Furthermore, we guide motion token learning with auxiliary decoders that promote fine-grained, semantically aligned, and depth-aware motion embeddings. Extensive experiments show that X-UniMotion outperforms state-of-the-art methods, producing highly expressive animations with superior motion fidelity and identity preservation.
Abstract (translated)
我们提出了一种名为X-UniMotion的统一且表达性强的隐式潜在表示方法，用于全身人体运动，包括面部表情、身体姿态和手部动作。与以往依赖显式的骨骼姿势和启发式的跨身份调整的方法不同，我们的方法直接从单张图像中编码多粒度的运动信息，并将其压缩为四个解耦的潜在标记——一个用于面部表情，一个用于身体姿态，每个手各一个。这些运动潜变量既高度表达又能跨越不同的个体，使得在具有多样化身份、姿势和空间配置的不同主体之间进行高质量、细致的跨身份动作转移成为可能。 为了实现这一目标，我们引入了一个自我监督的端到端框架，该框架同时学习了运动编码器和潜在表示以及一个基于DiT（Diffusion Transformer）的视频生成模型，并在大规模、多样化的全身人类运动数据集上进行训练。通过二维空间和颜色增强以及跨身份主体对共享姿势下的合成三维渲染来强制执行动作-身份解耦。此外，我们还使用辅助解码器引导动作令牌学习，这些解码器促进精细的语义对齐和深度感知的动作嵌入。 广泛的实验表明，X-UniMotion在保持运动真实性和身份一致性的前提下，能够生成高度表达性动画，在性能上超越了现有的先进技术。
URL
https://arxiv.org/abs/2508.09383
PDF
-
FusionEnsemble-Net: An Attention-Based Ensemble of Spatiotemporal Networks for Multimodal Sign Language Recognition
2025-08-12 21:44:23Md. Milon Islam, Md Rezwanul Haque, S M Taslim Uddin Raju, Fakhri KarrayAbstract
Accurate recognition of sign language in healthcare communication poses a significant challenge, requiring frameworks that can accurately interpret complex multimodal gestures. To deal with this, we propose FusionEnsemble-Net, a novel attention-based ensemble of spatiotemporal networks that dynamically fuses visual and motion data to enhance recognition accuracy. The proposed approach processes RGB video and range Doppler map radar modalities synchronously through four different spatiotemporal networks. For each network, features from both modalities are continuously fused using an attention-based fusion module before being fed into an ensemble of classifiers. Finally, the outputs of these four different fused channels are combined in an ensemble classification head, thereby enhancing the model's robustness. Experiments demonstrate that FusionEnsemble-Net outperforms state-of-the-art approaches with a test accuracy of 99.44% on the large-scale MultiMeDaLIS dataset for Italian Sign Language. Our findings indicate that an ensemble of diverse spatiotemporal networks, unified by attention-based fusion, yields a robust and accurate framework for complex, multimodal isolated gesture recognition tasks. The source code is available at: this https URL.
Abstract (translated)
在医疗通信中准确识别手语是一项重大挑战，需要能够精确解析复杂多模态手势的框架。为应对这一问题，我们提出了FusionEnsemble-Net，这是一种基于注意力机制的时空网络集成模型，该模型可以动态融合视觉和运动数据以提高识别精度。提出的这种方法通过四种不同的时空网络同步处理RGB视频和范围多普勒地图雷达模式。对于每个网络，两种模态的数据特征连续地使用基于注意的融合模块进行融合后输入分类器集成。最后，这四个不同融合通道的输出在集成分类头中结合在一起，从而增强模型的鲁棒性。实验表明，在处理意大利手语的大规模MultiMeDaLIS数据集时，FusionEnsemble-Net以99.44%的测试准确率超过了当前最先进的方法。我们的研究结果表明，通过基于注意机制的融合统一起来的一系列时空网络集成提供了一种稳健且准确的方法来解决复杂多模态孤立手势识别任务。源代码可在以下网址获取：[此处插入链接]。
URL
https://arxiv.org/abs/2508.09362
PDF
-
Generation of Real-time Robotic Emotional Expressions Learning from Human Demonstration in Mixed Reality
2025-08-12 15:10:50Chao Wang, Michael Gienger, Fan ZhangAbstract
Expressive behaviors in robots are critical for effectively conveying their emotional states during interactions with humans. In this work, we present a framework that autonomously generates realistic and diverse robotic emotional expressions based on expert human demonstrations captured in Mixed Reality (MR). Our system enables experts to teleoperate a virtual robot from a first-person perspective, capturing their facial expressions, head movements, and upper-body gestures, and mapping these behaviors onto corresponding robotic components including eyes, ears, neck, and arms. Leveraging a flow-matching-based generative process, our model learns to produce coherent and varied behaviors in real-time in response to moving objects, conditioned explicitly on given emotional states. A preliminary test validated the effectiveness of our approach for generating autonomous expressions.
Abstract (translated)
机器人表达性行为在与人类互动时传达其情感状态方面至关重要。在这项工作中，我们提出了一种框架，该框架能够自主生成基于混合现实（MR）中捕捉到的人类专家演示的逼真且多样化的机器人情绪表情。我们的系统使专家可以从第一人称视角远程操作一个虚拟机器人，并捕获他们的面部表情、头部动作和上半身手势，并将这些行为映射到机器人的相应组件，包括眼睛、耳朵、颈部和手臂。通过基于流匹配的生成过程，我们的模型能够实时学习并根据给定的情绪状态产生连贯且多样的行为以应对移动物体。初步测试验证了我们方法在生成自主表情方面的有效性。
URL
https://arxiv.org/abs/2508.08999
PDF
-
Livia: An Emotion-Aware AR Companion Powered by Modular AI Agents and Progressive Memory Compression
2025-08-12 14:07:22Rui Xi, Xianghan WangAbstract
Loneliness and social isolation pose significant emotional and health challenges, prompting the development of technology-based solutions for companionship and emotional support. This paper introduces Livia, an emotion-aware augmented reality (AR) companion app designed to provide personalized emotional support by combining modular artificial intelligence (AI) agents, multimodal affective computing, progressive memory compression, and AR driven embodied interaction. Livia employs a modular AI architecture with specialized agents responsible for emotion analysis, dialogue generation, memory management, and behavioral orchestration, ensuring robust and adaptive interactions. Two novel algorithms-Temporal Binary Compression (TBC) and Dynamic Importance Memory Filter (DIMF)-effectively manage and prioritize long-term memory, significantly reducing storage requirements while retaining critical context. Our multimodal emotion detection approach achieves high accuracy, enhancing proactive and empathetic engagement. User evaluations demonstrated increased emotional bonds, improved satisfaction, and statistically significant reductions in loneliness. Users particularly valued Livia's adaptive personality evolution and realistic AR embodiment. Future research directions include expanding gesture and tactile interactions, supporting multi-user experiences, and exploring customized hardware implementations.
Abstract (translated)
孤独和社会隔离带来了重大的情感和健康挑战，促使开发基于技术的解决方案来提供陪伴和支持。本文介绍了一款名为Livia的应用程序，它是一款结合模块化人工智能（AI）代理、多模态情感计算、渐进式记忆压缩以及增强现实（AR）驱动的具身互动的情感感知AR伴侣应用，旨在为用户提供个性化的心理支持。 Livia采用了一个模块化的人工智能架构，其中各个专门负责情绪分析、对话生成、记忆管理和行为编排的代理确保了强大而适应性的交互。该应用程序还采用了两种新颖算法——时间二进制压缩（TBC）和动态重要性记忆过滤器（DIMF），这些算法能够有效地管理和优先处理长期记忆，大大减少了存储需求的同时保留了关键上下文信息。 我们的多模态情绪检测方法实现了高精度，并增强了主动性和共情式的互动。用户评估表明，Livia的应用显著增加了情感联系、提高了满意度以及统计上明显地减轻了孤独感。特别是，用户特别赞赏Livia的适应性个性演变和真实的AR呈现方式。 未来的研究方向包括扩展手势与触觉交互的支持、支持多用户体验以及探索定制化硬件实现。
URL
https://arxiv.org/abs/2509.05298
PDF
-
Touch Speaks, Sound Feels: A Multimodal Approach to Affective and Social Touch from Robots to Humans
2025-08-11 10:45:43Qiaoqiao Ren, Tony BelpaemeAbstract
Affective tactile interaction constitutes a fundamental component of human communication. In natural human-human encounters, touch is seldom experienced in isolation; rather, it is inherently multisensory. Individuals not only perceive the physical sensation of touch but also register the accompanying auditory cues generated through contact. The integration of haptic and auditory information forms a rich and nuanced channel for emotional expression. While extensive research has examined how robots convey emotions through facial expressions and speech, their capacity to communicate social gestures and emotions via touch remains largely underexplored. To address this gap, we developed a multimodal interaction system incorporating a 5*5 grid of 25 vibration motors synchronized with audio playback, enabling robots to deliver combined haptic-audio stimuli. In an experiment involving 32 Chinese participants, ten emotions and six social gestures were presented through vibration, sound, or their combination. Participants rated each stimulus on arousal and valence scales. The results revealed that (1) the combined haptic-audio modality significantly enhanced decoding accuracy compared to single modalities; (2) each individual channel-vibration or sound-effectively supported certain emotions recognition, with distinct advantages depending on the emotional expression; and (3) gestures alone were generally insufficient for conveying clearly distinguishable emotions. These findings underscore the importance of multisensory integration in affective human-robot interaction and highlight the complementary roles of haptic and auditory cues in enhancing emotional communication.
Abstract (translated)
情感触觉互动构成了人类交流的基本组成部分。在自然的人际交往中，触摸很少是孤立体验的；相反，它本质上是一种多感官体验。个体不仅感知到接触带来的物理感觉，还会注意到由此产生的伴随声音线索。触觉和听觉信息的整合形成了一种丰富而细腻的情感表达通道。 虽然已有大量研究探讨了机器人如何通过面部表情和言语传达情感，但它们通过触摸来传递社交手势和情绪的能力仍然很大程度上未被探索。为填补这一空白，我们开发了一个多模态交互系统，该系统包括一个由25个振动电机组成的5x5网格，并与音频回放同步工作，使机器人能够提供结合了触觉和听觉刺激的输出。 在一项涉及32名中国参与者的实验中，十种情感和六种社交手势通过振动、声音或二者的组合呈现出来。参与者对每种刺激根据唤醒度和效价量表进行了评分。结果表明： 1. 与单一模态相比，触觉-听觉多模态结合显著提高了解码准确性。 2. 每个单独的通道——振动或声音——有效地支持了某些情感识别，并且不同的情感表达具有各自的优势。 3. 单独的手势通常不足以传达清晰可辨识的情绪。 这些发现强调了在情感人机交互中多感官整合的重要性，同时也突显了触觉和听觉线索在增强情绪交流中的互补作用。
URL
https://arxiv.org/abs/2508.07839
PDF
-
Evaluation of a Sign Language Avatar on Comprehensibility, User Experience & Acceptability
2025-08-07 13:06:42Fenya Wasserroth, Eleftherios Avramidis, Vera Czehmann, Tanja Kojic, Fabrizio Nunnari, Sebastian M\"ollerAbstract
This paper presents an investigation into the impact of adding adjustment features to an existing sign language (SL) avatar on a Microsoft Hololens 2 device. Through a detailed analysis of interactions of expert German Sign Language (DGS) users with both adjustable and non-adjustable avatars in a specific use case, this study identifies the key factors influencing the comprehensibility, the user experience (UX), and the acceptability of such a system. Despite user preference for adjustable settings, no significant improvements in UX or comprehensibility were observed, which remained at low levels, amid missing SL elements (mouthings and facial expressions) and implementation issues (indistinct hand shapes, lack of feedback and menu positioning). Hedonic quality was rated higher than pragmatic quality, indicating that users found the system more emotionally or aesthetically pleasing than functionally useful. Stress levels were higher for the adjustable avatar, reflecting lower performance, greater effort and more frustration. Additionally, concerns were raised about whether the Hololens adjustment gestures are intuitive and easy to familiarise oneself with. While acceptability of the concept of adjustability was generally positive, it was strongly dependent on usability and animation quality. This study highlights that personalisation alone is insufficient, and that SL avatars must be comprehensible by default. Key recommendations include enhancing mouthing and facial animation, improving interaction interfaces, and applying participatory design.
Abstract (translated)
本文探讨了在微软Hololens 2设备上为现有的手语（SL）化身添加调整功能的影响。通过详细分析精通德国手语（DGS）的专家用户与可调和不可调化身之间的互动，研究确定了影响此类系统清晰度、用户体验(UX) 和接受度的关键因素。尽管用户偏爱可调设置，但在缺失的手语元素（口型动作和面部表情）以及实现问题（不明确的手势形状、缺乏反馈及菜单定位）的影响下，并未观察到UX或清晰度的显著提升，这两者仍处于较低水平。在愉悦质量方面评分高于实用质量表明用户认为系统更具情感吸引力而非功能性有用。对于可调化身而言，压力水平更高，反映出性能较差、努力更大且更令人沮丧的情况。此外，还提出了关于Hololens调整手势是否直观易懂的疑问。尽管对可调节性的概念普遍持积极态度，但这种接受度强烈依赖于可用性和动画质量。 该研究强调单靠个性化是不够的，手语化身必须默认具备清晰性。关键建议包括增强口型动作和面部动画、改善交互界面以及应用参与式设计。
URL
https://arxiv.org/abs/2508.05358
PDF
-
Improving Tactile Gesture Recognition with Optical Flow
2025-08-06 11:33:21Shaohong Zhong, Alessandro Albini, Giammarco Caroleo, Giorgio Cannata, Perla MaiolinoAbstract
Tactile gesture recognition systems play a crucial role in Human-Robot Interaction (HRI) by enabling intuitive communication between humans and robots. The literature mainly addresses this problem by applying machine learning techniques to classify sequences of tactile images encoding the pressure distribution generated when executing the gestures. However, some gestures can be hard to differentiate based on the information provided by tactile images alone. In this paper, we present a simple yet effective way to improve the accuracy of a gesture recognition classifier. Our approach focuses solely on processing the tactile images used as input by the classifier. In particular, we propose to explicitly highlight the dynamics of the contact in the tactile image by computing the dense optical flow. This additional information makes it easier to distinguish between gestures that produce similar tactile images but exhibit different contact dynamics. We validate the proposed approach in a tactile gesture recognition task, showing that a classifier trained on tactile images augmented with optical flow information achieved a 9% improvement in gesture classification accuracy compared to one trained on standard tactile images.
Abstract (translated)
触觉手势识别系统在人机交互（HRI）中扮演着至关重要的角色，通过实现人类与机器人之间的直观交流来发挥作用。现有文献主要通过应用机器学习技术来分类编码手势执行过程中产生的压力分布的触觉图像序列，以此解决这一问题。然而，仅凭触觉图像是难以区分某些手势的。在这篇论文中，我们提出了一种简单而有效的方法来提高手势识别分类器的准确性。我们的方法专注于处理分类器使用的输入触觉图像。具体来说，我们建议通过计算密集光流来明确突出触觉图像中的接触动态，这将有助于区分那些虽然生成相似触觉图但表现出不同接触动态的手势。我们在一个触觉手势识别任务中验证了所提出的方法，并展示了基于光学流量信息增强的触觉图像训练得到的分类器在手势分类准确性方面相较于仅使用标准触觉图像训练提高了9%。
URL
https://arxiv.org/abs/2508.04338
PDF
-
Motion is the Choreographer: Learning Latent Pose Dynamics for Seamless Sign Language Generation
2025-08-06 03:23:10Jiayi He, Xu Wang, Shengeng Tang, Yaxiong Wang, Lechao Cheng, Dan GuoAbstract
Sign language video generation requires producing natural signing motions with realistic appearances under precise semantic control, yet faces two critical challenges: excessive signer-specific data requirements and poor generalization. We propose a new paradigm for sign language video generation that decouples motion semantics from signer identity through a two-phase synthesis framework. First, we construct a signer-independent multimodal motion lexicon, where each gloss is stored as identity-agnostic pose, gesture, and 3D mesh sequences, requiring only one recording per sign. This compact representation enables our second key innovation: a discrete-to-continuous motion synthesis stage that transforms retrieved gloss sequences into temporally coherent motion trajectories, followed by identity-aware neural rendering to produce photorealistic videos of arbitrary signers. Unlike prior work constrained by signer-specific datasets, our method treats motion as a first-class citizen: the learned latent pose dynamics serve as a portable "choreography layer" that can be visually realized through different human appearances. Extensive experiments demonstrate that disentangling motion from identity is not just viable but advantageous - enabling both high-quality synthesis and unprecedented flexibility in signer personalization.
Abstract (translated)
手语视频生成需要在精确的语义控制下产生自然的手势动作和逼真的外观，但面临着两个关键挑战：对手势特定数据的需求过多以及泛化能力差。我们提出了一种新的手语视频生成范式，通过一个两阶段合成框架将手势语义与执行者身份解耦。 首先，我们构建了一个与执行者无关的多模态动作词典，在其中每个词（gloss）都以无身份标识的姿态、手势和3D网格序列的形式存储。这种表示方式只需要为每个手语记录一次数据即可。这使得我们的第二个关键创新成为可能：从检索到的手势序列中进行离散到连续的动作合成阶段，随后通过面向执行者外观的神经渲染生成任意执行者的逼真视频。 不同于之前受限于特定执行者数据集的方法，我们的方法将动作视为重要组成部分：“学习到的姿态动力学”可以作为便携式的“编舞层”，可以在不同的人类外表中视觉实现。广泛的实验表明，分离手势和身份不仅可行而且具有优势——这使得高质量的合成与前所未有的个性化成为可能。
URL
https://arxiv.org/abs/2508.04049
PDF
-
Theatre in the Loop: A Rehearsal-Based, Collaborative Workflow for Expressive Robotic Behaviours
2025-08-05 14:42:06Pavlos Panagiotidis, Victor Zhi Heung Ngo, Sean Myatt, Roma Patel, Rachel Ramchurn, Alan Chamberlain, Ayse KucukyilmazAbstract
In this paper, we propose theatre-in-the-loop, a framework for developing expressive robot behaviours tailored to artistic performance through a director-guided puppeteering workflow. Leveraging theatrical methods, we use narrative objectives to direct a puppeteer in generating improvised robotic gestures that convey specific emotions. These improvisations are captured and curated to build a dataset of reusable movement templates for standalone playback in future autonomous performances. Initial trials demonstrate the feasibility of this approach, illustrating how the workflow enables precise sculpting of robotic gestures into coherent emotional arcs while revealing challenges posed by the robot's mechanical constraints. We argue that this practice-led framework provides a model for interdisciplinary teams creating socially expressive robot behaviours, contributing to (1) theatre as an interactive training ground for human-robot interaction and (2) co-creation methodologies between humans and machines.
Abstract (translated)
在这篇论文中，我们提出了“theatre-in-the-loop”（剧场闭环），这是一种通过导演指导的木偶操作工作流程来开发针对艺术表演的富有表现力的机器人行为的框架。利用戏剧方法，我们使用叙事目标引导木偶师生成传达特定情绪的即兴机器人手势。这些即兴创作被捕捉并整理成一个可重复使用的运动模板数据集，用于未来自主表演中的独立播放。初步试验展示了这种方法的可行性，说明了该工作流程如何能够精确地将机器人的动作塑造成连贯的情感弧线，并揭示了机器人机械限制所带来的挑战。我们主张这种以实践为导向的框架为跨学科团队创建具有社会表现力的机器人行为提供了一个模型，从而贡献于（1）作为人机交互互动训练场地的戏剧以及（2）人类与机器之间的协同创作方法论。
URL
https://arxiv.org/abs/2508.03514
PDF
-
Live Demonstration: Neuromorphic Radar for Gesture Recognition
2025-08-05 11:06:22Satyapreet Singh Yadav, Chandra Sekhar Seelamantula, Chetan Singh ThakurAbstract
We present a neuromorphic radar framework for real-time, low-power hand gesture recognition (HGR) using an event-driven architecture inspired by biological sensing. Our system comprises a 24 GHz Doppler radar front-end and a custom neuromorphic sampler that converts intermediate-frequency (IF) signals into sparse spike-based representations via asynchronous sigma-delta encoding. These events are directly processed by a lightweight neural network deployed on a Cortex-M0 microcontroller, enabling low-latency inference without requiring spectrogram reconstruction. Unlike conventional radar HGR pipelines that continuously sample and process data, our architecture activates only when meaningful motion is detected, significantly reducing memory, power, and computation overhead. Evaluated on a dataset of five gestures collected from seven users, our system achieves > 85% real-time accuracy. To the best of our knowledge, this is the first work that employs bio-inspired asynchronous sigma-delta encoding and an event-driven processing framework for radar-based HGR.
Abstract (translated)
我们提出了一种基于生物感知启发的事件驱动架构，用于实时、低功耗的手势识别（HGR）的神经形态雷达框架。我们的系统包括一个24 GHz多普勒雷达前端和一个定制的神经形态采样器，该采样器通过异步Sigma-Delta编码将中频(IF)信号转换为稀疏的基于脉冲(Spike)表示形式。这些事件由部署在Cortex-M0微控制器上的轻量级神经网络直接处理，从而能够在不进行光谱图重建的情况下实现低延迟推理。与传统的雷达HGR流水线连续采样和处理数据不同，我们的架构仅在检测到有意义的运动时才激活，这大大减少了内存、功耗和计算开销。 在一个包含七名用户采集五种手势的数据集上评估后，我们的系统实现了超过85%的实时准确率。据我们所知，这是第一个采用生物启发式异步Sigma-Delta编码和事件驱动处理框架用于雷达基手势识别的工作。
URL
https://arxiv.org/abs/2508.03324
PDF
-
Set the Stage: Enabling Storytelling with Multiple Robots through Roleplaying Metaphors
2025-08-03 12:29:43Tyrone Justin Sta Maria, Faith Griffin, Jordan Aiko DejaAbstract
Gestures are an expressive input modality for controlling multiple robots, but their use is often limited by rigid mappings and recognition constraints. To move beyond these limitations, we propose roleplaying metaphors as a scaffold for designing richer interactions. By introducing three roles: Director, Puppeteer, and Wizard, we demonstrate how narrative framing can guide the creation of diverse gesture sets and interaction styles. These roles enable a variety of scenarios, showing how roleplay can unlock new possibilities for multi-robot systems. Our approach emphasizes creativity, expressiveness, and intuitiveness as key elements for future human-robot interaction design.
Abstract (translated)
手势是一种用于控制多台机器人的表达性输入方式，但其使用通常受到固定映射和识别限制的约束。为了超越这些局限，我们提出采用角色扮演隐喻作为设计更丰富交互体验的基础框架。通过引入三个角色：导演、木偶师和巫师，我们展示了叙事构架如何指导多样手势集及互动风格的设计与创造。这些角色能够支持多种应用场景，并展示出角色扮演如何为多机器人系统解锁新的可能性。我们的方法强调创造性、表达性和直观性作为未来人机交互设计的关键要素。
URL
https://arxiv.org/abs/2508.01736
PDF
-
SPARTA: Advancing Sparse Attention in Spiking Neural Networks via Spike-Timing-Based Prioritization
2025-08-03 08:11:33Minsuk Jang, Changick KimAbstract
Current Spiking Neural Networks (SNNs) underutilize the temporal dynamics inherent in spike-based processing, relying primarily on rate coding while overlooking precise timing information that provides rich computational cues. We propose SPARTA (Spiking Priority Attention with Resource-Adaptive Temporal Allocation), a framework that leverages heterogeneous neuron dynamics and spike-timing information to enable efficient sparse attention. SPARTA prioritizes tokens based on temporal cues, including firing patterns, spike timing, and inter-spike intervals, achieving 65.4% sparsity through competitive gating. By selecting only the most salient tokens, SPARTA reduces attention complexity from O(N^2) to O(K^2) with k << n, while maintaining high accuracy. Our method achieves state-of-the-art performance on DVS-Gesture (98.78%) and competitive results on CIFAR10-DVS (83.06%) and CIFAR-10 (95.3%), demonstrating that exploiting spike timing dynamics improves both computational efficiency and accuracy.
Abstract (translated)
当前的脉冲神经网络（SNNs）未能充分利用基于脉冲处理中固有的时间动态特性，主要依赖于频率编码，而忽略了提供丰富计算线索的确切时间信息。我们提出了SPARTA（具有资源自适应时序分配的脉冲优先注意机制），这是一个框架，它利用异构神经元动力学和脉冲时间信息来实现高效的稀疏注意力。SPARTA 通过包括放电模式、脉冲时间和脉冲间间隔的时间线索对令牌进行优先级排序，并通过竞争性门控实现了65.4%的稀疏度。通过仅选择最重要的令牌，SPARTA 将注意力复杂度从 O(N^2) 减少到 O(K^2)，其中 k << n，同时保持高精度。 我们的方法在 DVS-Gesture 数据集上达到了 98.78% 的最佳性能，并且在 CIFAR10-DVS（83.06%）和 CIFAR-10（95.3%）数据集上取得了有竞争力的结果。这表明利用脉冲时间动力学可以提高计算效率并改善准确性。
URL
https://arxiv.org/abs/2508.01646
PDF
-
ParaRevSNN: A Parallel Reversible Spiking Neural Network for Efficient Training and Inference
2025-08-02 06:40:59Changqing Xu, Guoqing Sun, Yi Liu, Xinfang Liao, Yintang YangAbstract
Reversible Spiking Neural Networks (RevSNNs) enable memory-efficient training by reconstructing forward activations during backpropagation, but suffer from high latency due to strictly sequential computation. To overcome this limitation, we propose ParaRevSNN, a parallel reversible SNN architecture that decouples sequential dependencies between reversible blocks while preserving reversibility. This design enables inter-block parallelism, significantly accelerating training and inference while retaining the memory-saving benefits of reversibility. Experiments on CIFAR10, CIFAR100, CIFAR10-DVS, and DVS128 Gesture demonstrate that ParaRevSNN matches or exceeds the accuracy of standard RevSNNs, while reducing training time by up to 35.2\% and inference time to 18.15\%, making it well-suited for deployment in resource-constrained scenarios.
Abstract (translated)
可逆脉冲神经网络（RevSNN）通过在反向传播过程中重构前向激活来实现高效的内存训练，但因其严格遵循的顺序计算导致了较高的延迟。为克服这一限制，我们提出了ParaRevSNN，这是一种并行的可逆SNN架构，在解耦各个可逆块之间的顺序依赖性的同时保留了可逆性。这种设计使跨块并行化成为可能，显著加速了训练和推理过程，同时保持了可逆性的内存节约优势。在CIFAR10、CIFAR100、CIFAR10-DVS以及DVS128手势数据集上的实验表明，ParaRevSNN的精度与标准RevSNN相当或更优，并且将训练时间减少了高达35.2%，推理时间缩短至18.15%，使其非常适合在资源受限场景中部署。
URL
https://arxiv.org/abs/2508.01223
PDF
-
Is It Really You? Exploring Biometric Verification Scenarios in Photorealistic Talking-Head Avatar Videos
2025-08-01 16:23:27Laura Pedrouzo-Rodriguez, Pedro Delgado-DeRobles, Luis F. Gomez, Ruben Tolosana, Ruben Vera-Rodriguez, Aythami Morales, Julian FierrezAbstract
Photorealistic talking-head avatars are becoming increasingly common in virtual meetings, gaming, and social platforms. These avatars allow for more immersive communication, but they also introduce serious security risks. One emerging threat is impersonation: an attacker can steal a user's avatar-preserving their appearance and voice-making it nearly impossible to detect its fraudulent usage by sight or sound alone. In this paper, we explore the challenge of biometric verification in such avatar-mediated scenarios. Our main question is whether an individual's facial motion patterns can serve as reliable behavioral biometrics to verify their identity when the avatar's visual appearance is a facsimile of its owner. To answer this question, we introduce a new dataset of realistic avatar videos created using a state-of-the-art one-shot avatar generation model, GAGAvatar, with genuine and impostor avatar videos. We also propose a lightweight, explainable spatio-temporal Graph Convolutional Network architecture with temporal attention pooling, that uses only facial landmarks to model dynamic facial gestures. Experimental results demonstrate that facial motion cues enable meaningful identity verification with AUC values approaching 80%. The proposed benchmark and biometric system are available for the research community in order to bring attention to the urgent need for more advanced behavioral biometric defenses in avatar-based communication systems.
Abstract (translated)
逼真的说话头像在虚拟会议、游戏和社交平台中越来越常见。这些头像能够促进更沉浸式的交流，但同时也带来了严重的安全风险。其中一项新兴威胁是冒充：攻击者可以窃取用户的头像——保留其外观和声音——使得仅凭视觉或听觉难以辨别其欺诈性使用。在这篇论文中，我们探讨了在这种以头像为媒介的情境下进行生物特征验证的挑战。我们的主要问题是，个人的面部运动模式是否能够作为可靠的生物行为标志来验证身份，即使当头像的外观与其所有者完全相同。为了回答这个问题，我们引入了一个新的数据集，该数据集包含使用最先进的单次生成模型GAGAvatar创建的真实和冒充者的逼真头像视频。此外，我们还提出了一种轻量级且可解释的空间-时间图卷积网络架构，并结合了时序注意力池化机制，仅利用面部地标来建模动态面部手势。实验结果表明，面部运动线索能够实现有意义的身份验证，其AUC值接近80%。所提出的基准和生物特征系统已向研究社区开放，旨在强调在基于头像的通信系统中需要更先进的行为生物识别防御措施。
URL
https://arxiv.org/abs/2508.00748
PDF
