Gesture
-
Towards Context-Aware Human-like Pointing Gestures with RL Motion Imitation
2025-09-16 09:30:42Anna Deichler, Siyang Wang, Simon Alexanderson, Jonas BeskowAbstract
Pointing is a key mode of interaction with robots, yet most prior work has focused on recognition rather than generation. We present a motion capture dataset of human pointing gestures covering diverse styles, handedness, and spatial targets. Using reinforcement learning with motion imitation, we train policies that reproduce human-like pointing while maximizing precision. Results show our approach enables context-aware pointing behaviors in simulation, balancing task performance with natural dynamics.
Abstract (translated)
指向是与机器人互动的关键方式，然而大多数先前的研究主要集中在识别而非生成上。我们提出了一套人类指向手势的动作捕捉数据集，涵盖了多样化的风格、左右手使用偏好及空间目标。通过结合运动模仿的强化学习方法，我们训练出了能够再现类人指向动作并最大化精度的策略。实验结果显示，我们的方法能够在模拟环境中实现情境感知的指向行为，在保持任务性能的同时兼顾自然动态表现。
URL
https://arxiv.org/abs/2509.12880
PDF
-
A Novel Skill Modeling Approach: Integrating Vergnaud's Scheme with Cognitive Architectures
2025-09-16 09:09:15Antoine L\'enat (Cetim, LS2N, LS2N - \'equipe RoMas, LS2N - \'equipe PACCE), Olivier Cheminat (Cetim), Damien Chablat (LS2N - \'equipe RoMas, LS2N), Camilo Charron (LS2N, UR2, LS2N - \'equipe PACCE)Abstract
Human-machine interaction is increasingly important in industry, and this trend will only intensify with the rise of Industry 5.0. Human operators have skills that need to be adapted when using machines to achieve the best results. It is crucial to highlight the operator's skills and understand how they use and adapt them [18]. A rigorous description of these skills is necessary to compare performance with and without robot assistance. Predicate logic, used by Vergnaud within Piaget's scheme concept, offers a promising approach. However, this theory doesn't account for cognitive system constraints, such as the timing of actions, the limitation of cognitive resources, the parallelization of tasks, or the activation of automatic gestures contrary to optimal knowledge. Integrating these constraints is essential for representing agent skills understanding skill transfer between biological and mechanical structures. Cognitive architectures models [2] address these needs by describing cognitive structure and can be combined with the scheme for mutual benefit. Welding provides a relevant case study, as it highlights the challenges faced by operators, even highly skilled ones. Welding's complexity stems from the need for constant skill adaptation to variable parameters like part position and process. This adaptation is crucial, as weld quality, a key factor, is only assessed afterward via destructive testing. Thus, the welder is confronted with a complex perception-decision-action cycle, where the evaluation of the impact of his actions is delayed and where errors are definitive. This dynamic underscores the importance of understanding and modeling the skills of operators.
Abstract (translated)
人机交互在工业领域日益重要，随着Industry 5.0的到来，这一趋势只会加剧。人类操作员具备的技能需要适应机器使用以达到最佳效果。突出操作者的技能并理解他们如何运用和调整这些技能至关重要[18]。为了比较有无机器人协助时的表现，详细描述这些技能是必要的。维尔格诺在皮亚杰方案概念中使用的谓词逻辑提供了一个很有前途的方法。然而，该理论没有考虑认知系统的限制，如动作时间、认知资源的局限性、任务并行化以及与最优知识相悖的手动习惯被激活的情况。将这些约束整合进来对于表示代理技能和理解生物结构向机械结构的技能转移至关重要。认知架构模型[2]通过描述认知结构来满足这一需求，并且可以结合方案以实现互利。 焊接提供了一个相关的案例研究，它突显了操作者面临的挑战，即使是高度熟练的操作者也不例外。焊接复杂性的根源在于需要不断调整技能以适应变化的因素，如零件位置和工艺过程。这种调整至关重要，因为焊缝质量——一个关键因素——只能通过破坏性测试来评估。因此，焊工面临着复杂的感知-决策-行动循环，在这个过程中，他们对自身行为影响的评价被延迟了，并且错误是不可逆转的。这一动态强调了理解并建模操作员技能的重要性。
URL
https://arxiv.org/abs/2509.12851
PDF
-
Gesture Evaluation in Virtual Reality
Abstract
Gestures are central to human communication, enriching interactions through non-verbal expression. Virtual avatars increasingly use AI-generated gestures to enhance life-likeness, yet evaluations have largely been confined to 2D. Virtual Reality (VR) provides an immersive alternative that may affect how gestures are perceived. This paper presents a comparative evaluation of computer-generated gestures in VR and 2D, examining three models from the 2023 GENEA Challenge. Results show that gestures viewed in VR were rated slightly higher on average, with the strongest effect observed for motion-capture "true movement." While model rankings remained consistent across settings, VR influenced participants' overall perception and offered unique benefits over traditional 2D evaluation.
Abstract (translated)
手势在人类交流中至关重要，通过非言语表达丰富了互动体验。虚拟化身越来越多地使用AI生成的手势来增强其逼真度，但评估工作大多局限于二维空间内进行。虚拟现实（VR）提供了一种沉浸式的替代方案，可能会影响人们对手势的感知方式。本文呈现了一个关于计算机生成的手势在VR和2D环境中的比较性评价研究，考察了来自2023年GENEA挑战赛的三种模型。结果显示，在VR中观看的手势总体评分略高一些，尤其是对于基于动作捕捉技术的“真实运动”效果最为明显。虽然不同模型在各个设置下的排名保持一致，但VR环境对参与者整体感知的影响独特，并且提供了传统二维评估所不具备的优势。
URL
https://arxiv.org/abs/2509.12816
PDF
-
Learning to Generate Pointing Gestures in Situated Embodied Conversational Agents
2025-09-15 23:15:15Anna Deichler, Siyang Wang, Simon Alexanderson, Jonas BeskowAbstract
One of the main goals of robotics and intelligent agent research is to enable natural communication with humans in physically situated settings. While recent work has focused on verbal modes such as language and speech, non-verbal communication is crucial for flexible interaction. We present a framework for generating pointing gestures in embodied agents by combining imitation and reinforcement learning. Using a small motion capture dataset, our method learns a motor control policy that produces physically valid, naturalistic gestures with high referential accuracy. We evaluate the approach against supervised learning and retrieval baselines in both objective metrics and a virtual reality referential game with human users. Results show that our system achieves higher naturalness and accuracy than state-of-the-art supervised models, highlighting the promise of imitation-RL for communicative gesture generation and its potential application to robots.
Abstract (translated)
机器人和智能代理研究的主要目标之一是实现与人类在物理环境中的自然沟通。虽然最近的工作主要集中在语言和语音等口头模式上，但非言语交流对于灵活互动至关重要。我们提出了一种结合模仿学习和强化学习的框架，用于生成具身化代理人的指向手势。通过使用一个小规模的动作捕捉数据集，我们的方法能够学习一种运动控制策略，该策略可以产生物理上有效且自然的手势，并具有很高的指代准确性。我们在客观指标以及与人类用户进行虚拟现实指代游戏中的基准测试中评估了这种方法相对于监督学习和检索基线的表现。实验结果显示，我们系统在自然度和准确性方面均优于最新的监督模型，这凸显了模仿-强化学习在生成沟通手势方面的潜力及其在未来机器人应用中的潜在价值。
URL
https://arxiv.org/abs/2509.12507
PDF
-
Gesture-Based Robot Control Integrating Mm-wave Radar and Behavior Trees
2025-09-15 14:52:22Yuqing Song, Cesare Tonola, Stefano Savazzi, Sanaz Kianoush, Nicola Pedrocchi, Stephan SiggAbstract
As robots become increasingly prevalent in both homes and industrial settings, the demand for intuitive and efficient human-machine interaction continues to rise. Gesture recognition offers an intuitive control method that does not require physical contact with devices and can be implemented using various sensing technologies. Wireless solutions are particularly flexible and minimally invasive. While camera-based vision systems are commonly used, they often raise privacy concerns and can struggle in complex or poorly lit environments. In contrast, radar sensing preserves privacy, is robust to occlusions and lighting, and provides rich spatial data such as distance, relative velocity, and angle. We present a gesture-controlled robotic arm using mm-wave radar for reliable, contactless motion recognition. Nine gestures are recognized and mapped to real-time commands with precision. Case studies are conducted to demonstrate the system practicality, performance and reliability for gesture-based robotic manipulation. Unlike prior work that treats gesture recognition and robotic control separately, our system unifies both into a real-time pipeline for seamless, contactless human-robot interaction.
Abstract (translated)
随着机器人在家庭和工业环境中的应用日益广泛，对直观高效的人机交互需求也在不断增加。手势识别提供了一种无需直接接触设备的自然控制方法，并可通过多种传感技术实现。无线解决方案尤其灵活且侵入性小。尽管基于摄像头的视觉系统常常被采用，但它们经常引发隐私问题，并且在复杂或光线不足的环境中表现不佳。相比之下，雷达感应能保护用户隐私，对遮挡和照明条件具有鲁棒性，还能提供丰富的空间数据，如距离、相对速度和角度等信息。 本文介绍了一种使用毫米波雷达的手势控制机械臂系统，可实现可靠且无接触的动作识别。该系统能够精确地识别并映射九种手势到实时命令中。通过案例研究展示了基于手势的机器人操作系统的实用性和性能可靠性。不同于以往将手势识别和机器人控制视为独立过程的研究，我们的系统首次统一这两者，构建了一个实时管道，实现了无缝、无接触的人机交互。
URL
https://arxiv.org/abs/2509.12008
PDF
-
SignClip: Leveraging Mouthing Cues for Sign Language Translation by Multimodal Contrastive Fusion
2025-09-12 14:08:06Wenfang Wu, Tingting Yuan, Yupeng Li, Daling Wang, Xiaoming FuAbstract
Sign language translation (SLT) aims to translate natural language from sign language videos, serving as a vital bridge for inclusive communication. While recent advances leverage powerful visual backbones and large language models, most approaches mainly focus on manual signals (hand gestures) and tend to overlook non-manual cues like mouthing. In fact, mouthing conveys essential linguistic information in sign languages and plays a crucial role in disambiguating visually similar signs. In this paper, we propose SignClip, a novel framework to improve the accuracy of sign language translation. It fuses manual and non-manual cues, specifically spatial gesture and lip movement features. Besides, SignClip introduces a hierarchical contrastive learning framework with multi-level alignment objectives, ensuring semantic consistency across sign-lip and visual-text modalities. Extensive experiments on two benchmark datasets, PHOENIX14T and How2Sign, demonstrate the superiority of our approach. For example, on PHOENIX14T, in the Gloss-free setting, SignClip surpasses the previous state-of-the-art model SpaMo, improving BLEU-4 from 24.32 to 24.71, and ROUGE from 46.57 to 48.38.
Abstract (translated)
手语翻译（SLT）旨在将视频中的手势语言转化为自然语言，作为包容性沟通的重要桥梁。尽管最近的研究进展利用了强大的视觉骨干网络和大型语言模型，但大多数方法主要关注手动信号（如手部动作），而往往忽略了非手动线索，例如口型。实际上，在手语中，口型传达了重要的语言信息，并且在区分视觉相似的手势时起着关键作用。 本文提出了一种新的框架——SignClip，旨在提高手语翻译的准确性。该框架融合了手工信号和非手工线索，特别是空间手势与唇部运动特征。此外，SignClip 引入了一个分层对比学习框架，并采用了多级对齐目标，确保在符号-唇形和视觉-文本模式之间保持语义一致性。 在两个基准数据集 PHOENIX14T 和 How2Sign 上进行的广泛实验表明了我们方法的优势。例如，在 PHOENIX14T 数据集中，无 Gloss（注：Gloss 指的是手语翻译中的一种简化表示形式）设置下，SignClip 超越了之前的最新模型 SpaMo，将 BLEU-4 评分从 24.32 提高到了 24.71，并将 ROUGE 评分从 46.57 提升至 48.38。
URL
https://arxiv.org/abs/2509.10266
PDF
-
Pragmatic Frames Evoked by Gestures: A FrameNet Brasil Approach to Multimodality in Turn Organization
2025-09-11 19:14:57Helen de Andrade Abreu, Tiago Timponi Torrent, Ely Edison da Silva MatosAbstract
This paper proposes a framework for modeling multimodal conversational turn organization via the proposition of correlations between language and interactive gestures, based on analysis as to how pragmatic frames are conceptualized and evoked by communicators. As a means to provide evidence for the analysis, we developed an annotation methodology to enrich a multimodal dataset (annotated for semantic frames) with pragmatic frames modeling conversational turn organization. Although conversational turn organization has been studied by researchers from diverse fields, the specific strategies, especially gestures used by communicators, had not yet been encoded in a dataset that can be used for machine learning. To fill this gap, we enriched the Frame2 dataset with annotations of gestures used for turn organization. The Frame2 dataset features 10 episodes from the Brazilian TV series Pedro Pelo Mundo annotated for semantic frames evoked in both video and text. This dataset allowed us to closely observe how communicators use interactive gestures outside a laboratory, in settings, to our knowledge, not previously recorded in related literature. Our results have confirmed that communicators involved in face-to-face conversation make use of gestures as a tool for passing, taking and keeping conversational turns, and also revealed variations of some gestures that had not been documented before. We propose that the use of these gestures arises from the conceptualization of pragmatic frames, involving mental spaces, blending and conceptual metaphors. In addition, our data demonstrate that the annotation of pragmatic frames contributes to a deeper understanding of human cognition and language.
Abstract (translated)
本文提出了一种通过探讨语言与互动手势之间的相关性来建模多模式对话轮次组织的框架，该框架基于对实用语境如何由交际者构想和唤起的分析。为了为这一分析提供证据，我们开发了一种标注方法，以增强一个已用语义框架标注过的多模态数据集，并对其进行对话轮次组织的实用性框架建模。尽管来自不同领域的研究人员已经研究了对话轮次组织，但具体策略（尤其是交际者使用的手势）尚未在可用于机器学习的数据集中编码。为填补这一空白，我们通过添加用于转话语组织的手势标注来丰富Frame2数据集。Frame2数据集包含了10个取自巴西电视剧《Pedro Pelo Mundo》的片段，并对视频和文本中唤起的语义框架进行了标注。该数据集使我们可以仔细观察交际者在实验室外的情境下如何使用互动手势，据我们所知，这些情境之前并未被相关文献记录。 我们的研究结果证实了面对面对话中的交际者利用手势作为传递、获取和保持对话轮次的工具，并揭示了一些以前未被记录的手势的变化形式。我们认为这些手势的使用源于实用性框架的概念化，包括心理空间、融合以及概念隐喻。此外，我们的数据还表明，对实用语境进行标注有助于更深入地理解人类认知与语言的关系。
URL
https://arxiv.org/abs/2509.09804
PDF
-
Skeleton-based sign language recognition using a dual-stream spatio-temporal dynamic graph convolutional network
Abstract
Isolated Sign Language Recognition (ISLR) is challenged by gestures that are morphologically similar yet semantically distinct, a problem rooted in the complex interplay between hand shape and motion trajectory. Existing methods, often relying on a single reference frame, struggle to resolve this geometric ambiguity. This paper introduces Dual-SignLanguageNet (DSLNet), a dual-reference, dual-stream architecture that decouples and models gesture morphology and trajectory in separate, complementary coordinate systems. Our approach utilizes a wrist-centric frame for view-invariant shape analysis and a facial-centric frame for context-aware trajectory modeling. These streams are processed by specialized networks-a topology-aware graph convolution for shape and a Finsler geometry-based encoder for trajectory-and are integrated via a geometry-driven optimal transport fusion mechanism. DSLNet sets a new state-of-the-art, achieving 93.70%, 89.97% and 99.79% accuracy on the challenging WLASL-100, WLASL-300 and LSA64 datasets, respectively, with significantly fewer parameters than competing models.
Abstract (translated)
孤立手语识别（ISLR）面临着形态相似但语义不同的手势的挑战，这个问题源于手形和运动轨迹之间的复杂相互作用。现有的方法通常依赖于单一参考框架，难以解决这种几何歧义问题。本文介绍了一种双参照、双流架构——Dual-SignLanguageNet (DSLNet)，该架构解耦并分别在互补坐标系统中建模手势形态与轨迹。我们的方法利用以手腕为中心的帧进行视图不变的手形分析，并采用以面部为中心的帧来感知上下文的运动轨迹建模。这些数据流由专门设计的网络处理——拓扑感知图卷积用于手形，基于芬斯勒几何的编码器用于轨迹，并通过一种几何驱动的最佳传输融合机制集成。DSLNet在具有挑战性的WLASL-100、WLASL-300和LSA64数据集上分别达到了93.70%、89.97%和99.79%的准确率，参数量远少于竞争模型。
URL
https://arxiv.org/abs/2509.08661
PDF
-
EHWGesture -- A dataset for multimodal understanding of clinical gestures
2025-09-09 09:00:03Gianluca Amprimo, Alberto Ancilotto, Alessandro Savino, Fabio Quazzolo, Claudia Ferraris, Gabriella Olmo, Elisabetta Farella, Stefano Di CarloAbstract
Hand gesture understanding is essential for several applications in human-computer interaction, including automatic clinical assessment of hand dexterity. While deep learning has advanced static gesture recognition, dynamic gesture understanding remains challenging due to complex spatiotemporal variations. Moreover, existing datasets often lack multimodal and multi-view diversity, precise ground-truth tracking, and an action quality component embedded within gestures. This paper introduces EHWGesture, a multimodal video dataset for gesture understanding featuring five clinically relevant gestures. It includes over 1,100 recordings (6 hours), captured from 25 healthy subjects using two high-resolution RGB-Depth cameras and an event camera. A motion capture system provides precise ground-truth hand landmark tracking, and all devices are spatially calibrated and synchronized to ensure cross-modal alignment. Moreover, to embed an action quality task within gesture understanding, collected recordings are organized in classes of execution speed that mirror clinical evaluations of hand dexterity. Baseline experiments highlight the dataset's potential for gesture classification, gesture trigger detection, and action quality assessment. Thus, EHWGesture can serve as a comprehensive benchmark for advancing multimodal clinical gesture understanding.
Abstract (translated)
手势理解在人机交互的多个应用中至关重要，包括自动临床评估手部灵巧度。虽然深度学习已经显著提升了静态手势识别的能力，但动态手势的理解仍然面临挑战，因为这些手势包含复杂的时空变化。此外，现有的数据集往往缺乏多模态和多视角多样性、精确的真值追踪以及嵌入手势中的动作质量成分。 本文介绍了一种名为EHWGesture的数据集，它是一个用于手势理解的多模态视频数据集，包含了五个具有临床相关性的手势。该数据集中包含超过1,100个记录（总计6小时），这些记录由25名健康受试者使用两台高分辨率RGB-Depth相机和一台事件相机录制而成。一个运动捕捉系统提供了精确的手部地标追踪真值，并且所有设备均经过空间校准并同步以确保跨模式对齐。此外，为了在手势理解中嵌入动作质量任务，收集的记录按照执行速度类别进行分类，这反映了临床对手部灵巧度评估的方式。 基础实验突显了该数据集在手势分类、手势触发检测和动作质量评估方面的潜力。因此，EHWGesture可以作为推进多模态临床手势理解研究的一个全面基准。
URL
https://arxiv.org/abs/2509.07525
PDF
-
'It was Tragic': Exploring the Impact of a Robot's Shutdown
Abstract
It is well established that people perceive robots as social entities, even when they are not designed for social interaction. We evaluated whether the social interpretation of robotic gestures should also be considered when turning off a robot. In the experiment, participants engaged in a brief preliminary neutral interaction while a robotic arm showed interest in their actions. At the end of the task, participants were asked to turn off the robotic arm under two conditions: (1) a Non-designed condition, where all of the robot's engines were immediately and simultaneously turned off, as robots typically shut down; (2) a Designed condition, where the robot's engines gradually folded inward in a motion resembling "falling asleep." Our findings revealed that all participants anthropomorphized the robot's movement when it was turned off. In the Non-designed condition, most participants interpreted the robot's turn-off movement negatively, as if the robot had "died." In the Designed condition, most participants interpreted it more neutrally, stating that the robot "went to sleep." The robot's turn-off movement also impacted its perception, leading to higher likeability, perceived intelligence, and animacy in the Designed condition. We conclude that the impact of common edge interactions, such as turning off a robot, should be carefully designed while considering people's automatic tendency to perceive robots as social entities.
Abstract (translated)
人们早已认识到，即使机器人并非为社交互动而设计，也会将其视为社会实体。我们评估了在关闭机器人的过程中是否也应该考虑其行为的社会性解释。实验中，参与者与一只表现出对其动作感兴趣的手臂机器人进行了简短的初步中立互动。任务结束时，在两种条件下要求参与者关闭手臂机器人：(1)非设计条件，即所有机器人的引擎立即且同时关机，就像典型的机器人关机方式一样；(2)设计条件，机器人各部依次逐渐向内折叠，其动作看起来像“进入睡眠”。 研究结果表明，当在实验中关闭机器人时，参与者都会对机器人运动进行拟人化。在非设计条件下，大多数参与者将机器人的关机行为解读为负面的，仿佛机器人“死亡”了。而在设计条件下的情况下，多数参与者则将其解读得更为中立，表示机器人只是“入睡”。 此外，机器人的关机方式也影响到了其感知，使得设计条件下的机器人在喜欢程度、感知智能度和生机感方面都得到了更高的评价。 因此，我们得出结论，在考虑人们自动将机器人视为社会实体的趋势时，应当仔细设计诸如关闭机器人等常见互动边缘的交互行为。
URL
https://arxiv.org/abs/2509.06934
PDF
-
InterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarios
2025-09-06 15:36:47Leo Ho, Yinghao Huang, Dafei Qin, Mingyi Shi, Wangpok Tse, Wei Liu, Junichi Yamagishi, Taku KomuraAbstract
We address the problem of accurate capture of interactive behaviors between two people in daily scenarios. Most previous works either only consider one person or solely focus on conversational gestures of two people, assuming the body orientation and/or position of each actor are constant or barely change over each interaction. In contrast, we propose to simultaneously model two people's activities, and target objective-driven, dynamic, and semantically consistent interactions which often span longer duration and cover bigger space. To this end, we capture a new multi-modal dataset dubbed InterAct, which is composed of 241 motion sequences where two people perform a realistic and coherent scenario for one minute or longer over a complete interaction. For each sequence, two actors are assigned different roles and emotion labels, and collaborate to finish one task or conduct a common interaction activity. The audios, body motions, and facial expressions of both persons are captured. InterAct contains diverse and complex motions of individuals and interesting and relatively long-term interaction patterns barely seen before. We also demonstrate a simple yet effective diffusion-based method that estimates interactive face expressions and body motions of two people from speech inputs. Our method regresses the body motions in a hierarchical manner, and we also propose a novel fine-tuning mechanism to improve the lip accuracy of facial expressions. To facilitate further research, the data and code is made available at this https URL .
Abstract (translated)
我们解决了在日常场景中准确捕捉两人互动行为的问题。大多数先前的工作要么只考虑一个人，要么仅关注两个人之间的对话手势，假设每个演员的身体朝向和/或位置保持不变或变化很小。相比之下，我们提出同时建模两人的活动，并针对目标驱动的、动态且语义一致的交互，这些交互通常持续时间较长并覆盖更大的空间范围。为此，我们收集了一个新的多模态数据集，名为InterAct，该数据集包含241个动作序列，在每个序列中两人共同表演一个真实且连贯的情景超过一分钟或更长时间直至完成整个互动过程。对于每一个序列，两名演员被分配不同的角色和情感标签，并合作完成一项任务或者进行一种常见的交互活动。捕捉了两个人的声音、身体动作和面部表情。 InterAct数据集包含个人多样化且复杂的动作以及以前很少见的有趣而相对长期的人际互动模式。我们还演示了一种简单但有效的基于扩散的方法，该方法能够根据语音输入估算出两人的互动面部表情和身体动作。我们的方法以分层的方式回归身体运动，并提出了一种新颖的微调机制来提高面部表情的唇部精度。 为了促进进一步的研究，数据集和代码可在以下链接获得：[此URL]。
URL
https://arxiv.org/abs/2509.05747
PDF
-
Pointing-Guided Target Estimation via Transformer-Based Attention
2025-09-05 11:42:03Luca M\"uller, Hassan Ali, Philipp Allgeuer, Luk\'a\v{s} Gajdo\v{s}ech, Stefan WermterAbstract
Deictic gestures, like pointing, are a fundamental form of non-verbal communication, enabling humans to direct attention to specific objects or locations. This capability is essential in Human-Robot Interaction (HRI), where robots should be able to predict human intent and anticipate appropriate responses. In this work, we propose the Multi-Modality Inter-TransFormer (MM-ITF), a modular architecture to predict objects in a controlled tabletop scenario with the NICOL robot, where humans indicate targets through natural pointing gestures. Leveraging inter-modality attention, MM-ITF maps 2D pointing gestures to object locations, assigns a likelihood score to each, and identifies the most likely target. Our results demonstrate that the method can accurately predict the intended object using monocular RGB data, thus enabling intuitive and accessible human-robot collaboration. To evaluate the performance, we introduce a patch confusion matrix, providing insights into the model's predictions across candidate object locations. Code available at: this https URL.
Abstract (translated)
指示性手势（如指向）是非语言沟通的一种基本形式，能够帮助人类将注意力集中到特定的对象或位置上。这种能力在人机交互（HRI）中至关重要，因为机器人需要具备预测人的意图并作出适当回应的能力。在这项工作中，我们提出了一种模块化架构——多模态互变换器（Multi-Modality Inter-TransFormer, MM-ITF），用于在一个受控的桌面上与NICOL机器人互动的情境下，通过自然的手指指向手势来预测目标对象。利用跨模式注意机制，MM-ITF将2D指向手势映射到物体位置上，并为每个可能的目标分配一个可能性分数，从而确定最有可能的目标。我们的实验结果表明，该方法能够仅使用单目RGB数据准确地预测意图中的对象，从而使人类与机器人的协作更加直观和易于实现。为了评估性能，我们引入了一个补丁混淆矩阵，提供了模型在候选目标位置上预测情况的深入见解。 代码可在以下网址获取：this https URL.
URL
https://arxiv.org/abs/2509.05031
PDF
-
Exploring persuasive Interactions with generative social robots: An experimental framework
2025-09-03 11:42:24Stephan Vonschallen, Larissa Julia Corina Finsler, Theresa Schmiedel, Friederike EysselAbstract
Integrating generative AI such as large language models into social robots has improved their ability to engage in natural, human-like communication. This study presents a method to examine their persuasive capabilities. We designed an experimental framework focused on decision making and tested it in a pilot that varied robot appearance and self-knowledge. Using qualitative analysis, we evaluated interaction quality, persuasion effectiveness, and the robot's communicative strategies. Participants generally experienced the interaction positively, describing the robot as competent, friendly, and supportive, while noting practical limits such as delayed responses and occasional speech-recognition errors. Persuasiveness was highly context dependent and shaped by robot behavior: participants responded well to polite, reasoned suggestions and expressive gestures, but emphasized the need for more personalized, context-aware arguments and clearer social roles. These findings suggest that generative social robots can influence user decisions, but their effectiveness depends on communicative nuance and contextual relevance. We propose refinements to the framework to further study persuasive dynamics between robots and human users.
Abstract (translated)
将生成式人工智能（如大型语言模型）集成到社交机器人中，提高了它们进行自然、人性化的交流的能力。本研究提出了一种方法来考察这些机器人的说服力。我们设计了一个以决策制定为中心的实验框架，并在一个试点项目中测试了这一框架，该试点项目调整了机器人的外观和自我认知情况。通过定性分析，我们评估了交互质量、说服效果以及机器人的沟通策略。 参与者通常对与机器人互动体验感到积极，他们认为机器人既聪明又友好且支持性强，但同时也指出了实际存在的限制，如延迟回应和偶尔出现的语音识别错误。说服力高度依赖于具体情境，并受机器人行为的影响：参与者对于礼貌、有理据的支持以及表达性手势做出了良好反应，但也强调需要更多个性化的、语境相关的论点以及更明确的社会角色定义。 这些发现表明，生成式的社交机器人能够影响用户决策，但其效果取决于沟通技巧的微妙性和与情境的相关性。我们建议对实验框架进行改进，以便进一步研究机器人和人类用户之间说服动态的变化规律。
URL
https://arxiv.org/abs/2509.03231
PDF
-
NSPDI-SNN: An efficient lightweight SNN based on nonlinear synaptic pruning and dendritic integration
2025-08-29 12:22:00Wuque Cai, Hongze Sun, Jiayi He, Qianqian Liao, Yunliang Zang, Duo Chen, Dezhong Yao, Daqing GuoAbstract
Spiking neural networks (SNNs) are artificial neural networks based on simulated biological neurons and have attracted much attention in recent artificial intelligence technology studies. The dendrites in biological neurons have efficient information processing ability and computational power; however, the neurons of SNNs rarely match the complex structure of the dendrites. Inspired by the nonlinear structure and highly sparse properties of neuronal dendrites, in this study, we propose an efficient, lightweight SNN method with nonlinear pruning and dendritic integration (NSPDI-SNN). In this method, we introduce nonlinear dendritic integration (NDI) to improve the representation of the spatiotemporal information of neurons. We implement heterogeneous state transition ratios of dendritic spines and construct a new and flexible nonlinear synaptic pruning (NSP) method to achieve the high sparsity of SNN. We conducted systematic experiments on three benchmark datasets (DVS128 Gesture, CIFAR10-DVS, and CIFAR10) and extended the evaluation to two complex tasks (speech recognition and reinforcement learning-based maze navigation task). Across all tasks, NSPDI-SNN consistently achieved high sparsity with minimal performance degradation. In particular, our method achieved the best experimental results on all three event stream datasets. Further analysis showed that NSPDI significantly improved the efficiency of synaptic information transfer as sparsity increased. In conclusion, our results indicate that the complex structure and nonlinear computation of neuronal dendrites provide a promising approach for developing efficient SNN methods.
Abstract (translated)
脉冲神经网络（SNNs）是基于模拟生物神经元的人工神经网络，在最近的人工智能技术研究中引起了广泛关注。生物神经元中的树突具有高效的信息处理能力和计算能力；然而，目前的SNN神经元很少能匹配到这种复杂的树突结构。受树突非线性结构和高度稀疏特性的启发，我们在本研究中提出了一种高效的、轻量级的方法——带非线性剪枝和树突发射整合（NSPDI-SNN）的脉冲神经网络方法。在此方法中，我们引入了非线性树突发射整合（NDI），以增强脉冲神经元时空信息的表示能力；同时，通过构建异质状态转换比例的树突棘结构，并建立了一种新的、灵活的非线性突触剪枝（NSP）方法，实现了SNN的高度稀疏化。 我们在三个基准数据集（DVS128手势识别、CIFAR10-DVS和CIFAR10）上进行了系统实验，并扩展到了两个复杂任务（语音识别和基于强化学习的迷宫导航任务）。在所有这些任务中，NSPDI-SNN始终能够在保持性能轻微下降的前提下达到高稀疏度。特别是在三个事件流数据集上，我们的方法取得了最佳的实验结果。 进一步分析表明，随着稀疏性的提高，NSP显著提升了突触信息传输效率。综上所述，我们的研究结果显示：神经元树突复杂的结构和非线性计算为开发高效的SNN方法提供了一种有前景的方法。
URL
https://arxiv.org/abs/2508.21566
PDF
-
InfinityHuman: Towards Long-Term Audio-Driven Human
2025-08-27 18:36:30Xiaodi Li, Pan Xie, Yi Ren, Qijun Gan, Chen Zhang, Fangyuan Kong, Xiang Yin, Bingyue Peng, Zehuan YuanAbstract
Audio-driven human animation has attracted wide attention thanks to its practical applications. However, critical challenges remain in generating high-resolution, long-duration videos with consistent appearance and natural hand motions. Existing methods extend videos using overlapping motion frames but suffer from error accumulation, leading to identity drift, color shifts, and scene instability. Additionally, hand movements are poorly modeled, resulting in noticeable distortions and misalignment with the audio. In this work, we propose InfinityHuman, a coarse-to-fine framework that first generates audio-synchronized representations, then progressively refines them into high-resolution, long-duration videos using a pose-guided refiner. Since pose sequences are decoupled from appearance and resist temporal degradation, our pose-guided refiner employs stable poses and the initial frame as a visual anchor to reduce drift and improve lip synchronization. Moreover, to enhance semantic accuracy and gesture realism, we introduce a hand-specific reward mechanism trained with high-quality hand motion data. Experiments on the EMTD and HDTF datasets show that InfinityHuman achieves state-of-the-art performance in video quality, identity preservation, hand accuracy, and lip-sync. Ablation studies further confirm the effectiveness of each module. Code will be made public.
Abstract (translated)
基于音频的人体动画由于其实用性吸引了广泛的关注。然而，生成高分辨率、长时间且外观一致、手部动作自然的视频仍然面临重大挑战。现有方法通过使用重叠运动帧来延长视频，但会遭受错误积累的问题，导致身份漂移、色彩偏移和场景不稳定。此外，手部动作建模不足也会造成明显的失真与音频不同步现象。为此，我们提出了InfinityHuman框架，这是一种从粗到细的生成模型，首先生成与音频同步的表示，然后逐步细化为高分辨率长时间视频，通过姿势引导的精炼器完成这一过程。 由于姿势序列与外观解耦且能够抵抗时序退化，我们的姿势引导式精炼器利用稳定的姿势和初始帧作为视觉锚点来减少漂移并提高唇部同步效果。为了增强语义准确性及手势真实性，我们引入了一种针对手部动作的特定奖励机制，并通过高质量的手部运动数据进行训练。 在EMTD和HDTF数据集上的实验表明，InfinityHuman在视频质量、身份保持、手部精度以及唇部同步方面均达到了当前最佳性能。消融研究进一步证实了每个模块的有效性。代码将公开发布。
URL
https://arxiv.org/abs/2508.20210
PDF
-
Take That for Me: Multimodal Exophora Resolution with Interactive Questioning for Ambiguous Out-of-View Instructions
2025-08-22 07:09:06Akira Oyama, Shoichi Hasegawa, Akira Taniguchi, Yoshinobu Hagiwara, Tadahiro TaniguchiAbstract
Daily life support robots must interpret ambiguous verbal instructions involving demonstratives such as ``Bring me that cup,'' even when objects or users are out of the robot's view. Existing approaches to exophora resolution primarily rely on visual data and thus fail in real-world scenarios where the object or user is not visible. We propose Multimodal Interactive Exophora resolution with user Localization (MIEL), which is a multimodal exophora resolution framework leveraging sound source localization (SSL), semantic mapping, visual-language models (VLMs), and interactive questioning with GPT-4o. Our approach first constructs a semantic map of the environment and estimates candidate objects from a linguistic query with the user's skeletal data. SSL is utilized to orient the robot toward users who are initially outside its visual field, enabling accurate identification of user gestures and pointing directions. When ambiguities remain, the robot proactively interacts with the user, employing GPT-4o to formulate clarifying questions. Experiments in a real-world environment showed results that were approximately 1.3 times better when the user was visible to the robot and 2.0 times better when the user was not visible to the robot, compared to the methods without SSL and interactive questioning. The project website is this https URL.
Abstract (translated)
日常生活中，支持机器人的任务之一是理解含糊的口头指令，例如“把那个杯子给我”，即使物体或用户不在机器人视野范围内。现有解决外指问题的方法主要依赖于视觉数据，在实际场景中当对象或用户不可见时则无法发挥作用。 为此，我们提出了一种多模态交互式外指解析框架（Multimodal Interactive Exophora resolution with user Localization, MIEL），该框架结合了声音源定位（SSL）、语义地图构建、视觉-语言模型（VLMs）以及与用户互动提问的方法。MIEL首先基于环境信息构建一个语义地图，并使用用户的骨骼数据估计指令中提到的潜在对象候选者。利用SSL，机器人可以转向当前不在其视野范围内的用户方向，从而准确识别手势和指向指示。在存在歧义的情况下，机器人主动与用户进行交互，通过GPT-4o生成澄清问题以获得更多信息。 实验结果显示，在真实环境中，当用户处于可见范围内时，MIEL的性能大约是不使用SSL和互动提问方法的1.3倍；而当用户不可见时，这一提升更是达到了2.0倍。项目的官方网站位于此链接：https URL。
URL
https://arxiv.org/abs/2508.16143
PDF
-
Prompting with Sign Parameters for Low-resource Sign Language Instruction Generation
2025-08-22 04:11:28Md Tariquzzaman, Md Farhan Ishmam, Saiyma Sittul Muna, Md Kamrul Hasan, Hasan MahmudAbstract
Sign Language (SL) enables two-way communication for the deaf and hard-of-hearing community, yet many sign languages remain under-resourced in the AI space. Sign Language Instruction Generation (SLIG) produces step-by-step textual instructions that enable non-SL users to imitate and learn SL gestures, promoting two-way interaction. We introduce BdSLIG, the first Bengali SLIG dataset, used to evaluate Vision Language Models (VLMs) (i) on under-resourced SLIG tasks, and (ii) on long-tail visual concepts, as Bengali SL is unlikely to appear in the VLM pre-training data. To enhance zero-shot performance, we introduce Sign Parameter-Infused (SPI) prompting, which integrates standard SL parameters, like hand shape, motion, and orientation, directly into the textual prompts. Subsuming standard sign parameters into the prompt makes the instructions more structured and reproducible than free-form natural text from vanilla prompting. We envision that our work would promote inclusivity and advancement in SL learning systems for the under-resourced communities.
Abstract (translated)
手语（SL）为聋人和听力障碍社区提供了双向交流的方式，然而许多手语在人工智能领域仍处于资源匮乏状态。手语指令生成（SLIG）能够产生分步骤的文本指令，使非手语使用者能够模仿并学习手语手势，从而促进双向互动。我们引入了BdSLIG，这是第一个孟加拉语手语指令生成数据集，用于评估视觉语言模型（VLMs）在资源匮乏的手语指令生成任务上的表现，并且针对长尾视觉概念进行测试，因为孟加拉语手语不太可能出现在VLM的预训练数据中。为了提升零样本性能，我们引入了融合手势参数提示（SPI），该方法将标准的手势参数，如手形、动作和朝向等直接融入文本提示中。将这些标准的手势参数嵌入到提示中，使得指令比普通的自由形式自然语言更具结构化且易于复现。我们希望我们的工作能够促进在资源匮乏社区中的手语学习系统的发展，并提高其包容性和先进性。
URL
https://arxiv.org/abs/2508.16076
PDF
-
Bridging Generalization and Personalization in Wearable Human Activity Recognition via On-Device Few-Shot Learning
2025-08-21 10:08:20Pixi Kang, Julian Moosmann, Mengxi Liu, Bo Zhou, Michele Magno, Paul Lukowicz, Sizhen BianAbstract
Human Activity Recognition (HAR) using wearable devices has advanced significantly in recent years, yet its generalization remains limited when models are deployed to new users. This degradation in performance is primarily due to user-induced concept drift (UICD), highlighting the importance of efficient personalization. In this paper, we present a hybrid framework that first generalizes across users and then rapidly adapts to individual users using few-shot learning directly on-device. By updating only the classifier layer with user-specific data, our method achieves robust personalization with minimal computational and memory overhead. We implement this framework on the energy-efficient RISC-V-based GAP9 microcontroller and validate it across three diverse HAR scenarios: RecGym, QVAR-Gesture, and Ultrasound-Gesture. Post-deployment adaptation yields consistent accuracy improvements of 3.73\%, 17.38\%, and 3.70\% respectively. These results confirm that fast, lightweight, and effective personalization is feasible on embedded platforms, paving the way for scalable and user-aware HAR systems in the wild \footnote{this https URL}.
Abstract (translated)
近年来，使用可穿戴设备进行人体活动识别（HAR）技术取得了显著进步，然而当模型部署到新用户时，其泛化能力依然有限。这种性能下降主要是由于由用户引起的概念漂移（UICD），这突显了高效个性化的必要性。在本文中，我们提出了一种混合框架，该框架首先使模型能够在不同用户之间进行泛化，然后通过设备上直接使用少量样本学习快速适应个体用户。我们的方法仅更新分类器层，并结合用户特定数据以实现稳健的个性化，同时保持计算和内存开销最小。 我们在基于低功耗RISC-V架构的GAP9微控制器上实现了这一框架，并在三个不同的HAR场景中进行了验证：RecGym、QVAR-Gesture和Ultrasound-Gesture。部署后的适应性改进分别带来了3.73%，17.38% 和 3.70% 的准确性提升。 这些结果证实了快速、轻量且有效的个性化策略在嵌入式平台上是可行的，为可穿戴设备中大规模和用户感知HAR系统的实施铺平了道路。\[1\] 参考文献： \[1\] this https URL
URL
https://arxiv.org/abs/2508.15413
PDF
-
InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
2025-08-19 17:55:23Shaoshu Yang, Zhe Kong, Feng Gao, Meng Cheng, Xiangyu Liu, Yong Zhang, Zhuoliang Kang, Wenhan Luo, Xunliang Cai, Ran He, Xiaoming WeiAbstract
Recent breakthroughs in video AIGC have ushered in a transformative era for audio-driven human animation. However, conventional video dubbing techniques remain constrained to mouth region editing, resulting in discordant facial expressions and body gestures that compromise viewer immersion. To overcome this limitation, we introduce sparse-frame video dubbing, a novel paradigm that strategically preserves reference keyframes to maintain identity, iconic gestures, and camera trajectories while enabling holistic, audio-synchronized full-body motion editing. Through critical analysis, we identify why naive image-to-video models fail in this task, particularly their inability to achieve adaptive conditioning. Addressing this, we propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing. This architecture leverages temporal context frames for seamless inter-chunk transitions and incorporates a simple yet effective sampling strategy that optimizes control strength via fine-grained reference frame positioning. Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance. Quantitative metrics confirm superior visual realism, emotional coherence, and full-body motion synchronization.
Abstract (translated)
近期在视频AIGC（人工智能生成内容）领域的突破，标志着音频驱动的人体动画进入了一个变革性的时代。然而，传统的视频配音技术仍然局限于对嘴部区域的编辑，这导致面部表情和身体动作与声音不协调，影响了观众的沉浸感。为了解决这一局限性，我们引入了一种新的范式——稀疏帧视频配音，该方法策略性地保留参考关键帧以维持身份、标志性手势以及摄像机轨迹，同时允许进行全面的身体运动编辑，使整个身体的动作与音频同步。 通过批判性的分析，我们确定了为什么简单的图像到视频模型在这个任务中失败，尤其是它们无法实现自适应条件设置。为了解决这个问题，我们提出了InfiniteTalk，这是一种专为无限长度长序列配音设计的流式音频驱动生成器架构。该架构利用时间上下文帧以实现无缝的跨片段过渡，并结合了一种简单而有效的采样策略，通过细粒度参考帧定位来优化控制强度。 在HDTF、CelebV-HQ和EMTD数据集上的全面评估表明，InfiniteTalk达到了最先进的性能水平。定量指标证实了其在视觉真实感、情感连贯性和全身运动同步方面的优越性。
URL
https://arxiv.org/abs/2508.14033
PDF
-
Federated Action Recognition for Smart Worker Assistance Using FastPose
2025-08-18 09:28:15Vinit Hegiste, Vidit Goyal, Tatjana Legler, Martin RuskowskiAbstract
In smart manufacturing environments, accurate and real-time recognition of worker actions is essential for productivity, safety, and human-machine collaboration. While skeleton-based human activity recognition (HAR) offers robustness to lighting, viewpoint, and background variations, most existing approaches rely on centralized datasets, which are impractical in privacy-sensitive industrial scenarios. This paper presents a federated learning (FL) framework for pose-based HAR using a custom skeletal dataset of eight industrially relevant upper-body gestures, captured from five participants and processed using a modified FastPose model. Two temporal backbones, an LSTM and a Transformer encoder, are trained and evaluated under four paradigms: centralized, local (per-client), FL with weighted federated averaging (FedAvg), and federated ensemble learning (FedEnsemble). On the global test set, the FL Transformer improves over centralized training by +12.4 percentage points, with FedEnsemble delivering a +16.3 percentage points gain. On an unseen external client, FL and FedEnsemble exceed centralized accuracy by +52.6 and +58.3 percentage points, respectively. These results demonstrate that FL not only preserves privacy but also substantially enhances cross-user generalization, establishing it as a practical solution for scalable, privacy-aware HAR in heterogeneous industrial settings.
Abstract (translated)
在智能制造环境中，准确且实时地识别工人的动作对于提高生产效率、保障安全以及促进人机协作至关重要。尽管基于骨架的人体活动识别（HAR）方法能够应对光照变化、视角变化和背景干扰等挑战，但大多数现有的方法都依赖于集中化的数据集，在注重隐私保护的工业场景中难以实施。本文提出了一种用于姿态识别的联邦学习（FL）框架，并使用了一个定制的骨骼数据集，该数据集包含了八种与工业相关的上半身手势动作，这些动作由五位参与者进行捕捉并利用修改后的FastPose模型进行处理。文中采用了两种时间序列骨干网络结构——LSTM和Transformer编码器，在四种不同的训练范式下进行了训练和评估：集中化训练、本地（每个客户端）训练、带有加权联邦平均算法的联邦学习（FedAvg），以及联邦集成学习（FedEnsemble）。在全局测试集上，FL Transformer相比集中化训练提高了12.4个百分点，而FedEnsemble则带来了额外提升，提高了16.3个百分点。对于未见过的外部客户端，在精度方面，FL和FedEnsemble分别比集中化方法高出52.6和58.3个百分点。这些结果表明联邦学习不仅能够保护隐私信息，还显著提升了跨用户泛化能力，从而确立了其作为异构工业环境中可扩展、隐私保护型HAR解决方案的实用性。
URL
https://arxiv.org/abs/2508.14113
PDF
