Gesture
-
Development of ML model for triboelectric nanogenerator based sign language detection system
2026-03-26 08:35:02Meshv Patel, Bikash Baro, Sayan Bayan, Mohendra RoyAbstract
Sign language recognition (SLR) is vital for bridging communication gaps between deaf and hearing communities. Vision-based approaches suffer from occlusion, computational costs, and physical constraints. This work presents a comparison of machine learning (ML) and deep learning models for a custom triboelectric nanogenerator (TENG)-based sensor glove. Utilizing multivariate time-series data from five flex sensors, the study benchmarks traditional ML algorithms, feedforward neural networks, LSTM-based temporal models, and a multi-sensor MFCC CNN-LSTM architecture across 11 sign classes (digits 1-5, letters A-F). The proposed MFCC CNN-LSTM architecture processes frequency-domain features from each sensor through independent convolutional branches before fusion. It achieves 93.33% accuracy and 95.56% precision, a 23-point improvement over the best ML algorithm (Random Forest: 70.38%). Ablation studies reveal 50-timestep windows offer a tradeoff between temporal context and training data volume, yielding 84.13% accuracy compared to 58.06% with 100-timestep windows. MFCC feature extraction maps temporal variations to execution-speed-invariant spectral representations, and data augmentation methods (time warping, noise injection) are essential for generalization. Results demonstrate that frequency-domain feature representations combined with parallel multi-sensor processing architectures offer enhancement over classical algorithms and time-domain deep learning for wearable sensor-based gesture recognition. This aids assistive technology development.
Abstract (translated)
URL
https://arxiv.org/abs/2604.06220
PDF
-
Teacher-Student Diffusion Model for Text-Driven 3D Hand Motion Generation
2026-03-25 15:21:04Ching-Lam Cheng, Bin Zhu, Shengfeng HeAbstract
Generating realistic 3D hand motion from natural language is vital for VR, robotics, and human-computer interaction. Existing methods either focus on full-body motion, overlooking detailed hand gestures, or require explicit 3D object meshes, limiting generality. We propose TSHaMo, a model-agnostic teacher-student diffusion framework for text-driven hand motion generation. The student model learns to synthesize motions from text alone, while the teacher leverages auxiliary signals (e.g., MANO parameters) to provide structured guidance during training. A co-training strategy enables the student to benefit from the teacher's intermediate predictions while remaining text-only at inference. Evaluated using two diffusion backbones on GRAB and H2O, TSHaMo consistently improves motion quality and diversity. Ablations confirm its robustness and flexibility in using diverse auxiliary inputs without requiring 3D objects at test time.
Abstract (translated)
从自然语言生成逼真的3D手部运动对于虚拟现实、机器人技术和人机交互至关重要。现有方法要么专注于全身运动，忽略了精细的手势细节；要么需要显式的3D物体网格，限制了通用性。我们提出了TSHaMo，一个与模型无关的教师-学生扩散框架，用于文本驱动的手部运动生成。学生模型仅从文本中学习合成运动，而教师则利用辅助信号（例如MANO参数）在训练期间提供结构化指导。一种协同训练策略使学生能够从教师的中期预测中获益，同时在推理时仍保持仅依赖文本。在GRAB和H2O数据集上使用两种扩散骨干网络进行评估，TSHaMo持续提升了运动质量和多样性。消融实验证实了其鲁棒性和灵活性，能够利用多样化的辅助输入，且无需在测试时使用3D物体。
URL
https://arxiv.org/abs/2603.24407
PDF
-
B-MoE: A Body-Part-Aware Mixture-of-Experts 'All Parts Matter' Approach to Micro-Action Recognition
2026-03-25 12:33:11Nishit Poddar, Aglind Reka, Diana-Laura Borza, Snehashis Majhi, Michal Balazia, Abhijit Das, Francois BremondAbstract
Micro-actions, fleeting and low-amplitude motions, such as glances, nods, or minor posture shifts, carry rich social meaning but remain difficult for current action recognition models to recognize due to their subtlety, short duration, and high inter-class ambiguity. In this paper, we introduce B-MoE, a Body-part-aware Mixture-of-Experts framework designed to explicitly model the structured nature of human motion. In B-MoE, each expert specializes in a distinct body region (head, body, upper limbs, lower limbs), and is based on the lightweight Macro-Micro Motion Encoder (M3E) that captures long-range contextual structure and fine-grained local motion. A cross-attention routing mechanism learns inter-region relationships and dynamically selects the most informative regions for each micro-action. B-MoE uses a dual-stream encoder that fuses these region-specific semantic cues with global motion features to jointly capture spatially localized cues and temporally subtle variations that characterize micro-actions. Experiments on three challenging benchmarks (MA-52, SocialGesture, and MPII-GroupInteraction) show consistent state-of-theart gains, with improvements in ambiguous, underrepresented, and low amplitude classes.
Abstract (translated)
微动作，即那些转瞬即逝、幅度细微的动作（如瞥视、点头或轻微的姿态调整），蕴含着丰富的社交意义。然而，由于其细微性、持续时间短以及类间模糊性高，当前的动作识别模型难以有效识别这类动作。本文提出B-MoE（Body-part-aware Mixture-of-Experts）框架，该框架通过显式建模人体运动的结构化特性进行设计。在B-MoE中，每个专家专注于不同的身体区域（头部、躯干、上肢、下肢），并基于轻量级的宏观-微观运动编码器（M3E），该编码器能够捕捉长程上下文结构与细粒度局部运动。交叉注意力路由机制可学习区域间关系，并为每个微动作动态选择最具信息量的区域。B-MoE采用双流编码器，将这些区域特定的语义线索与全局运动特征相融合，共同捕捉微动作所特有的空间局部化线索与时间细微变化。在三个具有挑战性的基准数据集（MA-52、SocialGesture和MPII-GroupInteraction）上的实验表明，该方法持续取得最先进的性能提升，尤其在模糊、代表性不足及低幅度类别上表现显著。
URL
https://arxiv.org/abs/2603.24245
PDF
-
A Multimodal Framework for Human-Multi-Agent Interaction
2026-03-24 14:35:40Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq IqbalAbstract
Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.
Abstract (translated)
人机交互正日益向多机器人、社会具身化的环境发展。现有系统难以在统一框架内整合多模态感知、具身表达与协调决策，这限制了共享物理空间中的自然且可扩展的交互。我们通过引入一个面向人-多智能体交互的多模态框架来解决这一缺口：每个机器人作为自主认知智能体运行，具备集成多模态感知与具身化的大语言模型（LLM）驱动规划。在团队层面，集中式协调机制管理轮次转换与智能体参与，防止语音重叠与动作冲突。该框架在两台仿人机器人上实现，通过融合语音、手势、注视与移动的交互策略，使多智能体能够开展连贯交互。典型交互实例展示了跨智能体的多模态协调推理与具身化响应。未来工作将聚焦更大规模用户研究，并深入探索社会具身化的多智能体交互动态。
URL
https://arxiv.org/abs/2603.23271
PDF
-
PHANTOM Hand
2026-03-24 12:52:18Teng Yan, Jiongxu Chen, Qixiang Hua, Yue Yu, Zihang Wang, Yaohua Liu, Bingzhuo ZhongAbstract
Tendon-driven underactuated hands excel in adaptive grasping but often suffer from kinematic unpredictability and highly non-linear force transmission. This ambiguity limits their ability to perform precise free-motion shaping and deliver reliable payloads for complex manipulation tasks. To address this, we introduce the PHANTOM Hand (Hybrid Precision-Augmented Compliance): a modular, 1:1 human-scale system featuring 6 actuators and 15 degrees of freedom (DoFs). We propose a unified framework that bridges the gap between precise analytic shaping and robust compliant grasping. By deriving a sparse mapping from physical geometry and integrating a mechanics-based compensation model, we effectively suppress kinematic drift caused by spring counter-tension and tendon elasticity. This approach achieves sub-degree kinematic reproducibility for free-motion planning while retaining the inherent mechanical compliance required for stable physical interaction. Experimental validation confirms the system's capabilities through (1) kinematic analysis verifying sub-degree global accuracy across the workspace; (2) static expressibility tests demonstrating complex hand gestures; (3) diverse grasping experiments covering power, precision, and tool-use categories; and (4) quantitative fingertip force characterization. The results demonstrate that the PHANTOM hand successfully combines analytic kinematic precision with continuous, predictable force output, significantly expanding the payload and dexterity of underactuated hands. To drive the development of the underactuated manipulation ecosystem, all hardware designs and control scripts are fully open-sourced for community engagement.
Abstract (translated)
腱驱动欠驱动手在自适应抓取方面表现卓越，但常受运动学不可预测性及高度非线性的力传递问题困扰。这种模糊性限制了其在复杂操作任务中实现精确自由运动塑形和可靠负载输送的能力。针对此问题，我们提出了PHANTOM手（混合精度增强顺应性系统）：一个模块化、1:1人体尺度的系统，配备6个执行器和15个自由度（DoFs）。我们提出一个统一框架，弥合精确解析塑形与稳健顺应抓取之间的鸿沟。通过从物理几何推导稀疏映射，并整合基于力学的补偿模型，我们有效抑制了由弹簧反向张力和腱弹性引起的运动学漂移。该方法在保持稳定物理交互所需固有机械顺应性的同时，实现了自由运动规划的亚度级运动学复现精度。实验验证通过以下四点确认了系统能力：（1）运动学分析验证了工作空间内亚度级的全局精度；（2）静态表现力测试展示了复杂手势；（3）涵盖动力抓取、精确抓取及工具使用类别的多样化抓取实验；（4）定量指尖力表征。结果表明，PHANTOM手成功结合了解析运动学精度与连续、可预测的力输出，显著拓展了欠驱动手的负载能力和灵巧性。为推动欠驱动操作生态系统发展，所有硬件设计与控制脚本均已完全开源以供社区使用。
URL
https://arxiv.org/abs/2603.23152
PDF
-
Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning
2026-03-23 11:39:16Domonkos VargaAbstract
Reliable evaluation is essential in machine learning research, yet methodological flaws-particularly data leakage-continue to undermine the validity of reported results. In this work, we investigate whether large language models (LLMs) can act as independent analytical agents capable of identifying such issues in published studies. As a case study, we analyze a gesture-recognition paper reporting near-perfect accuracy on a small, human-centered dataset. We first show that the evaluation protocol is consistent with subject-level data leakage due to non-independent training and test splits. We then assess whether this flaw can be detected independently by six state-of-the-art LLMs, each analyzing the original paper without prior context using an identical prompt. All models consistently identify the evaluation as flawed and attribute the reported performance to non-independent data partitioning, supported by indicators such as overlapping learning curves, minimal generalization gap, and near-perfect classification results. These findings suggest that LLMs can detect common methodological issues based solely on published artifacts. While not definitive, their consistent agreement highlights their potential as complementary tools for improving reproducibility and supporting scientific auditing.
Abstract (translated)
URL
https://arxiv.org/abs/2604.14161
PDF
-
Timing In stand-up Comedy: Text, Audio, Laughter, Kinesics : Pipeline and Database for the Multimodal Study of Comedic Timing
2026-03-23 10:46:15Yaelle Zribi (ENC), Florian Cafiero (ENC, LRE), Vincent L\'epinay (CJM, LIPN), Chahan Vidal-Gor\`ene (CJM, LIPN)Abstract
Stand-up comedy, and humor in general, are often studied through their verbal content. Yet live performance relies just as much on embodied presence and audience feedback. We introduce TIC-TALK, a multimodal resource with 5,400+ temporally aligned topic segments capturing language, gesture, and audience response across 90 professionally filmed stand-up comedy specials (2015-2024). The pipeline combines BERTopic for 60 s thematic segmentation with dense sentence embeddings, Whisper-AT for 0.8 s laughter detection, a fine-tuned YOLOv8-cls shot classifier, and YOLOv8s-pose for raw keypoint extraction at 1 fps. Raw 17-joint skeletal coordinates are retained without prior clustering, enabling the computation of continuous kinematic signals-arm spread, kinetic energy, and trunk lean-that serve as proxies for performance dynamics. All streams are aligned by hierarchical temporal containment without resampling, and each topic segment stores its sentence-BERT embedding for downstream similarity and clustering tasks. As a concrete use case, we study laughter dynamics across 24 thematic topics: kinetic energy negatively predicts audience laughter rate (r = -0.75, N = 24), consistent with a stillness-before-punchline pattern; personal and bodily content elicits more laughter than geopolitical themes; and shot close-up proportion correlates positively with laughter (r = +0.28), consistent with reactive montage.
Abstract (translated)
URL
https://arxiv.org/abs/2603.21803
PDF
-
Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech?
2026-03-20 10:17:10Lokesh Kumar, Nirmesh Shah, Ashishkumar P. Gudmalwar, Pankaj WasnikAbstract
Human communication seamlessly integrates speech and bodily motion, where hand gestures naturally complement vocal prosody to express intent, emotion, and emphasis. While recent text-to-speech (TTS) systems have begun incorporating multimodal cues such as facial expressions or lip movements, the role of hand gestures in shaping prosody remains largely underexplored. We propose a novel multimodal TTS framework, Gesture2Speech, that leverages visual gesture cues to modulate prosody in synthesized speech. Motivated by the observation that confident and expressive speakers coordinate gestures with vocal prosody, we introduce a multimodal Mixture-of-Experts (MoE) architecture that dynamically fuses linguistic content and gesture features within a dedicated style extraction module. The fused representation conditions an LLM-based speech decoder, enabling prosodic modulation that is temporally aligned with hand movements. We further design a gesture-speech alignment loss that explicitly models their temporal correspondence to ensure fine-grained synchrony between gestures and prosodic contours. Evaluations on the PATS dataset show that Gesture2Speech outperforms state-of-the-art baselines in both speech naturalness and gesture-speech synchrony. To the best of our knowledge, this is the first work to utilize hand gesture cues for prosody control in neural speech synthesis. Demo samples are available at this https URL
Abstract (translated)
人类交流无缝融合了言语与肢体动作，其中手势自然地与语音韵律相辅相成，以表达意图、情感和强调。尽管近期文本转语音（TTS）系统已开始整合面部表情或唇部动作等多模态线索，但手势在塑造韵律方面的作用仍 largely 未被探索。我们提出了一种新颖的多模态 TTS 框架 Gesture2Speech，该框架利用视觉手势线索来调节合成语音的韵律。受自信且富有表现力的演讲者会协调手势与语音韵律这一观察的启发，我们引入了一种多模态混合专家（MoE）架构，可在专用的风格提取模块内动态融合语言内容与手势特征。融合后的表征条件着一个基于大语言模型的语音解码器，从而实现与手部运动时间对齐的韵律调节。我们还设计了一种手势-语音对齐损失，显式建模它们的时间对应关系，以确保手势与韵律轮廓之间的细粒度同步。在 PATS 数据集上的评估表明，Gestuer2Speech 在语音自然度和手势-语音同步性两方面均优于最新基线方法。据我们所知，这是首个利用手部手势线索进行神经语音合成中韵律控制的工作。演示样本可在此 https URL 获取。
URL
https://arxiv.org/abs/2603.19831
PDF
-
SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions
2026-03-19 23:47:18Vasco Xu, Brian Chen, Eric J. Gonzalez, Andrea Cola\c{c}o, Henry Hoffmann, Mar Gonzalez-Franco, Karan AhujaAbstract
Mid-air gestures in Extended Reality (XR) often cause fatigue and imprecision. Surface-based interactions offer improved accuracy and comfort, but current egocentric vision methods struggle due to hand tracking challenges and unreliable surface plane estimation. We introduce SurfaceXR, a sensor fusion approach combining headset-based hand tracking with smartwatch IMU data to enable robust inputs on everyday surfaces. Our insight is that these modalities are complementary: hand tracking provides 3D positional data while IMUs capture high-frequency motion. A 21-participant study validates SurfaceXR's effectiveness for touch tracking and 8-class gesture recognition, demonstrating significant improvements over single-modality approaches.
Abstract (translated)
在扩展现实（XR）中，空中手势常导致疲劳和精度不足。基于表面的交互虽能提升准确性与舒适度，但当前第一人称视觉方法受限于手部追踪困难与不可靠的表面平面估计。我们提出SurfaceXR，一种通过头显手部追踪与智能手表惯性测量单元数据融合的方案，可在日常表面实现稳健输入。我们的核心见解在于这些模态具有互补性：手部追踪提供三维位置数据，而惯性测量单元能捕捉高频运动。一项涵盖21名参与者的研究验证了SurfaceXR在触摸追踪与八类手势识别中的有效性，相较于单模态方法展现出显著改进。
URL
https://arxiv.org/abs/2603.19529
PDF
-
'You've got a friend in me': Co-Designing a Peer Social Robot for Young Newcomers' Language and Cultural Learning
2026-03-19 11:54:10Neil Fernandes, Cheng Tang, Tehniyat Shahbaz, Alex Hauschildt, Emily Davies-Robinson, Yue Hu, Kerstin DautenhahnAbstract
Community literacy programs supporting young newcomer children in Canada face limited staffing and scarce one-to-one time, which constrains personalized English and cultural learning support. This paper reports on a co-design study with United for Literacy tutors that informed Maple, a table-top, peer-like Socially Assistive Robot (SAR) designed as a practice partner within tutor-mediated sessions. From shadowing and co-design interviews, we derived newcomer-specific requirements and added them in an integrated prototype that uses short story-based activities, multi-modal scaffolding (speech, facial feedback, gesture), and embedded quizzes that support attention while producing tutor-actionable formative signals. We contribute system design implications for tutor-in-the-loop SARs supporting language socialization in community settings and outline directions for child-centered evaluation in authentic programs.
Abstract (translated)
加拿大社区扫盲项目中支持新移民儿童的课程面临人员配备有限、一对一时间稀缺的问题，这限制了针对性的英语和文化学习支持。本文报告了一项与“联合扫盲组织”导师开展的协同设计研究，该研究为“枫叶”提供了设计依据——这是一款桌面式、拟人化的社会辅助机器人（SAR），旨在导师介导的会话中作为练习伙伴。通过观察与协同设计访谈，我们提取了新移民儿童的具体需求，并将其整合到一个集成原型中。该原型采用基于短篇故事的活动、多模态支架（语音、面部反馈、手势）以及嵌入式测验，既能支持儿童注意力，又能生成可供导师采取行动的形成性信号。我们提出了关于导师在环社会辅助机器人支持社区场景中语言社会化的系统设计启示，并概述了在真实项目中开展以儿童为中心评估的方向。
URL
https://arxiv.org/abs/2603.18804
PDF
-
Empathetic Motion Generation for Humanoid Educational Robots via Reasoning-Guided Vision--Language--Motion Diffusion Architecture
2026-03-19 11:26:04Fuze Sun, Lingyu Li, Lekan Dai, Xinyu FanAbstract
This article suggests a reasoning-guided vision-language-motion diffusion framework (RG-VLMD) for generating instruction-aware co-speech gestures for humanoid robots in educational scenarios. The system integrates multi-modal affective estimation, pedagogical reasoning, and teaching-act-conditioned motion synthesis to enable adaptive and semantically consistent robot behavior. A gated mixture-of-experts model predicts Valence/Arousal from input text, visual, and acoustic features, which then mapped to discrete teaching-act categories through an affect-driven this http URL signals condition a diffusion-based motion generator using clip-level intent and frame-level instructional schedules via additive latent restriction with auxiliary action-group supervision. Compared to a baseline diffusion model, our proposed method produces more structured and distinctive motion patterns, as verified by motion statics and pairwise distance analysis. Generated motion sequences remain physically plausible and can be retargeted to a NAO robot for real-time execution. The results reveal that reasoning-guided instructional conditioning improves gesture controllability and pedagogical expressiveness in educational human-robot interaction.
Abstract (translated)
本文提出了一种推理引导的视觉-语言-运动扩散框架（RG-VLMD），用于为教育场景中的人形机器人生成指令感知的共讲手势。该系统整合了多模态情感估计、教学推理和教学行为条件运动合成，以实现自适应且语义一致的机器人行为。一个门控专家混合模型从输入的文本、视觉和声学特征中预测效价/唤醒度，随后通过情感驱动信号将其映射到离散的教学行为类别。这些信号通过添加式潜在约束和辅助动作组监督，以片段级意图和帧级教学调度来条件化一个基于扩散的运动生成器。与基线扩散模型相比，本文方法产生了更具结构性和差异性的运动模式，这通过运动统计量和成对距离分析得到了验证。生成的运动序列保持物理合理性，并可重新定位到NAO机器人上实现实时执行。结果表明，推理引导的教学条件化提高了教育人机交互中的手势可控性和教学表现力。
URL
https://arxiv.org/abs/2603.18771
PDF
-
Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation
2026-03-18 06:18:46Rui Hong, Jana KoseckaAbstract
Estimating 3D hand pose from monocular RGB images is fundamental for applications in AR/VR, human-computer interaction, and sign language understanding. In this work we focus on a scenario where a discrete set of gesture labels is available and show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation. We present a two-stage framework: gesture-aware pretraining that learns an informative embedding space using coarse and fine gesture labels from InterHand2.6M, followed by a per-joint token Transformer guided by gesture embeddings as intermediate representations for final regression of MANO hand parameters. Training is driven by a layered objective over parameters, joints, and structural constraints. Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy over the state-of-the-art EANet baseline, and that the benefit transfers across architectures without any modification.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17396
PDF
-
SpikeCLR: Contrastive Self-Supervised Learning for Few-Shot Event-Based Vision using Spiking Neural Networks
2026-03-17 10:11:34Maxime Vaillant, Axel Carlier, Lai Xing Ng, Christophe Hurter, Benoit R. CottereauAbstract
Event-based vision sensors provide significant advantages for high-speed perception, including microsecond temporal resolution, high dynamic range, and low power consumption. When combined with Spiking Neural Networks (SNNs), they can be deployed on neuromorphic hardware, enabling energy-efficient applications on embedded systems. However, this potential is severely limited by the scarcity of large-scale labeled datasets required to effectively train such models. In this work, we introduce SpikeCLR, a contrastive self-supervised learning framework that enables SNNs to learn robust visual representations from unlabeled event data. We adapt prior frame-based methods to the spiking domain using surrogate gradient training and introduce a suite of event-specific augmentations that leverage spatial, temporal, and polarity transformations. Through extensive experiments on CIFAR10-DVS, N-Caltech101, N-MNIST, and DVS-Gesture benchmarks, we demonstrate that self-supervised pretraining with subsequent fine-tuning outperforms supervised learning in low-data regimes, achieving consistent gains in few-shot and semi-supervised settings. Our ablation studies reveal that combining spatial and temporal augmentations is critical for learning effective spatio-temporal invariances in event data. We further show that learned representations transfer across datasets, contributing to efforts for powerful event-based models in label-scarce settings.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16338
PDF
-
FG-SGL: Fine-Grained Semantic Guidance Learning via Motion Process Decomposition for Micro-Gesture Recognition
2026-03-17 08:57:54Jinsheng Wei, Zhaodi Xu, Guanming Lu, Haoyu Chen, Jingjie YanAbstract
Micro-gesture recognition (MGR) is challenging due to subtle inter-class variations. Existing methods rely on category-level supervision, which is insufficient for capturing subtle and localized motion differences. Thus, this paper proposes a Fine-Grained Semantic Guidance Learning (FG-SGL) framework that jointly integrates fine-grained and category-level semantics to guide vision--language models in perceiving local MG motions. FG-SA adopts fine-grained semantic cues to guide the learning of local motion features, while CP-A enhances the separability of MG features through category-level semantic guidance. To support fine-grained semantic guidance, this work constructs a fine-grained textual dataset with human annotations that describes the dynamic process of MGs in four refined semantic dimensions. Furthermore, a Multi-Level Contrastive Optimization strategy is designed to jointly optimize both modules in a coarse-to-fine pattern. Experiments show that FG-SGL achieves competitive performance, validating the effectiveness of fine-grained semantic guidance for MGR.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16269
PDF
-
End-to-End Dexterous Grasp Learning from Single-View Point Clouds via a Multi-Object Scene Dataset
2026-03-16 15:21:13Tao Geng, Dapeng Yang, Ziwei Liu, Le Zhang, Le Qi, WangYang Li, Yi Ren, Shan Luo, Fenglei NiAbstract
Dexterous grasping in multi-object scene constitutes a fundamental challenge in robotic manipulation. Current mainstream grasping datasets predominantly focus on single-object scenarios and predefined grasp configurations, often neglecting environmental interference and the modeling of dexterous pre-grasp gesture, thereby limiting their generalizability in real-world applications. To address this, we propose DGS-Net, an end-to-end grasp prediction network capable of learning dense grasp configurations from single-view point clouds in multi-object scene. Furthermore, we propose a two-stage grasp data generation strategy that progresses from dense single-object grasp synthesis to dense scene-level grasp generation. Our dataset comprises 307 objects, 240 multi-object scenes, and over 350k validated grasps. By explicitly modeling grasp offsets and pre-grasp configurations, the dataset provides more robust and accurate supervision for dexterous grasp learning. Experimental results show that DGS-Net achieves grasp success rates of 88.63\% in simulation and 78.98\% on a real robotic platform, while exhibiting lower penetration with a mean penetration depth of 0.375 mm and penetration volume of 559.45 mm^3, outperforming existing methods and demonstrating strong effectiveness and generalization capability. Our dataset is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15410
PDF
-
Pointing-Based Object Recognition
2026-03-16 15:16:53Luk\'a\v{s} Hajd\'uch, Viktor KocurAbstract
This paper presents a comprehensive pipeline for recognizing objects targeted by human pointing gestures using RGB images. As human-robot interaction moves toward more intuitive interfaces, the ability to identify targets of non-verbal communication becomes crucial. Our proposed system integrates several existing state-of-the-art methods, including object detection, body pose estimation, monocular depth estimation, and vision-language models. We evaluate the impact of 3D spatial information reconstructed from a single image and the utility of image captioning models in correcting classification errors. Experimental results on a custom dataset show that incorporating depth information significantly improves target identification, especially in complex scenes with overlapping objects. The modularity of the approach allows for deployment in environments where specialized depth sensors are unavailable.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15403
PDF
-
Scale-Gest: Scalable Model-Space Synthesis and Runtime Selection for On-Device Gesture Detection
2026-03-16 10:12:26Abdul Basit, Saim Rehman, Muhammad ShafiqueAbstract
Realizing on-device ML-based gesture detection under tight real-time performance, energy and memory constraints is challenging, especially when considering mobile devices with varying battery-power levels. Existing EdgeAI deployments typically rely on a single fixed detector, limiting optimization opportunities. We present Scale-Gest, a novel run-time adaptive gesture detection framework that expands the detector space into a dense family of tiny-YOLO architectures. We introduce multiple novel device-calibrated ACE (Accuracy-Complexity-Energy) profiles by analyzing different model-resolution-stride operating points. A lightweight run-time controller selects an appropriate ACE mode under user-defined and battery constraints, while a motion-aware hand-gesture-tracking ROI gate crops the input for reduced complexity detection. To evaluate performance of our system in real-world car driving scenarios, we introduce a temporally-annotated Driver Simulated Gesture (DSG-18) dataset. Scale-Gest maintains event-level F1 while significantly reducing energy and latency compared to single-detector approaches. On a battery-powered laptop running gesture streams, our ACE controller reduces per-frame energy by 4x (from 6.9 mJ to 1.6 mJ) while maintaining high gesture-detection performance (event-level F1 = 0.8-0.9) and low mean latency (6 ms).
Abstract (translated)
URL
https://arxiv.org/abs/2605.12506
PDF
-
Collapse or Preserve: Data-Dependent Temporal Aggregation for Spiking Neural Network Acceleration
Abstract
Spike sparsity is widely believed to enable efficient spiking neural network (SNN) inference on GPU hardware. We demonstrate this is an illusion: five distinct sparse computation strategies on Apple M3 Max all fail to outperform dense convolution, because SIMD architectures cannot exploit the fine-grained, unstructured sparsity of i.i.d. binary spikes. Instead, we propose Temporal Aggregated Convolution (TAC), which exploits convolution linearity to pre-aggregate $K$ spike frames before a single convolution call, reducing $T$ calls to $T/K$. On rate-coded data, TAC achieves 13.8times speedup with +1.6% accuracy on MNIST and +5.4% on Fashion-MNIST -- a simultaneous improvement in both speed and accuracy. However, on event-based data where the temporal dimension carries genuine motion information, TAC's temporal collapse is harmful. We therefore introduce TAC-TP (Temporal Preservation), which shares each group's convolution output across K independent LIF steps, preserving full temporal resolution for downstream layers. On DVS128-Gesture, TAC-TP achieves 95.1% accuracy (vs. 96.3% baseline) with 50% fewer convolution calls, while standard TAC drops to 91.3%. Our key finding is that the optimal temporal aggregation strategy is data-dependent: collapse the temporal dimension for rate-coded data (noise reduction) but preserve it for event data (information retention). Speedup is hardware-agnostic: TAC achieves 11.0times on NVIDIA V100, confirming the mechanism transfers across GPU architectures. All operators in the mlx-snn library are open source.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13810
PDF
-
Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering
2026-03-13 00:23:38Yura Choi, Roy Miles, Rolandos Alexandros Potamias, Ismail Elezi, Jiankang Deng, Stefanos ZafeiriouAbstract
Understanding and answering questions based on a user's pointing gesture is essential for next-generation egocentric AI assistants. However, current Multimodal Large Language Models (MLLMs) struggle with such tasks due to the lack of gesture-rich data and their limited ability to infer fine-grained pointing intent from egocentric video. To address this, we introduce EgoPointVQA, a dataset and benchmark for gesture-grounded egocentric question answering, comprising 4000 synthetic and 400 real-world videos across multiple deictic reasoning tasks. Built upon it, we further propose Hand Intent Tokens (HINT), which encodes tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model and interleaves them with the model input to provide explicit spatial and temporal context for interpreting pointing intent. We show that our model outperforms others in different backbones and model sizes. In particular, HINT-14B achieves 68.1% accuracy, on average over 6 tasks, surpassing the state-of-the-art, InternVL3-14B, by 6.6%. To further facilitate the open research, we will release the code, model, and dataset. Project page: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2603.12533
PDF
-
HiSync: Spatio-Temporally Aligning Hand Motion from Wearable IMU and On-Robot Camera for Command Source Identification in Long-Range HRI
2026-03-12 11:10:54Chengwen Zhang, Chun Yu, Borong Zhuang, Haopeng Jin, Qingyang Wan, Zhuojun Li, Zhe He, Zhoutong Ye, Yu Mei, Chang Liu, Weinan Shi, Yuanchun ShiAbstract
Long-range Human-Robot Interaction (HRI) remains underexplored. Within it, Command Source Identification (CSI) - determining who issued a command - is especially challenging due to multi-user and distance-induced sensor ambiguity. We introduce HiSync, an optical-inertial fusion framework that treats hand motion as binding cues by aligning robot-mounted camera optical flow with hand-worn IMU signals. We first elicit a user-defined (N=12) gesture set and collect a multimodal command gesture dataset (N=38) in long-range multi-user HRI scenarios. Next, HiSync extracts frequency-domain hand motion features from both camera and IMU data, and a learned CSINet denoises IMU readings, temporally aligns modalities, and performs distance-aware multi-window fusion to compute cross-modal similarity of subtle, natural gestures, enabling robust CSI. In three-person scenes up to 34m, HiSync achieves 92.32% CSI accuracy, outperforming the prior SOTA by 48.44%. HiSync is also validated on real-robot deployment. By making CSI reliable and natural, HiSync provides a practical primitive and design guidance for public-space HRI.
Abstract (translated)
长距离人机交互（HRI）领域尚待探索。其中，命令来源识别（CSI）——即确定哪位用户发出的指令——由于多用户环境和因距离导致的传感器模糊性而尤为具有挑战性。我们提出了一种光学-惯性融合框架HiSync，该框架通过将机器人安装摄像头捕捉到的手部运动光流与佩戴在手上的IMU信号对齐来处理手势绑定线索。 首先，我们从一组12名用户中定义并征集了一系列自定义手势，并收集了长距离多用户HRI场景下的跨模态命令手势数据集（N=38）。接下来，HiSync框架会分别从摄像头和IMU数据中提取频域下手部运动特征；通过CSINet算法对IMU读数进行去噪处理并完成不同传感器的时间同步工作，并采用具有距离感知能力的多窗口融合技术计算细微自然手势间的跨模态相似度，从而实现可靠的CSI。 在三人场景（最远34米）中，HiSync框架实现了92.32%的CSI准确性，比之前的最佳解决方案高出48.44%。此外，HiSync还在实际机器人部署中进行了验证。通过确保CSI既可靠又自然，HiSync为公共场所的人机交互提供了一种实用的基础技术，并对未来的系统设计提出了指导性建议。
URL
https://arxiv.org/abs/2603.11809
PDF
