Gesture
-
Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG
2026-03-09 23:00:42Nathaniel Dennler, Zhonghao Shi, Yiran Tao, Andreea Bobu, Stefanos Nikolaidis, Maja Matari\'cAbstract
Robots that interact with humans must adapt to individual users' preferences to operate effectively in human-centered environments. An intuitive and effective technique to learn non-expert users' preferences is through rankings of robot behaviors, e.g., trajectories, gestures, or voices. Existing techniques primarily focus on generating queries that optimize preference learning outcomes, such as sample efficiency or final preference estimation accuracy. However, the focus on outcome overlooks key user expectations in the process of providing these rankings, which can negatively impact users' adoption of robotic systems. This work proposes the Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG) algorithm. CMA-ES-IG explicitly incorporates user experience considerations into the preference learning process by suggesting perceptually distinct and informative trajectories for users to rank. We demonstrate these benefits through both simulated studies and real-robot experiments. CMA-ES-IG, compared to state-of-the-art alternatives, (1) scales more effectively to higher-dimensional preference spaces, (2) maintains computational tractability for high-dimensional problems, (3) is robust to noisy or inconsistent user feedback, and (4) is preferred by non-expert users in identifying their preferred robot behaviors. This project's code is available at this http URL
Abstract (translated)
与人类互动的机器人必须适应个别用户的偏好，才能在以人为中心的环境中有效运行。学习非专家用户偏好的直观且有效的技术之一是通过机器人的行为排名来进行，例如路径、手势或语音等。现有技术主要关注于生成优化偏好学习结果的查询，比如样本效率或者最终的偏好估计精度。然而，只关注结果忽视了用户提供这些排序过程中的重要期望，这可能会影响用户对机器人系统的采用。 本文提出了一种算法——协方差矩阵适应进化策略与信息增益（Covariance Matrix Adaptation Evolution Strategies with Information Gain, CMA-ES-IG）。CMA-ES-IG通过建议用户可以感知并具有信息量的路径来明确地将用户体验考虑纳入偏好学习过程中，使用户能够对其进行排名。我们通过模拟研究和真实机器人的实验展示了这种方法的好处。 与最先进的替代方案相比，CMA-ES-IG（1）更有效地扩展到更高的维度偏好数量空间；（2）对于高维问题保持计算可行性；（3）对嘈杂或不一致的用户反馈具有鲁棒性；（4）在识别非专家用户的优选机器人行为方面更受用户青睐。该项目的代码可在提供的网址中获取。 原文链接：请将“this http URL”替换为实际的有效URL，以访问项目代码。
URL
https://arxiv.org/abs/2603.09011
PDF
-
See and Switch: Vision-Based Branching for Interactive Robot-Skill Programming
2026-03-09 07:47:47Petr Vanc, Jan Kristof Behrens, V\'aclav Hlav\'a\v{c}, Karla StepanovaAbstract
Programming robots by demonstration (PbD) is an intuitive concept, but scaling it to real-world variability remains a challenge for most current teaching frameworks. Conditional task graphs are very expressive and can be defined incrementally, which fits very well with the PbD idea. However, acting using conditional task graphs requires reliable perception-grounded online branch selection. In this paper, we present See & Switch, an interactive teaching-and-execution framework that represents tasks as user-extendable graphs of skill parts connected via decision states (DS), enabling conditional branching during replay. Unlike prior approaches that rely on manual branching or low-dimensional signals (e.g., proprioception), our vision-based Switcher uses eye-in-hand images (high-dimensional) to select among competing successor skill parts and to detect out-of-distribution contexts that require new demonstrations. We integrate kinesthetic teaching, joystick control, and hand gestures via an input-modality-abstraction layer and demonstrate that our proposed method is teaching modality-independent, enabling efficient in-situ recovery demonstrations. The system is validated in experiments on three challenging dexterous manipulation tasks. We evaluate our method under diverse conditions and furthermore conduct user studies with 8 participants. We show that the proposed method reliably performs branch selection and anomaly detection for novice users, achieving 90.7 % and 87.9 % accuracy, respectively, across 576 real-robot rollouts. We provide all code and data required to reproduce our experiments at this http URL.
Abstract (translated)
通过演示编程机器人（Programming by Demonstration，PbD）是一个直观的概念，但在现实中扩大其应用范围以适应各种变化仍然对大多数当前的教学框架构成了挑战。条件任务图非常具有表现力，并且可以逐步定义，这与PbD的理念十分契合。然而，在使用条件任务图时，需要可靠的基于感知的在线分支选择来执行相应的操作。 本文介绍了一种称为“See & Switch”的互动教学和执行框架，该框架将任务表示为通过决策状态（Decision States, DS）连接起来的技能部分图形，使用户能够在重播过程中实现条件分支。与以前依赖手动分支或低维信号（如本体感受）的方法不同，我们的基于视觉的“Switcher”使用手眼相机图像（高维度信息），来在竞争性的后续技能部分之间选择，并检测需要新演示的异常情况。 我们通过输入模式抽象层集成了助动式教学、操纵杆控制和手势操作。这种方法独立于教学模态，使用户能够有效地进行现场恢复性示范。该系统已在三项具有挑战性的灵巧操控任务中进行了验证实验，涵盖了多样化的条件，并对8名参与者进行了用户研究。 我们展示了所提出的方法在执行分支选择和异常检测方面对于初学者用户的可靠性，在576个真实机器人演示中分别达到了90.7%的分支选择准确率和87.9%的异常检测准确率。所有代码和数据均可用，以重现我们的实验成果（参见文中提供的网址）。
URL
https://arxiv.org/abs/2603.08057
PDF
-
Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time
2026-03-09 05:08:02Weijie Zhou, Xuantang Xiong, Zhenlin Hu, Xiaomeng Zhu, Chaoyang Zhao, Honghui Dong, Zhengyou Zhang, Ming Tang, Jinqiao WangAbstract
In situated collaboration, speakers often use intentionally underspecified deictic commands (e.g., ``pass me \textit{that}''), whose referent becomes identifiable only by aligning speech with a brief co-speech pointing \emph{stroke}. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the \emph{audio--visual alignment} required by deictic interaction. To bridge this gap, we introduce \textbf{Egocentric Co-Speech Grounding (EcoG)}, where grounding is executable only if an agent jointly predicts \textit{What}, \textit{Where}, and \textit{When}. To operationalize this, we present \textbf{EcoG-Bench}, an evaluation-only bilingual (EN/ZH) diagnostic benchmark of \textbf{811} egocentric clips with dense spatial annotations and millisecond-level stroke supervision. It is organized under a \textbf{Progressive Cognitive Evaluation} protocol. Benchmarking state-of-the-art MLLMs reveals a severe executability gap: while human subjects achieve near-ceiling performance on EcoG-Bench (\textbf{96.9\%} strict Eco-Accuracy), the best native video-audio setting remains low (Gemini-3-Pro: \textbf{17.0\%}). Moreover, in a diagnostic ablation, replacing the native video--audio interface with timestamped frame samples and externally verified ASR (with word-level timing) substantially improves the same model (\textbf{17.0\%}$\to$\textbf{42.9\%}). Overall, EcoG-Bench provides a strict, executable testbed for event-level speech--gesture binding, and suggests that multimodal interfaces may bottleneck the observability of temporal alignment cues, independently of model reasoning.
Abstract (translated)
在情境合作中，说话者常常使用故意欠指代的指示性命令（例如，“把那个递给我”），其具体所指的是通过讲话与短暂的手势配合——即指示动作“笔画”来确定。然而，许多基于体感的评估标准允许只依赖语言的捷径，使得多模态大型语言模型（MLLMs）在不需要学习指示性互动所需的音视频对齐的情况下也能表现良好。为了弥补这一差距，我们引入了**第一人称视角伴随言语指代（Egocentric Co-Speech Grounding, EcoG）**，在这种机制下，只有当代理同时预测“什么”、“在哪里”和“何时”的时候，才允许执行指向操作。为实现这一点，我们提出了**EcoG-Bench**，这是一个针对**811个第一人称视角片段**的诊断评估基准，这些片段具有密集的空间注释以及毫秒级笔画监督，并且是双语（英语/中文）模式下进行的。该基准遵循**渐进认知评估**协议。 对最新多模态大型语言模型的评测揭示了执行能力上的显著差距：虽然人类在EcoG-Bench上接近满分地表现优异（严格第一人称视角准确率为96.9%），但在原生视频-音频设置中的最佳性能依然很低（通义千问3Pro: 17.0%）。此外，在一次诊断消融实验中，用带有时间戳的帧样本和外部验证过的自动语音识别（ASR，具有单词级定时）来代替原生音视频接口，极大地提升了同一模型的表现（从**17.0%**提升到**42.9%**）。 总体而言，EcoG-Bench为事件级别的言语-手势绑定提供了一个严格的可执行测试环境，并暗示多模态界面可能独立于模型推理能力之外限制了时间对齐线索的可观测性。
URL
https://arxiv.org/abs/2603.07966
PDF
-
Active Inference for Micro-Gesture Recognition: EFE-Guided Temporal Sampling and Adaptive Learning
2026-03-08 09:44:19Weijia Feng, Jingyu Yang, Ruojia Zhang, Fengtao Sun, Qian Gao, Chenyang Wang, Tongtong Su, Jia Guo, Xiaobai Li, Minglai ShaoAbstract
Micro-gestures are subtle and transient movements triggered by unconscious neural and emotional activities, holding great potential for human-computer interaction and clinical monitoring. However, their low amplitude, short duration, and strong inter-subject variability make existing deep models prone to degradation under low-sample, noisy, and cross-subject conditions. This paper presents an active inference-based framework for micro-gesture recognition, featuring Expected Free Energy (EFE)-guided temporal sampling and uncertainty-aware adaptive learning. The model actively selects the most discriminative temporal segments under EFE guidance, enabling dynamic observation and information gain maximization. Meanwhile, sample weighting driven by predictive uncertainty mitigates the effects of label noise and distribution shift. Experiments on the SMG dataset demonstrate the effectiveness of the proposed method, achieving consistent improvements across multiple mainstream backbones. Ablation studies confirm that both the EFE-guided observation and the adaptive learning mechanism are crucial to the performance gains. This work offers an interpretable and scalable paradigm for temporal behavior modeling under low-resource and noisy conditions, with broad applicability to wearable sensing, HCI, and clinical emotion monitoring.
Abstract (translated)
微手势是由于无意识的神经和情绪活动触发的细微且短暂的动作，它们在人机交互和临床监测方面具有巨大的潜力。然而，这些动作幅度小、持续时间短以及个体间的高度差异性使得现有的深度学习模型在样本量少、数据噪声大及跨不同用户的情况下容易退化。本文提出了一种基于主动推理框架的微手势识别方法，该方法结合了预期自由能（EFE）引导的时间采样和预测不确定性驱动的学习机制。通过EFE指导，模型能够主动选择最具判别力的时间片段，实现动态观察与信息增益的最大化。同时，由预测不确定性的样本权重调节减轻了标签噪声和分布变化的影响。 在SMG数据集上的实验表明，该方法在多个主流骨干网络上均取得了持续的性能提升。消融研究表明，EFE引导观测和自适应学习机制对于性能改进都是至关重要的因素。这项工作提供了一种可解释且可扩展的时间行为建模范式，在资源有限及噪声较大的条件下具有广泛的应用潜力，包括可穿戴传感、人机交互（HCI）以及临床情绪监测等领域。
URL
https://arxiv.org/abs/2603.07559
PDF
-
LiveSense: A Real-Time Wi-Fi Sensing Platform for Range-Doppler on COTS Laptop
2026-03-06 18:33:14Jessica Sanson, Rahul C. Shah, Maximilian Pinaroc, Cagri Tanriover, Valerio FrascollaAbstract
We present LiveSense - a cross-platform that transforms a commercial off-the-shelf (COTS) Wi-Fi Network Interface Card (NIC) on a laptop into a centimeter-level Range-Doppler sensor while preserving simultaneous communication capability. The laptops are equipped with COTS Intel AX211 (Wi-Fi 6E) or Intel BE201 (Wi-Fi 7) NICs. LiveSense can (i) Extract fully-synchronized channel state information (CSI) at >= 40 Hz, (ii) Perform time-phase alignment and self-interference cancellation on-device, and (iii) Provide a real-time stream of range, Doppler, subcarrier magnitude/phase and annotated video frames to a Python/Qt Graphical User Interface (GUI). The demo will showcase the ability to detect (i) Distance and radial velocity of attendees within a few meters of the device, (ii) Micro-motion (respiration), and (iii) Hand-gesture ranging. To the best of our knowledge, this is the first-ever demo to obtain accurate range information of targets from commercial Wi-Fi, despite the limited 160 MHz bandwidth.
Abstract (translated)
我们介绍 LiveSense——一个跨平台解决方案，它将笔记本电脑上的商用现货（COTS）Wi-Fi网络接口卡（NIC）转变为具备厘米级精度的测距-多普勒传感器，并同时保持通信功能。LiveSense 配备了 COTS Intel AX211 (Wi-Fi 6E) 或 Intel BE201 (Wi-Fi 7) NIC。 LiveSense 可以实现以下功能： (i) 提取全同步信道状态信息（CSI），频率高达 >= 40 Hz； (ii) 在设备上执行时间相位对齐和自干扰消除； (iii) 向 Python/Qt 图形用户界面 (GUI) 实时提供范围、多普勒频移、子载波幅度/相位以及标注的视频帧流。 演示将展示以下能力： (i) 检测距离及装置周围几米范围内参会者的径向速度； (ii) 微小运动（如呼吸）检测； (iii) 手势测距。 据我们所知，这是首次从商用 Wi-Fi 中获得准确目标范围信息的演示，尽管其带宽仅限于 160 MHz。
URL
https://arxiv.org/abs/2603.06545
PDF
-
GazeMoE: Perception of Gaze Target with Mixture-of-Experts
2026-03-06 13:16:29Zhuangzhuang Dai, Zhongxi Lu, Vincent G. Zakka, Luis J. Manso, Jose M Alcaraz Calero, Chen LiAbstract
Estimating human gaze target from visible images is a critical task for robots to understand human attention, yet the development of generalizable neural architectures and training paradigms remains challenging. While recent advances in pre-trained vision foundation models offer promising avenues for locating gaze targets, the integration of multi-modal cues -- including eyes, head poses, gestures, and contextual features -- demands adaptive and efficient decoding mechanisms. Inspired by Mixture-of-Experts (MoE) for adaptive domain expertise in large vision-language models, we propose GazeMoE, a novel end-to-end framework that selectively leverages gaze-target-related cues from a frozen foundation model through MoE modules. To address class imbalance in gaze target classification (in-frame vs. out-of-frame) and enhance robustness, GazeMoE incorporates a class-balancing auxiliary loss alongside strategic data augmentations, including region-specific cropping and photometric transformations. Extensive experiments on benchmark datasets demonstrate that our GazeMoE achieves state-of-the-art performance, outperforming existing methods on challenging gaze estimation tasks. The code and pre-trained models are released at this https URL
Abstract (translated)
从可见图像中估计人类视线目标是机器人理解人类注意力的关键任务，然而开发通用的神经网络架构和训练模式仍然具有挑战性。虽然近期在预训练视觉基础模型方面的进展为定位注视点提供了有前景的方法，但整合多模态线索——包括眼睛、头部姿态、手势及上下文特征——则需要灵活高效的解码机制。 受混合专家（Mixture-of-Experts, MoE）在大型视觉语言模型中适应领域专业知识的启发，我们提出了GazeMoE，这是一种新颖的端到端框架。该框架通过MoE模块选择性地利用一个冻结的基础模型中的与注视点相关的线索。为了应对视线目标分类中的类别不平衡问题（即画面内与画面外）以及增强其鲁棒性，GazeMoE引入了类平衡辅助损失，并采用了特定区域裁剪和光度变换等策略性的数据增强方法。 在基准数据集上进行的广泛实验表明，我们的GazeMoE框架实现了最先进的性能，在具有挑战性的视线估计任务中超越现有方法。相关代码与预训练模型可以在以下网址获取：[此 URL]
URL
https://arxiv.org/abs/2603.06256
PDF
-
Vision-Language System using Open-Source LLMs for Gestures in Medical Interpreter Robots
2026-03-05 23:21:39Thanh-Tung Ngo, Emma Murphy, Robert J. RossAbstract
Effective communication is vital in healthcare, especially across language barriers, where non-verbal cues and gestures are critical. This paper presents a privacy-preserving vision-language framework for medical interpreter robots that detects specific speech acts (consent and instruction) and generates corresponding robotic gestures. Built on locally deployed open-source models, the system utilizes a Large Language Model (LLM) with few-shot prompting for intent detection. We also introduce a novel dataset of clinical conversations annotated for speech acts and paired with gesture clips. Our identification module achieved 0.90 accuracy, 0.93 weighted precision, and a 0.91 weighted F1-Score. Our approach significantly improves computational efficiency and, in user studies, outperforms the speech-gesture generation baseline in human-likeness while maintaining comparable appropriateness.
Abstract (translated)
有效的沟通在医疗保健中至关重要，尤其是在语言障碍的情况下，非言语线索和手势在此时尤为重要。本文介绍了一种隐私保护的视觉-语言框架，用于医学翻译机器人，该框架能够检测特定的语言行为（如同意和指示）并生成相应的机器人动作。此系统基于本地部署的开源模型构建，并使用少量样本提示的大规模语言模型（LLM）进行意图识别。我们还引入了一个新的临床对话数据集，其中包含了针对言语行为标注的数据以及对应的肢体动作片段。 我们的身份识别模块达到了0.90的准确性、0.93加权精确度和0.91加权F1值。我们的方法显著提高了计算效率，并且在用户研究中，与现有的基于语音-手势生成的方法相比，在人机相似度方面表现更佳，同时保持了相当的适当性。 这段文字总结了一种用于医疗翻译机器人的系统框架及其性能特征，旨在通过有效的视觉和语言交互来克服语言障碍。
URL
https://arxiv.org/abs/2603.05751
PDF
-
From Phase Grounding to Intelligent Surgical Narratives
Abstract
Video surgery timelines are an important part of tool-assisted surgeries, as they allow surgeons to quickly focus on key parts of the procedure. Current methods involve the surgeon filling out a post-operation (OP) report, which is often vague, or manually annotating the surgical videos, which is highly time-consuming. Our proposed method sits between these two extremes: we aim to automatically create a surgical timeline and narrative directly from the surgical video. To achieve this, we employ a CLIP-based multi-modal framework that aligns surgical video frames with textual gesture descriptions. Specifically, we use the CLIP visual encoder to extract representations from surgical video frames and the text encoder to embed the corresponding gesture sentences into a shared embedding space. We then fine-tune the model to improve the alignment between video gestures and textual tokens. Once trained, the model predicts gestures and phases for video frames, enabling the construction of a structured surgical timeline. This approach leverages pretrained multi-modal representations to bridge visual gestures and textual narratives, reducing the need for manual video review and annotation by surgeons.
Abstract (translated)
视频手术时间线是辅助工具手术中非常重要的组成部分，因为它可以帮助外科医生快速聚焦于程序的关键部分。目前的方法包括外科医生填写术后报告，但这种报告通常不够详细；或者手动标注手术视频，这是一项耗时且繁琐的工作。我们提出了一种介于这两种方法之间的新方案：直接从手术视频自动创建手术时间线和叙述。 为了实现这一目标，我们采用了一个基于CLIP的多模态框架，该框架将手术视频帧与文本描述手势进行对齐。具体来说，我们使用CLIP视觉编码器来提取手术视频帧的表示，并利用文本编码器将其相应的手势句子嵌入到共享的嵌入空间中。然后，我们微调模型以改善视频手势和文本标记之间的对齐关系。一旦训练完成，该模型可以预测视频帧的手势及阶段，从而构建结构化的手术时间线。 这种方法通过利用预训练的多模态表示来连接视觉手势与文本叙述，极大地减少了外科医生对手术视频手动审查和标注的需求。
URL
https://arxiv.org/abs/2603.05732
PDF
-
Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry
2026-03-05 18:22:55Yifan Zhu, Mariah Bradford, Kenneth Lai, Timothy Obiso, Videep Venkatesha, James Pustejovsky, Nikhil KrishnaswamyAbstract
Establishing common ground, a shared set of beliefs and mutually recognized facts, is fundamental to collaboration, yet remains a challenge for current AI systems, especially in multimodal, multiparty settings, where the collaborators bring different information to the table. We introduce the Distributed Partial Information Puzzle (DPIP), a collaborative construction task that elicits rich multimodal communication under epistemic asymmetry. We present a multimodal dataset of these interactions, annotated and temporally aligned across speech, gesture, and action modalities to support reasoning over propositional content and belief dynamics. We then evaluate two paradigms for modeling common ground (CG): (1) state-of-the-art large language models (LLMs), prompted to infer shared beliefs from multimodal updates, and (2) an axiomatic pipeline grounded in Dynamic Epistemic Logic (DEL) that incrementally performs the same task. Results on the annotated DPIP data indicate that it poses a challenge to modern LLMs' abilities to track both task progression and belief state.
Abstract (translated)
建立共同基础，即共享的一套信念和双方认可的事实，是合作的基础，但对当前的人工智能系统来说仍然是一个挑战，尤其是在多模态、多方参与的场景中，参与者会带来不同的信息。我们引入了分布式部分信息拼图（Distributed Partial Information Puzzle, DPIP），这是一个在认识论不对称性条件下激发丰富多模态交流的合作构建任务。我们提供了一套此类互动的多模态数据集，并对言语、手势和行动等不同模态进行了标注和时间同步，以支持命题内容和信念动态的推理。接着，我们评估了两种建立共同基础（CG）的方法：(1) 基于最先进的大型语言模型（LLMs），通过提示它们从多模态更新中推断共享信念；以及 (2) 一种基于动态认识逻辑（Dynamic Epistemic Logic, DEL）的公理化流程，该流程逐步完成同样的任务。在标注的DPIP数据上的结果显示，它对现代大型语言模型跟踪任务进展和信念状态的能力提出了挑战。
URL
https://arxiv.org/abs/2603.05450
PDF
-
LEGS-POMDP: Language and Gesture-Guided Object Search in Partially Observable Environments
Abstract
To assist humans in open-world environments, robots must interpret ambiguous instructions to locate desired objects. Foundation model-based approaches excel at multimodal grounding, but they lack a principled mechanism for modeling uncertainty in long-horizon tasks. In contrast, Partially Observable Markov Decision Processes (POMDPs) provide a systematic framework for planning under uncertainty but are often limited in supported modalities and rely on restrictive environment assumptions. We introduce LanguagE and Gesture-Guided Object Search in Partially Observable Environments (LEGS-POMDP), a modular POMDP system that integrates language, gesture, and visual observations for open-world object search. Unlike prior work, LEGS-POMDP explicitly models two sources of partial observability: uncertainty over the target object's identity and its spatial location. In simulation, multimodal fusion significantly outperforms unimodal baselines, achieving an average success rate of 89\% across challenging environments and object categories. Finally, we demonstrate the full system on a quadruped mobile manipulator, where real-world experiments qualitatively validate robust multimodal perception and uncertainty reduction under ambiguous instructions.
Abstract (translated)
为了在开放环境中协助人类，机器人必须能够解析模糊指令来定位所需物体。基于基础模型的方法在多模态语义关联方面表现出色，但在长时间任务中缺乏一种系统的方式来建模不确定性。相比之下，部分可观察马尔科夫决策过程(POMDP)提供了一种在不确定条件下进行规划的系统框架，但通常支持的模态有限，并依赖于严格的环境假设。我们引入了一个模块化POMDP系统——LanguagE and Gesture-Guided Object Search in Partially Observable Environments (LEGS-POMDP)，该系统结合了语言、手势和视觉观察来实现开放世界中的物体搜索功能。 不同于以往的工作，LEGS-POMDP 明确建模了两种部分可观察性的来源：目标对象身份的不确定性及其空间位置的不确定性。在模拟环境中，多模式融合显著优于单模式基线方法，在具有挑战性环境及不同类别对象的情况下，平均成功率为89%。 最后，我们在四足移动机器人上展示了完整的系统，并通过实际实验定性地验证了其强大的多模态感知能力以及在模糊指令下不确定性减少的能力。
URL
https://arxiv.org/abs/2603.04705
PDF
-
MRPoS: Mixed Reality-Based Robot Navigation Interface Using Spatial Pointing and Speech with Large Language Model
2026-03-04 08:45:25Eduardo Iglesius, Masato Kobayashi, Yuki UranishiAbstract
Recent advancements have made robot navigation more intuitive by transitioning from traditional 2D displays to spatially aware Mixed Reality (MR) systems. However, current MR interfaces often rely on manual "air tap" gestures for goal placement, which can be repetitive and physically demanding, especially for beginners. This paper proposes the Mixed Reality-Based Robot Navigation Interface using Spatial Pointing and Speech (MRPoS). This novel framework replaces complex hand gestures with a natural, multimodal interface combining spatial pointing with Large Language Model (LLM)-based speech interaction. By leveraging both information, the system translates verbal intent into navigation goals visualized by MR technology. Comprehensive experiments comparing MRPoS against conventional gesture-based systems demonstrate that our approach significantly reduces task completion time and workload, providing a more accessible and efficient interface. For additional material, please check: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2603.13313
PDF
-
MIBURI: Towards Expressive Interactive Gesture Synthesis
2026-03-03 18:59:51M. Hamza Mughal, Rishabh Dabral, Vera Demberg, Christian TheobaltAbstract
Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions. Current large language model (LLM)-based conversational agents lack embodiment and the expressive gestures essential for natural interaction. Existing solutions for ECAs often produce rigid, low-diversity motions, that are unsuitable for human-like interaction. Alternatively, generative methods for co-speech gesture synthesis yield natural body gestures but depend on future speech context and require long run-times. To bridge this gap, we present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue. We employ body-part aware gesture codecs that encode hierarchical motion details into multi-level discrete tokens. These tokens are then autoregressively generated by a two-dimensional causal framework conditioned on LLM-based speech-text embeddings, modeling both temporal dynamics and part-level motion hierarchy in real time. Further, we introduce auxiliary objectives to encourage expressive and diverse gestures while preventing convergence to static poses. Comparative evaluations demonstrate that our causal and real-time approach produces natural and contextually aligned gestures against recent baselines. We urge the reader to explore demo videos on this https URL.
Abstract (translated)
实体对话代理（ECAs）旨在通过语音、手势和面部表情模拟面对面的人类互动。当前基于大型语言模型（LLM）的对话代理缺乏身体表现力以及自然互动所必需的表情动作。现有的ECAs解决方案通常会产生僵硬且多样性较低的动作，不适合模仿人类交互。另一方面，用于同步生成自然肢体动作的方法虽然依赖于未来语境中的言语内容，并能产生自然的身体手势，但需要较长的时间来运行。 为了弥合这一差距，我们介绍了MIBURI，这是一个在线、因果框架，用于实时合成与对话语音相匹配的富有表现力的全身动作和面部表情。我们采用肢体部位感知的手势编解码器，将分层运动细节编码为多级离散令牌，并通过基于LLM的言语-文本嵌入条件，在二维因果框架内自回归生成这些令牌，从而实时建模时间动态和部分级别的运动层次结构。此外，为了鼓励产生富有表现力且多样化的手势并防止收敛到静态姿态，我们引入了辅助目标。 对比评估表明，我们的因果性和实时性方法在最近的基准测试中产生的动作自然且与上下文相关。我们建议读者访问以下链接查看演示视频：[此处插入实际URL]（请根据实际情况提供具体链接）。
URL
https://arxiv.org/abs/2603.03282
PDF
-
Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion
2026-02-27 05:52:04Seungyeol Baek, Jaspreet Singh, Lala Shakti Swarup Ray, Hymalai Bello, Paul Lukowicz, Sungho SuhAbstract
Human operators are still frequently exposed to hazardous environments such as disaster zones and industrial facilities, where intuitive and reliable teleoperation of mobile robots and Unmanned Aerial Vehicles (UAVs) is essential. In this context, hands-free teleoperation enhances operator mobility and situational awareness, thereby improving safety in hazardous environments. While vision-based gesture recognition has been explored as one method for hands-free teleoperation, its performance often deteriorates under occlusions, lighting variations, and cluttered backgrounds, limiting its applicability in real-world operations. To overcome these limitations, we propose a multimodal gesture recognition framework that integrates inertial data (accelerometer, gyroscope, and orientation) from Apple Watches on both wrists with capacitive sensing signals from custom gloves. We design a late fusion strategy based on the log-likelihood ratio (LLR), which not only enhances recognition performance but also provides interpretability by quantifying modality-specific contributions. To support this research, we introduce a new dataset of 20 distinct gestures inspired by aircraft marshalling signals, comprising synchronized RGB video, IMU, and capacitive sensor data. Experimental results demonstrate that our framework achieves performance comparable to a state-of-the-art vision-based baseline while significantly reducing computational cost, model size, and training time, making it well suited for real-time robot control. We therefore underscore the potential of sensor-based multimodal fusion as a robust and interpretable solution for gesture-driven mobile robot and drone teleoperation.
Abstract (translated)
人类操作员仍然经常暴露在危险环境中，如灾区和工业设施中，在这些地方，直观且可靠的移动机器人和无人驾驶飞行器（UAV）远程操控是至关重要的。在这种情况下，无手操控制（hands-free teleoperation）能够增强操作人员的机动性和情境意识，从而提高危险环境中的安全性。虽然基于视觉的手势识别已被探索为一种无手操控的方法，但由于遮挡、光照变化和杂乱背景的影响，其性能经常下降，限制了其实用性。 为了克服这些局限性，我们提出了一种多模态手势识别框架，该框架集成了来自双腕Apple Watch的惯性数据（加速度计、陀螺仪和方向）以及定制手套提供的电容感应信号。我们设计了一种基于对数似然比（LLR）的晚期融合策略，这种策略不仅提升了识别性能，还通过量化各模式的具体贡献提供了可解释性。 为了支持这项研究，我们引入了一个新的数据集，该数据集中包含20个不同的手势，这些手势源自飞机引导信号。这个新数据集包括同步的RGB视频、IMU和电容传感器数据。实验结果表明，我们的框架在性能上与最先进的基于视觉的方法相媲美，同时显著降低了计算成本、模型大小和训练时间，使其非常适合实时机器人控制。 因此，我们强调传感器驱动多模态融合作为手势驱动移动机器人和无人机远程操控的稳健且可解释解决方案的巨大潜力。
URL
https://arxiv.org/abs/2602.23694
PDF
-
DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generation
2026-02-26 16:30:07Yichen Peng, Jyun-Ting Song, Siyeol Jung, Ruofan Liu, Haiyang Liu, Xuangeng Chu, Ruicong Liu, Erwin Wu, Hideki Koike, Kris KitaniAbstract
Generating realistic conversational gestures are essential for achieving natural, socially engaging interactions with digital humans. However, existing methods typically map a single audio stream to a single speaker's motion, without considering social context or modeling the mutual dynamics between two people engaging in conversation. We present DyaDiT, a multi-modal diffusion transformer that generates contextually appropriate human motion from dyadic audio signals. Trained on Seamless Interaction Dataset, DyaDiT takes dyadic audio with optional social-context tokens to produce context-appropriate motion. It fuses information from both speakers to capture interaction dynamics, uses a motion dictionary to encode motion priors, and can optionally utilize the conversational partner's gestures to produce more responsive motion. We evaluate DyaDiT on standard motion generation metrics and conduct quantitative user studies, demonstrating that it not only surpasses existing methods on objective metrics but is also strongly preferred by users, highlighting its robustness and socially favorable motion generation. Code and models will be released upon acceptance.
Abstract (translated)
生成逼真的对话手势对于实现自然、社交互动的数字人类交互至关重要。然而，现有方法通常将单一音频流映射到单个发言人的动作上，而不考虑社会背景或建模两个参与对话的人之间的相互动态关系。我们提出了DyaDiT，这是一种多模式扩散变压器，可以从双人音频信号生成上下文相关的肢体动作。在Seamless Interaction Dataset（无缝交互数据集）上训练后，DyaDiT可以接收带有可选社交背景标记的双人音频，并产生相应的适当动作。它融合了双方的信息以捕捉互动动态，使用动作词典来编码动作先验知识，并且可以选择利用对话伙伴的手势生成更灵敏的动作。 我们在标准动作生成指标上评估了DyaDiT，并进行了定量用户研究，证明了它不仅在客观指标上超越现有方法，而且也因其实用性和社交友好性而被用户强烈偏爱。这表明其稳健的上下文适应能力以及能够产生高度自然的社会互动行为。代码和模型将在接受后发布。 这段文字描述了一种新的人工智能系统DyaDiT的功能与优势，该系统能够生成更符合社会情境的动作，并且在用户体验方面有着显著的优势。
URL
https://arxiv.org/abs/2602.23165
PDF
-
SignVLA: A Gloss-Free Vision-Language-Action Framework for Real-Time Sign Language-Guided Robotic Manipulation
2026-02-26 01:16:27Xinyu Tan, Ningwei Bai, Harry Gardener, Zhengyang Zhong, Luoyu Zhang, Liuhaichen Yang, Zhekai Duan, Monkgogi Galeitsiwe, Zezhi TangAbstract
We present, to our knowledge, the first sign language-driven Vision-Language-Action (VLA) framework for intuitive and inclusive human-robot interaction. Unlike conventional approaches that rely on gloss annotations as intermediate supervision, the proposed system adopts a gloss-free paradigm and directly maps visual sign gestures to semantic instructions. This design reduces annotation cost and avoids the information loss introduced by gloss representations, enabling more natural and scalable multimodal interaction. In this work, we focus on a real-time alphabet-level finger-spelling interface that provides a robust and low-latency communication channel for robotic control. Compared with large-scale continuous sign language recognition, alphabet-level interaction offers improved reliability, interpretability, and deployment feasibility in safety-critical embodied environments. The proposed pipeline transforms continuous gesture streams into coherent language commands through geometric normalization, temporal smoothing, and lexical refinement, ensuring stable and consistent interaction. Furthermore, the framework is designed to support future integration of transformer-based gloss-free sign language models, enabling scalable word-level and sentence-level semantic understanding. Experimental results demonstrate the effectiveness of the proposed system in grounding sign-derived instructions into precise robotic actions under diverse interaction scenarios. These results highlight the potential of the framework to advance accessible, scalable, and multimodal embodied intelligence.
Abstract (translated)
我们提出了一种开创性的手语驱动的视觉-语言-动作（VLA）框架，旨在实现直观且包容的人机交互。与传统的依赖于词素注释作为中间监督的方法不同，所提出的系统采用无词素范式，并直接将视觉手势映射为语义指令。这种设计减少了标注成本，避免了由词素表示引入的信息损失，从而实现了更自然和可扩展的多模态互动。在本研究中，我们专注于一个实时的手字母级手指拼写界面，该界面提供了一个可靠且低延迟的机器人控制通信通道。 与大规模连续手语识别相比，字母级别的交互在安全关键性的实体环境中提供了更高的可靠性、可解释性和部署可行性。所提出的流水线通过几何标准化、时间平滑和词汇精炼将连续手势流转换为连贯的语言命令，确保了稳定且一致的互动体验。此外，该框架设计支持未来基于变压器的无词素手语模型的集成，从而能够实现单词级和句子级的语义理解。 实验结果证明，在各种交互场景中，所提出的系统能有效地将手势指令转化为精确的机器人动作，突显了该框架在推进可访问性、可扩展性和多模态实体智能方面的潜力。
URL
https://arxiv.org/abs/2602.22514
PDF
-
iMiGUE-Speech: A Spontaneous Speech Dataset for Affective Analysis
2026-02-25 00:38:19Sofoklis Kakouros, Fang Kang, Haoyu ChenAbstract
This work presents iMiGUE-Speech, an extension of the iMiGUE dataset that provides a spontaneous affective corpus for studying emotional and affective states. The new release focuses on speech and enriches the original dataset with additional metadata, including speech transcripts, speaker-role separation between interviewer and interviewee, and word-level forced alignments. Unlike existing emotional speech datasets that rely on acted or laboratory-elicited emotions, iMiGUE-Speech captures spontaneous affect arising naturally from real match outcomes. To demonstrate the utility of the dataset and establish initial benchmarks, we introduce two evaluation tasks for comparative assessment: speech emotion recognition and transcript-based sentiment analysis. These tasks leverage state-of-the-art pre-trained representations to assess the dataset's ability to capture spontaneous affective states from both acoustic and linguistic modalities. iMiGUE-Speech can also be synchronously paired with micro-gesture annotations from the original iMiGUE dataset, forming a uniquely multimodal resource for studying speech-gesture affective dynamics. The extended dataset is available at this https URL.
Abstract (translated)
这项工作介绍了iMiGUE-Speech，这是对iMiGUE数据集的扩展，提供了用于研究情绪和情感状态的自发性情感语料库。新版本专注于语音，并通过添加额外的元数据来丰富原始数据集，包括语音转录、访谈者与受访者的角色分离以及词级别强制对齐信息。 不同于现有的基于表演或实验室诱发的情绪语音数据集，iMiGUE-Speech 捕捉到了自然来源于真实比赛结果中的自发性情感。为了展示该数据集的实用性并建立初步基准，我们引入了两个用于比较评估的任务：语音情绪识别和基于转录的情感分析。这些任务利用最先进的预训练表示来评估数据集捕捉来自声学和语言模态的自发情感状态的能力。 iMiGUE-Speech还可以与原始iMiGUE数据集中微手势注释同步配对，形成一个独特的多模式资源，用于研究语音-手势情感动态。扩展后的数据集可以在[此链接](https://this https URL/)获取。
URL
https://arxiv.org/abs/2602.21464
PDF
-
SARAH: Spatially Aware Real-time Agentic Humans
2026-02-20 18:59:35Evonne Ng, Siwei Zhang, Zhang Chen, Michael Zollhoefer, Alexander RichardAbstract
As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see this https URL for details.
Abstract (translated)
随着虚拟现实（VR）、远程呈现和数字人类应用的发展，实体代理的角色变得越来越重要。这些代理的动作需要超越与语音同步的手势，它们应该能够面向用户、响应用户的移动，并保持自然的眼神交流。目前的方法缺乏这种空间感知能力。我们填补了这一空白，推出了首个实时的、完全因果关系的空间感知对话动作生成方法，能够在流媒体VR头显上部署。 我们的方法基于给定用户的位置和双向音频信息来生成全身运动，使手势与语音同步的同时根据用户的移动调整代理的方向。我们的架构结合了一个因果Transformer-based变分自编码器（VAE）以及交错的隐变量令牌以支持流式推理，还有一个受控于用户轨迹和音频的数据匹配模型。 为了适应不同的眼神交流偏好，我们引入了一个眼神评分机制，并使用无分类指导来分离学习与控制：模型从数据中捕捉自然的空间对应关系，而用户则可以在推断时调整目光接触的强度。在Embody 3D数据集上，我们的方法实现了超过每秒300帧的速度（比非因果基准快三倍）并捕获了自然对话中的微妙空间动态变化。 我们通过一个实时VR系统验证了这一方案的有效性，将具有空间感知能力的对话代理带到了现实世界的部署中。详情请参阅此[链接](https://example.com)。
URL
https://arxiv.org/abs/2602.18432
PDF
-
HiAER-Spike Software-Hardware Reconfigurable Platform for Event-Driven Neuromorphic Computing at Scale
2026-02-20 08:55:47Gwenevere Frank, Gopabandhu Hota, Keli Wang, Christopher Deng, Krish Arora, Diana Vins, Abhinav Uppal, Omowuyi Olajide, Kenneth Yoshimoto, Qingbo Wang, Mari Yamaoka, Johannes Leugering, Stephen Deiss, Leif Gibb, Gert CauwenberghsAbstract
In this work, we present HiAER-Spike, a modular, reconfigurable, event-driven neuromorphic computing platform designed to execute large spiking neural networks with up to 160 million neurons and 40 billion synapses - roughly twice the neurons of a mouse brain at faster than real time. This system, assembled at the UC San Diego Supercomputer Center, comprises a co-designed hard- and software stack that is optimized for run-time massively parallel processing and hierarchical address-event routing (HiAER) of spikes while promoting memory-efficient network storage and execution. The architecture efficiently handles both sparse connectivity and sparse activity for robust and low-latency event-driven inference for both edge and cloud computing. A Python programming interface to HiAER-Spike, agnostic to hardware-level detail, shields the user from complexity in the configuration and execution of general spiking neural networks with minimal constraints in topology. The system is made easily available over a web portal for use by the wider community. In the following, we provide an overview of the hard- and software stack, explain the underlying design principles, demonstrate some of the system's capabilities and solicit feedback from the broader neuromorphic community. Examples are shown demonstrating HiAER-Spike's capabilities for event-driven vision on benchmark CIFAR-10, DVS event-based gesture, MNIST, and Pong tasks.
Abstract (translated)
在这项工作中，我们介绍了HiAER-Spike，这是一个模块化、可重构的事件驱动神经形态计算平台，旨在执行高达1.6亿个神经元和400亿个突触的大规模脉冲神经网络——这大约是老鼠大脑神经元数量的两倍，并且运行速度超过实时。该系统在加州大学圣地亚哥分校超级计算机中心组装而成，它包含了一个专门为大规模并行处理而优化、采用层次化地址事件路由（HiAER）的软硬件堆栈，在提高内存效率的同时支持网络存储和执行。这种架构能够高效地应对稀疏连接和稀疏活动，适用于边缘计算和云计算中的稳健且低延迟的事件驱动推理。我们还提供了一个Python编程接口给HiAER-Spike，该接口不依赖于硬件细节，并能简化用户在配置和执行一般脉冲神经网络时的操作复杂性，仅需遵守有限的拓扑约束。通过一个网页门户，这个系统被广泛地社区成员所轻松访问并使用。 接下来，我们概述了软硬件堆栈，解释了背后的设计原则，并展示了该系统的部分能力，同时邀请更广泛的神经形态计算社区提供反馈意见。文中还提供了实例展示HiAER-Spike在事件驱动视觉（基准CIFAR-10）、DVS基于事件的手势识别、MNIST任务和Pong游戏等领域的应用能力。
URL
https://arxiv.org/abs/2602.18072
PDF
-
On the Evaluation Protocol of Gesture Recognition for UAV-based Rescue Operation based on Deep Learning: A Subject-Independence Perspective
Abstract
This paper presents a methodological analysis of the gesture-recognition approach proposed by Liu and Szirányi, with a particular focus on the validity of their evaluation protocol. We show that the reported near-perfect accuracy metrics result from a frame-level random train-test split that inevitably mixes samples from the same subjects across both sets, causing severe data leakage. By examining the published confusion matrix, learning curves, and dataset construction, we demonstrate that the evaluation does not measure generalization to unseen individuals. Our findings underscore the importance of subject-independent data partitioning in vision-based gesture-recognition research, especially for applications - such as UAV-human interaction - that require reliable recognition of gestures performed by previously unseen people.
Abstract (translated)
本文对刘和Szirányi提出的姿态识别方法进行了方法论分析，特别关注其评估协议的有效性。我们展示了所报告的近乎完美的准确率指标是由帧级别的随机训练-测试拆分导致的结果，这种拆分不可避免地将同一主体的数据混入了两个数据集之中，从而造成了严重的信息泄露问题。通过审查已发布的混淆矩阵、学习曲线以及数据集构建情况，我们证明该评估无法衡量对未见过个体的泛化能力。我们的研究结果强调，在基于视觉的姿态识别研究中进行独立于主体的数据拆分的重要性，尤其是在如无人机与人交互这样的应用领域，这些应用需要可靠地识别人群中新出现的手势。
URL
https://arxiv.org/abs/2602.17854
PDF
-
FR-GESTURE: An RGBD Dataset For Gesture-based Human-Robot Interaction In First Responder Operations
2026-02-19 17:31:08Konstantinos Foteinos, Georgios Angelidis, Aggelos Psiris, Vasileios Argyriou, Panagiotis Sarigiannidis, Georgios Th. PapadopoulosAbstract
The ever increasing intensity and number of disasters make even more difficult the work of First Responders (FRs). Artificial intelligence and robotics solutions could facilitate their operations, compensating these difficulties. To this end, we propose a dataset for gesture-based UGV control by FRs, introducing a set of 12 commands, drawing inspiration from existing gestures used by FRs and tactical hand signals and refined after incorporating feedback from experienced FRs. Then we proceed with the data collection itself, resulting in 3312 RGBD pairs captured from 2 viewpoints and 7 distances. To the best of our knowledge, this is the first dataset especially intended for gesture-based UGV guidance by FRs. Finally we define evaluation protocols for our RGBD dataset, termed FR-GESTURE, and we perform baseline experiments, which are put forward for improvement. We have made data publicly available to promote future research on the domain: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.17573
PDF
