Gesture
-
Plant-Inspired Robot Design Metaphors for Ambient HRI
2026-01-29 22:55:04Victor Nikhil Antony, Adithya R N, Sarah Derrick, Zhili Gong, Peter M. Donley, Chien-Ming HuangAbstract
Plants offer a paradoxical model for interaction: they are ambient, low-demand presences that nonetheless shape atmosphere, routines, and relationships through temporal rhythms and subtle expressions. In contrast, most human-robot interaction (HRI) has been grounded in anthropomorphic and zoomorphic paradigms, producing overt, high-demand forms of engagement. Using a Research through Design (RtD) methodology, we explore plants as metaphoric inspiration for HRI; we conducted iterative cycles of ideation, prototyping, and reflection to investigate what design primitives emerge from plant metaphors and morphologies, and how these primitives can be combined into expressive robotic forms. We present a suite of speculative, open-source prototypes that help probe plant-inspired presence, temporality, form, and gestures. We deepened our learnings from design and prototyping through prototype-centered workshops that explored people's perceptions and imaginaries of plant-inspired robots. This work contributes: (1) Set of plant-inspired robotic artifacts; (2) Designerly insights on how people perceive plant-inspired robots; and (3) Design consideration to inform how to use plant metaphors to reshape HRI.
Abstract (translated)
URL
https://arxiv.org/abs/2601.22387
PDF
-
Tendon-based modelling, estimation and control for a simulated high-DoF anthropomorphic hand model
2026-01-28 15:10:02P\'eter Polcz, Katalin Sch\"affer, Mikl\'os KollerAbstract
Tendon-driven anthropomorphic robotic hands often lack direct joint angle sensing, as the integration of joint encoders can compromise mechanical compactness and dexterity. This paper presents a computational method for estimating joint positions from measured tendon displacements and tensions. An efficient kinematic modeling framework for anthropomorphic hands is first introduced based on the Denavit-Hartenberg convention. Using a simplified tendon model, a system of nonlinear equations relating tendon states to joint positions is derived and solved via a nonlinear optimization approach. The estimated joint angles are then employed for closed-loop control through a Jacobian-based proportional-integral (PI) controller augmented with a feedforward term, enabling gesture tracking without direct joint sensing. The effectiveness and limitations of the proposed estimation and control framework are demonstrated in the MuJoCo simulation environment using the Anatomically Correct Biomechatronic Hand, featuring five degrees of freedom for each long finger and six degrees of freedom for the thumb.
Abstract (translated)
URL
https://arxiv.org/abs/2601.20682
PDF
-
Graph Modelling Analysis of Speech-Gesture Interaction for Aphasia Severity Estimation
2026-01-27 14:11:36Navya Martin Kollapally, Christa Akers, Renjith Nelson JosephAbstract
Aphasia is an acquired language disorder caused by injury to the regions of the brain that are responsible for language. Aphasia may impair the use and comprehension of written and spoken language. The Western Aphasia Battery-Revised (WAB-R) is an assessment tool administered by speech-language pathologists (SLPs) to evaluate the aphasia type and severity. Because the WAB-R measures isolated linguistic skills, there has been growing interest in the assessment of discourse production as a more holistic representation of everyday language abilities. Recent advancements in speech analysis focus on automated estimation of aphasia severity from spontaneous speech, relying mostly in isolated linguistic or acoustical features. In this work, we propose a graph neural network-based framework for estimating aphasia severity. We represented each participant's discourse as a directed multi-modal graph, where nodes represent lexical items and gestures and edges encode word-word, gesture-word, and word-gesture transitions. GraphSAGE is employed to learn participant-level embeddings, thus integrating information from immediate neighbors and overall graph structure. Our results suggest that aphasia severity is not encoded in isolated lexical distribution, but rather emerges from structured interactions between speech and gesture. The proposed architecture offers a reliable automated aphasia assessment, with possible uses in bedside screening and telehealth-based monitoring.
Abstract (translated)
失语症是一种由于大脑语言区域受损而获得的语言障碍。它可能会影响口语和书面语言的使用与理解。西方失语症评估量表修订版（WAB-R）是由言语-语言治疗师（SLPs）用来评估失语类型及严重程度的一种评估工具。因为WAB-R测量的是孤立的语言技能，所以越来越多的人开始对话语产出的评估感兴趣，以作为日常语言能力更全面的表现形式。最近在语音分析方面的进展主要集中在从自然语言中自动估计失语症的严重程度上，这主要依赖于孤立的语言或声学特征。在这项工作中，我们提出了一种基于图神经网络框架来估算失语症严重程度的方法。我们将每个参与者的对话表示为一个有向多模态图，在这种图中节点代表词汇和手势，边则编码词-词、手势-词以及词-手势的转换关系。使用GraphSAGE来学习参与者级别的嵌入，从而整合了来自直接邻居的信息及整个图形结构的信息。我们的结果显示，失语症严重程度并非体现在孤立的词汇分布上，而是从言语与手势之间的有机构成互动中浮现出来。所提出的架构提供了一种可靠的自动化失语评估方法，可能在床边筛查和基于远程医疗监测方面有着潜在的应用价值。
URL
https://arxiv.org/abs/2602.20163
PDF
-
MA-LipNet: Multi-Dimensional Attention Networks for Robust Lipreading
2026-01-27 09:19:18Matteo RossiAbstract
Lipreading, the technology of decoding spoken content from silent videos of lip movements, holds significant application value in fields such as public security. However, due to the subtle nature of articulatory gestures, existing lipreading methods often suffer from limited feature discriminability and poor generalization capabilities. To address these challenges, this paper delves into the purification of visual features from temporal, spatial, and channel dimensions. We propose a novel method named Multi-Attention Lipreading Network(MA-LipNet). The core of MA-LipNet lies in its sequential application of three dedicated attention modules. Firstly, a \textit{Channel Attention (CA)} module is employed to adaptively recalibrate channel-wise features, thereby mitigating interference from less informative channels. Subsequently, two spatio-temporal attention modules with distinct granularities-\textit{Joint Spatial-Temporal Attention (JSTA)} and \textit{Separate Spatial-Temporal Attention (SSTA)}-are leveraged to suppress the influence of irrelevant pixels and video frames. The JSTA module performs a coarse-grained filtering by computing a unified weight map across the spatio-temporal dimensions, while the SSTA module conducts a more fine-grained refinement by separately modeling temporal and spatial attentions. Extensive experiments conducted on the CMLR and GRID datasets demonstrate that MA-LipNet significantly reduces the Character Error Rate (CER) and Word Error Rate (WER), validating its effectiveness and superiority over several state-of-the-art methods. Our work highlights the importance of multi-dimensional feature refinement for robust visual speech recognition.
Abstract (translated)
URL
https://arxiv.org/abs/2601.20881
PDF
-
GhostUI: Unveiling Hidden Interactions in Mobile UI
2026-01-27 06:40:29Minkyu Kweon, Seokhyeon Park, Soohyun Lee, You Been Lee, Jeongmin Rhee, Jinwook SeoAbstract
Modern mobile applications rely on hidden interactions--gestures without visual cues like long presses and swipes--to provide functionality without cluttering interfaces. While experienced users may discover these interactions through prior use or onboarding tutorials, their implicit nature makes them difficult for most users to uncover. Similarly, mobile agents--systems designed to automate tasks on mobile user interfaces, powered by vision language models (VLMs)--struggle to detect veiled interactions or determine actions for completing tasks. To address this challenge, we present GhostUI, a new dataset designed to enable the detection of hidden interactions in mobile applications. GhostUI provides before-and-after screenshots, simplified view hierarchies, gesture metadata, and task descriptions, allowing VLMs to better recognize concealed gestures and anticipate post-interaction states. Quantitative evaluations with VLMs show that models fine-tuned on GhostUI outperform baseline VLMs, particularly in predicting hidden interactions and inferring post-interaction screens, underscoring GhostUI's potential as a foundation for advancing mobile task automation.
Abstract (translated)
URL
https://arxiv.org/abs/2601.19258
PDF
-
3DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control
2026-01-26 12:57:36Xuanmeng Sha, Liyun Zhang, Tomohiro Mashita, Naoya Chiba, Yuki UranishiAbstract
Generating holistic co-speech gestures that integrate full-body motion with facial expressions suffers from semantically incoherent coordination on body motion and spatially unstable meaningless movements due to existing part-decomposed or frame-level regression methods, We introduce 3DGesPolicy, a novel action-based framework that reformulates holistic gesture generation as a continuous trajectory control problem through diffusion policy from robotics. By modeling frame-to-frame variations as unified holistic actions, our method effectively learns inter-frame holistic gesture motion patterns and ensures both spatially and semantically coherent movement trajectories that adhere to realistic motion manifolds. To further bridge the gap in expressive alignment, we propose a Gesture-Audio-Phoneme (GAP) fusion module that can deeply integrate and refine multi-modal signals, ensuring structured and fine-grained alignment between speech semantics, body motion, and facial expressions. Extensive quantitative and qualitative experiments on the BEAT2 dataset demonstrate the effectiveness of our 3DGesPolicy across other state-of-the-art methods in generating natural, expressive, and highly speech-aligned holistic gestures.
Abstract (translated)
URL
https://arxiv.org/abs/2601.18451
PDF
-
SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis
2026-01-26 09:35:10Xuan Wang, Siyuan Su, Quantong Fu, Yongxiang Hu, Yangfan ZhouAbstract
With the widespread adoption of Graphical User Interface (GUI) agents for automating GUI interaction tasks, substantial research focused on improving GUI perception to ground task instructions into concrete action steps. However, the step execution capability of these agents has gradually emerged as a new bottleneck for task completion. In particular, existing GUI agents often adopt overly simplified strategies for handling swipe interactions, preventing them from accurately replicating human-like behavior. To address this limitation, we decompose human swipe gestures into multiple quantifiable dimensions and propose an automated pipeline SwipeGen to synthesize human-like swipe interactions through GUI exploration. Based on this pipeline, we construct and release the first benchmark for evaluating the swipe execution capability of GUI agents. Furthermore, leveraging the synthesized data, we propose GUISwiper, a GUI agent with enhanced interaction execution capabilities. Experimental results demonstrate that GUISwiper achieves a swipe execution accuracy of 69.07%, representing a 214% improvement over existing VLM baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2601.18305
PDF
-
NeuroManip: Prosthetic Hand Manipulation System Based on EMG and Eye Tracking Powered by the Neuromorphic Processor AltAi
2026-01-25 20:44:03Roman Akinshin, Elizaveta Lopatina, Kirill Bogatikov, Nikolai Kiz, Anna V. Makarova, Mikhail Lebedev, Miguel Altamirano Cabrera, Dzmitry Tsetserukou, Valerii KanglerAbstract
This paper presents a novel neuromorphic control architecture for upper-limb prostheses that combines surface electromyography (sEMG) with gaze-guided computer vision. The system uses a spiking neural network deployed on the neuromorphic processor AltAi to classify EMG patterns in real time while an eye-tracking headset and scene camera identify the object within the user's focus. In our prototype, the same EMG recognition model that was originally developed for a conventional GPU is deployed as a spiking network on AltAi, achieving comparable accuracy while operating in a sub-watt power regime, which enables a lightweight, wearable implementation. For six distinct functional gestures recorded from upper-limb amputees, the system achieves robust recognition performance comparable to state-of-the-art myoelectric interfaces. When the vision pipeline restricts the decision space to three context-appropriate gestures for the currently viewed object, recognition accuracy increases to roughly 95% while excluding unsafe, object-inappropriate grasps. These results indicate that the proposed neuromorphic, context-aware controller can provide energy-efficient and reliable prosthesis control and has the potential to improve safety and usability in everyday activities for people with upper-limb amputation.
Abstract (translated)
URL
https://arxiv.org/abs/2601.17991
PDF
-
Real-Time Synchronized Interaction Framework for Emotion-Aware Humanoid Robots
2026-01-24 03:53:09Yanrong Chen, Xihan BianAbstract
As humanoid robots increasingly introduced into social scene, achieving emotionally synchronized multimodal interaction remains a significant challenges. To facilitate the further adoption and integration of humanoid robots into service roles, we present a real-time framework for NAO robots that synchronizes speech prosody with full-body gestures through three key innovations: (1) A dual-channel emotion engine where large language model (LLM) simultaneously generates context-aware text responses and biomechanically feasible motion descriptors, constrained by a structured joint movement library; (2) Duration-aware dynamic time warping for precise temporal alignment of speech output and kinematic motion keyframes; (3) Closed-loop feasibility verification ensuring gestures adhere to NAO's physical joint limits through real-time adaptation. Evaluations show 21% higher emotional alignment compared to rule-based systems, achieved by coordinating vocal pitch (arousal-driven) with upper-limb kinematics while maintaining lower-body stability. By enabling seamless sensorimotor coordination, this framework advances the deployment of context-aware social robots in dynamic applications such as personalized healthcare, interactive education, and responsive customer service platforms.
Abstract (translated)
URL
https://arxiv.org/abs/2601.17287
PDF
-
Glove2UAV: A Wearable IMU-Based Glove for Intuitive Control of UAV
2026-01-22 09:03:57Amir Habel, Ivan Snegirev, Elizaveta Semenyakina, Miguel Altamirano Cabrera, Jeffrin Sam, Fawad Mehboob, Roohan Ahmed Khan, Muhammad Ahsan Mustafa, Dzmitry TsetserukouAbstract
This paper presents Glove2UAV, a wearable IMU-glove interface for intuitive UAV control through hand and finger gestures, augmented with vibrotactile warnings for exceeding predefined speed thresholds. To promote safer and more predictable interaction in dynamic flight, Glove2UAV is designed as a lightweight and easily deployable wearable interface intended for real-time operation. Glove2UAV streams inertial measurements in real time and estimates palm and finger orientations using a compact processing pipeline that combines median-based outlier suppression with Madgwick-based orientation estimation. The resulting motion estimations are mapped to a small set of control primitives for directional flight (forward/backward and lateral motion) and, when supported by the platform, to object-interaction commands. Vibrotactile feedback is triggered when flight speed exceeds predefined threshold values, providing an additional alert channel during operation. We validate real-time feasibility by synchronizing glove signals with UAV telemetry in both simulation and real-world flights. The results show fast gesture-based command execution, stable coupling between gesture dynamics and platform motion, correct operation of the core command set in our trials, and timely delivery of vibratile warning cues.
Abstract (translated)
本文介绍了Glove2UAV，这是一种可穿戴的IMU（惯性测量单元）手套界面，通过手部和手指手势进行直观的无人机控制，并辅以振动反馈提醒超过预定义速度阈值的情况。为了促进动态飞行中更安全、更具预测性的交互，Glove2UAV被设计成轻便且易于部署的手持式接口，旨在实现实时操作。 Glove2UAV能够实现实时惯性测量数据流传输，并利用一个结合了基于中位数的异常值抑制和Madgwick姿态估计的方法的小型处理管道来估算手掌和手指的姿态。由此产生的运动估计被映射到一组有限的控制基本动作，用于方向飞行（前后和横向移动），并在平台支持的情况下映射为对象交互命令。 当飞行速度超过预定义阈值时，振动反馈会被触发，这在操作过程中提供了一种额外的警报渠道。通过在模拟环境和真实世界飞行中同步手套信号与无人机遥测数据，我们验证了其实时可行性。实验结果表明：基于手势的命令执行速度快；手势动态变化与平台运动之间耦合稳定；我们的试验中核心命令集运行正确；振动警告提示及时送达。
URL
https://arxiv.org/abs/2601.15775
PDF
-
Arabic Sign Language Recognition using Multimodal Approach
2026-01-20 09:21:43Ghadeer Alanazi, Abir BenabidAbstract
Arabic Sign Language (ArSL) is an essential communication method for individuals in the Deaf and Hard-of-Hearing community. However, existing recognition systems face significant challenges due to their reliance on single sensor approaches like Leap Motion or RGB cameras. These systems struggle with limitations such as inadequate tracking of complex hand orientations and imprecise recognition of 3D hand movements. This research paper aims to investigate the potential of a multimodal approach that combines Leap Motion and RGB camera data to explore the feasibility of recognition of ArSL. The system architecture includes two parallel subnetworks: a custom dense neural network for Leap Motion data, incorporating dropout and L2 regularization, and an image subnetwork based on a fine-tuned VGG16 model enhanced with data augmentation techniques. Feature representations from both modalities are concatenated in a fusion model and passed through fully connected layers, with final classification performed via SoftMax activation to analyze spatial and temporal features of hand gestures. The system was evaluated on a custom dataset comprising 18 ArSL words, of which 13 were correctly recognized, yielding an overall accuracy of 78%. These results offer preliminary insights into the viability of multimodal fusion for sign language recognition and highlight areas for further optimization and dataset expansion.
Abstract (translated)
URL
https://arxiv.org/abs/2601.17041
PDF
-
Dynamic Hand Gesture Recognition for Robot Manipulator Tasks
2026-01-19 10:17:20Dharmendra Sharma, Peeyush Thakur, Sandeep Gupta, Narendra Kumar Dhar, Laxmidhar BeheraAbstract
This paper proposes a novel approach to recognizing dynamic hand gestures facilitating seamless interaction between humans and robots. Here, each robot manipulator task is assigned a specific gesture. There may be several such tasks, hence, several gestures. These gestures may be prone to several dynamic variations. All such variations for different gestures shown to the robot are accurately recognized in real-time using the proposed unsupervised model based on the Gaussian Mixture model. The accuracy during training and real-time testing prove the efficacy of this methodology.
Abstract (translated)
本文提出了一种新颖的手势识别方法，旨在促进人类与机器人之间无缝交互。该方法为每个机器人操作任务分配一个特定手势。可能有许多这样的任务，因此也会有多个手势。这些手势可能会出现多种动态变化。使用基于高斯混合模型的无监督模型，可以准确地实时识别展示给机器人的不同手势的所有此类变化。在训练和实时测试期间的准确性证明了该方法的有效性。
URL
https://arxiv.org/abs/2601.12918
PDF
-
Rapport du Projet de Recherche TRAIMA
2026-01-19 08:55:50Julie Ran\c{c}on (UP, FoReLLIS, Poitiers), Jean-Fran\c{c}ois Cerisier (Techn\'e, Poitiers), Emilie Remond (Techn\'e, Poitiers), Aur\'elien Nguyen (Techn\'e, Poitiers), Andrew Peterson (Techn\'e, Poitiers), Ladjel Bellatreche (ISAE-ENSMA, IDD, A\&S)Abstract
The TRAIMA project (TRaitement Automatique des Interactions Multimodales en Apprentissage), conducted between March 2019 and June 2020, investigates the potential of automatic processing of multimodal interactions in educational settings. The project addresses a central methodological challenge in educational and interactional research: the analysis of verbal, paraverbal, and non-verbal data is currently carried out manually, making it extremely time-consuming and difficult to scale. TRAIMA explores how machine learning approaches could contribute to the categorisation and classification of such interactions. The project focuses specifically on explanatory and collaborative sequences occurring in classroom interactions, particularly in French as a Foreign Language (FLE) and French as a First Language (FLM) contexts. These sequences are analysed as inherently multimodal phenomena, combining spoken language with prosody, gestures, posture, gaze, and spatial positioning. A key theoretical contribution of the project is the precise linguistic and interactional definition of explanatory discourse as a tripartite sequence (opening, explanatory core, closure), drawing on discourse analysis and interactional linguistics. A substantial part of the research is devoted to the methodological foundations of transcription, which constitute a critical bottleneck for any form of automation. The report provides a detailed state of the art of existing transcription conventions (ICOR, Mondada, GARS, VALIBEL, Ferr{é}), highlighting their respective strengths and limitations when applied to multimodal classroom data. Through comparative analyses of manually transcribed sequences, the project demonstrates the inevitable variability and interpretative dimension of transcription practices, depending on theoretical positioning and analytical goals. Empirical work is based on several corpora, notably the INTER-EXPLIC corpus (approximately 30 hours of classroom interaction) and the EXPLIC-LEXIC corpus, which serve both as testing grounds for manual annotation and as reference datasets for future automation. Particular attention is paid to teacher gestures (kin{é}sic and proxemic resources), prosodic features, and their functional role in meaning construction and learner comprehension. The project also highlights the strategic role of the Techn{é}LAB platform, which provides advanced multimodal data capture (multi-camera video, synchronized audio, eye-tracking, digital interaction traces) and constitutes both a research infrastructure and a test environment for the development of automated tools. In conclusion, TRAIMA does not aim to deliver a fully operational automated system, but rather to establish a rigorous methodological framework for the automatic processing of multimodal pedagogical interactions. The project identifies transcription conventions, annotation categories, and analytical units that are compatible with machine learning approaches, while emphasizing the need for theoretical explicitness and researcher reflexivity. TRAIMA thus lays the groundwork for future interdisciplinary research at the intersection of didactics, discourse analysis, multimodality, and artificial intelligence in education.
Abstract (translated)
TRAIMA项目（2019年3月至2020年6月期间进行）旨在研究教育环境中自动处理多模态互动的潜力。该项目解决了一个在教育和互动研究中的核心方法论挑战：目前，对于言语、副语言以及非言语数据的分析都是手动完成的，这使得工作既耗时又难以扩展。TRAIMA探索了机器学习方法如何有助于此类交互的数据分类和归类。 项目特别关注课堂中解释性和协作性的序列，特别是在作为外语（FLE）和母语（FLM）背景下的法国语言教学环境中。这些序列被分析为多模态现象，结合口头言语、韵律、手势、姿态、目光接触以及空间定位等元素进行研究。项目的一个重要理论贡献在于精确地用三部分结构定义解释性话语：引入部分、核心阐释和结束部分，并借鉴了语篇分析与互动语言学的研究成果。 项目的重要一部分致力于转录方法论的基础，这是任何自动化形式的关键瓶颈之一。报告详述了现有转录惯例（如ICOR、Mondada、GARS、VALIBEL和Ferré）的现状，并突出显示这些惯例在应用于多模态教室数据时各自的优缺点与限制。 通过对比分析手动转录序列，项目展示了转录实践不可避免的变异性和解释维度，这取决于理论定位和分析目标。实证研究基于几个语料库，包括INTER-EXPLIC（大约30小时的课堂互动）和EXPLIC-LEXIC语料库，这些不仅用于测试人工注释方法，还作为未来自动化的参考数据集。 项目特别关注教师的手势（身体动作与空间距离资源）、韵律特征及其在意义构建和学习者理解中的功能角色。此外，项目强调了TechnéLAB平台的战略作用，该平台提供先进的多模态数据采集工具（包括多摄像头视频、同步音频、眼动追踪以及数字互动记录），并同时作为研究基础设施和自动化工具开发的测试环境。 最终，TRAIMA项目的目的是建立一个严格的自动处理多模态教育交互的方法论框架，而不是试图提供一套完全可操作的自动化系统。项目确定了与机器学习方法兼容的转录惯例、注释类别以及分析单位，并强调理论清晰度和研究者反思的重要性。因此，TRAIMA为跨学科研究奠定了基础，在这一领域中，教学法、语篇分析、多模态性及教育领域的人工智能相互交叉。
URL
https://arxiv.org/abs/2601.12844
PDF
-
VR$^2$: A Co-Located Dual-Headset Platform for Touch-Enabled Human-Robot Interaction Research
2026-01-18 13:11:41Chao Wang, Anna Belardinelli, Michael GiengerAbstract
Touch-rich human-robot interaction (HRI) is difficult to study: building and programming physical robots is costly and slow, while VR-based robot prototypes often remove physical contact or break the tight coupling between an agent's body and the user's felt touch. We present VR2VR, a co-located dual VR-headset platform for HRI research in which a participant and a hidden operator share the same physical space while experiencing different virtual embodiments. The participant sees an expressive virtual robot that interacts face-to-face in a shared virtual environment. In real time, the robot's upper-body gestures, head and gaze behaviors, and facial expressions are mapped from the operator's tracked motion and face signals. Because the operator is physically co-present and calibrated into the same coordinate frame, the operator can also physically touch the participant, enabling the participant to perceive robot touch aligned with the robot's hands; finger and hand motion are mapped to the robot using inverse kinematics to support precise contact. Beyond faithful motion retargeting for limb teleoperation, our VR2VR system supports experimental control by retargeting or selectively enabling nonverbal channels (e.g., head only vs. head+eyes vs. head+eyes+facial expressions) while keeping physical interaction constant. We detail the system design, calibration workflow, and safety considerations, and demonstrate the platform through a touch-based Wizard-of-Oz HRI study, illustrating how VR2VR lowers barriers for rapidly prototyping and rigorously evaluating embodied, touch-centric robot behaviors.
Abstract (translated)
触觉丰富的机器人交互（HRI）研究颇具挑战性：制造和编程实体机器人成本高昂且耗时，而基于虚拟现实的机器人原型往往会移除物理接触或破坏代理身体与用户感知到的触摸之间的紧密耦合。我们提出了一种名为VR2VR的双VR头显平台，用于共存空间中的HRI研究，在这种平台上，参与者和隐藏的操作员共享同一物理空间但拥有不同的虚拟化身。参与者可以看到一个具有表现力的虚拟机器人在共享的虚拟环境中进行面对面交互。通过实时映射操作员被追踪的动作及面部信号，机器人的上半身手势、头部与视线行为以及面部表情得以再现。由于操作者在同一坐标系中身体共存且已校准，因此他们可以物理地接触参与者，从而使参与者能够感知到与机器人手部动作相一致的触觉；手指和手臂运动通过逆向动力学映射至机器人的手部以支持精确接触。 除了为肢体远程操作提供忠实的动作重定向外，我们的VR2VR系统还通过重新定位或选择性启用非语言通道（例如仅头部、头部+眼睛或头部+眼睛+面部表情）来进行实验控制，同时保持物理交互不变。我们详细介绍了系统的架构设计、校准工作流程以及安全考虑，并通过基于触觉的“魔术师之助”HRI研究展示了该平台的能力，说明了VR2VR如何降低快速原型制作和严格评估以触摸为中心的机器人行为的门槛。
URL
https://arxiv.org/abs/2601.12395
PDF
-
Model selection and real-time skill assessment for suturing in robotic surgery
2026-01-17 11:28:59Zhaoyang Jacopo Hu, Alex Ranne, Alaa Eldin Abdelaal, Kiran Bhattacharyya, Etienne Burdet, Allison M. Okamura, Ferdinando Rodriguez y BaenaAbstract
Automated feedback systems have the potential to provide objective skill assessment for training and evaluation in robot-assisted surgery. In this study, we examine methods to achieve real-time prediction of surgical skill level in real-time based on Objective Structured Assessment of Technical Skills (OSATS) scores. Using data acquired from the da Vinci Surgical System, we carry out three main analyses, focusing on model design, their real-time performance, and their skill-level-based cross-validation training. For the model design, we evaluate the effectiveness of multimodal deep learning models for predicting surgical skill levels using synchronized kinematic and vision data. Our models include separate unimodal baselines and fusion architectures that integrate features from both modalities and are evaluated using mean Spearman's correlation coefficients, demonstrating that the fusion model consistently outperforms unimodal models for real-time predictions. For the real-time performance, we observe the prediction's trend over time and highlight correlation with the surgeon's gestures. For the skill-level-based cross-validation, we separately trained models on surgeons with different skill levels, which showed that high-skill demonstrations allow for better performance than those trained on low-skilled ones and generalize well to similarly skilled participants. Our findings show that multimodal learning allows more stable fine-grained evaluation of surgical performance and highlights the value of expert-level training data for model generalization.
Abstract (translated)
自动化反馈系统有潜力为机器人辅助手术中的培训和评估提供客观的技能评定。在这项研究中，我们探讨了基于客观结构化技术评分（OSATS）得分实现实时预测外科医生技能水平的方法。利用达芬奇外科系统的数据，我们进行了三项主要分析：模型设计、其实时性能以及基于技能水平的交叉验证训练。 在模型设计方面，我们评估了多模态深度学习模型的有效性，这些模型使用同步运动学和视觉数据来预测手术技能水平。我们的模型包括单一模式基准线及融合架构，后者整合了两种模式的特征，并通过平均Spearman相关系数进行评价。结果显示，融合模型在实时预测中始终优于单模式模型。 对于实时性能分析，我们观察了随时间变化的预测趋势，并强调与外科医生手势的相关性。在基于技能水平的交叉验证中，我们在不同技能水平的外科医生上分别训练了模型，结果表明，由高技能演示培训的模型比那些基于低技能数据训练的模型表现更好且能很好地泛化至类似技能水平的参与者。 我们的研究发现表明，多模态学习能够进行更为稳定细致的手术性能评估，并强调专家级训练数据对模型泛化的价值。
URL
https://arxiv.org/abs/2601.12012
PDF
-
An AI-IoT Based Smart Wheelchair with Gesture-Controlled Mobility, Deep Learning-Based Obstacle Detection, Multi-Sensor Health Monitoring, and Emergency Alert System
2026-01-17 09:37:11Md. Asiful Islam, Abdul Hasib, Tousif Mahmud Emon, Khandaker Tabin Hasan, A. S. M. Ahsanul Sarkar AkibAbstract
The growing number of differently-abled and elderly individuals demands affordable, intelligent wheelchairs that combine safe navigation with health monitoring. Traditional wheelchairs lack dynamic features, and many smart alternatives remain costly, single-modality, and limited in health integration. Motivated by the pressing demand for advanced, personalized, and affordable assistive technologies, we propose a comprehensive AI-IoT based smart wheelchair system that incorporates glove-based gesture control for hands-free navigation, real-time object detection using YOLOv8 with auditory feedback for obstacle avoidance, and ultrasonic for immediate collision avoidance. Vital signs (heart rate, SpO$_2$, ECG, temperature) are continuously monitored, uploaded to ThingSpeak, and trigger email alerts for critical conditions. Built on a modular and low-cost architecture, the gesture control achieved a 95.5\% success rate, ultrasonic obstacle detection reached 94\% accuracy, and YOLOv8-based object detection delivered 91.5\% Precision, 90.2\% Recall, and a 90.8\% F1-score. This integrated, multi-modal approach offers a practical, scalable, and affordable solution, significantly enhancing user autonomy, safety, and independence by bridging the gap between innovative research and real-world deployment.
Abstract (translated)
越来越多的残障人士和老年人需要一种既经济又智能的轮椅，这种轮椅能够结合安全导航与健康监测。传统轮椅缺乏动态特性，而许多智能化替代品依然昂贵，并且大多局限于单一功能并且在健康整合方面有限。 鉴于对先进的、个性化的以及可负担得起的辅助技术的迫切需求，我们提出了一种基于AI-IoT（人工智能和物联网）的综合智能轮椅系统。该系统包括用于手控自由导航的手套控制装置，使用YOLOv8进行实时物体检测，并结合听觉反馈来避开障碍物；同时采用超声波传感器以实现即时碰撞避免功能。此外，该系统还能够连续监测心率、血氧饱和度（SpO2）、心电图（ECG）和体温等生命体征信息，并将这些数据上传至ThingSpeak平台，在出现紧急情况时触发电子邮件警报。 我们的设计基于模块化且成本低廉的架构：手部控制装置达到了95.5%的成功率，超声波障碍物检测准确率达到94%，而基于YOLOv8的目标检测则分别实现了91.5% 的精确度、90.2% 的召回率和90.8% 的F1得分。 这种集成的多模式方法提供了一种实用且可扩展的解决方案，以合理的价格显著提升了用户的自主性、安全性和独立性，从而在创新研究与实际应用之间架起了桥梁。
URL
https://arxiv.org/abs/2601.11983
PDF
-
Telling Human and Machine Handwriting Apart
2026-01-16 18:45:16Luis A. Leiva, Moises Diaz, Nuwan T. Attygalle, Miguel A. Ferrer, Rejean PlamondonAbstract
Handwriting movements can be leveraged as a unique form of behavioral biometrics, to verify whether a real user is operating a device or application. This task can be framed as a reverse Turing test in which a computer has to detect if an input instance has been generated by a human or artificially. To tackle this task, we study ten public datasets of handwritten symbols (isolated characters, digits, gestures, pointing traces, and signatures) that are artificially reproduced using seven different synthesizers, including, among others, the Kinematic Theory (Sigma h model), generative adversarial networks, Transformers, and Diffusion models. We train a shallow recurrent neural network that achieves excellent performance (98.3 percent Area Under the ROC Curve (AUC) score and 1.4 percent equal error rate on average across all synthesizers and datasets) using nonfeaturized trajectory data as input. In few-shot settings, we show that our classifier achieves such an excellent performance when trained on just 10 percent of the data, as evaluated on the remaining 90% of the data as a test set. We further challenge our classifier in out-of-domain settings, and observe very competitive results as well. Our work has implications for computerized systems that need to verify human presence, and adds an additional layer of security to keep attackers at bay.
Abstract (translated)
手写动作可以被用作一种独特的行为生物识别方式，以验证实际用户是否在操作设备或应用程序。这一任务可以被视为一个反向图灵测试，在这种测试中，计算机需要检测输入实例是由真人生成的还是人工合成的。为了解决这个问题，我们研究了十个公开的手写符号数据集（包括孤立字符、数字、手势、指点轨迹和签名），这些数据集使用七种不同的生成器进行了人工再现，其中包括动学理论（Sigma h 模型）、生成对抗网络、Transformer 和扩散模型等。 我们训练了一个浅层循环神经网络，在不进行特征化处理的情况下以轨迹数据为输入，该网络在所有合成器和数据集中实现了卓越的性能（98.3% 的接收者操作特性曲线（ROC）面积得分和1.4% 的平均错误接受率）。此外，我们在少量样本设置下展示了当训练集仅为总数据量的10%，而剩余90%的数据作为测试集时，我们的分类器也能实现如此卓越的表现。我们进一步在跨域场景中挑战了这一分类器，并观察到了非常具有竞争力的结果。 这项工作对需要验证人类存在的计算机系统有着重要的影响，并为防止攻击者侵入添加了一层额外的安全保障。
URL
https://arxiv.org/abs/2601.11700
PDF
-
TinyMyo: a Tiny Foundation Model for Flexible EMG Signal Processing at the Edge
2026-01-15 18:05:25Matteo Fasulo, Giusy Spacone, Thorir Mar Ingolfsson, Yawei Li, Luca Benini, Andrea CossettiniAbstract
Objective: Surface electromyography (EMG) is a non-invasive sensing modality widely used in biomechanics, rehabilitation, prosthetic control, and human-machine interfaces. Despite decades of use, achieving robust generalization across subjects, recording systems, and acquisition protocols remains challenging. While foundation models (FMs) are gaining traction for EMG, existing approaches remain limited to single downstream tasks and lack deployability on embedded platforms. This work addresses these limitations. Methods: We present TinyMyo, a lightweight FM based on a Transformer encoder architecture. The model is pre-trained in a self-supervised manner using masked reconstruction on publicly available datasets. With only 3.6M parameters, TinyMyo is designed to support multiple downstream tasks through minimal task-specific head adaptations. Results: We demonstrate generalization across hand gesture classification, hand kinematic regression, speech production and speech recognition, with performance comparable to or surpassing the state of the art (SoA), and model size below 5M parameters. We achieve SoA results compared to previous FM-based works on the NinaPro DB5 (89.4%), UCI-EMG (97.56%), and EPN-612 (96.74%) datasets. We demonstrate the first-time deployment of an EMG FM on an ultra-low power microcontroller (GAP9), with an inference time of 0.785 s, energy of 44.91 mJ and power envelope of 57.18 mW. Conclusion: TinyMyo demonstrates that compact, self-supervised EMG FM can guarantee strong generalization across multiple downstream tasks while remaining compatible with low-power edge devices. Significance: TinyMyo is the first EMG FM for ultra-low power edge devices, enabling scalable and energy-efficient sensing for motor intent decoding, neuromuscular assessment, and biosignal driven human-machine interaction.
Abstract (translated)
目标：表面肌电图（EMG）是一种广泛应用于生物力学、康复、假肢控制和人机界面的非侵入式传感模式。尽管经过几十年的应用，实现跨受试者、记录系统及采集协议的良好泛化仍然具有挑战性。虽然基础模型（FMs）在EMG应用中正日益受到重视，但现有方法依然局限于单一下游任务，并且无法部署于嵌入式平台上。这项工作旨在解决这些限制。 方法：我们提出了TinyMyo，一种基于Transformer编码器架构的轻量级基础模型。该模型通过公开可用的数据集上自监督的方式进行预训练，采用掩码重构方式。仅包含3.6M参数的小型模型被设计为能够支持多种下游任务，并且只需要对特定任务头部进行微小调整即可。 结果：我们展示了TinyMyo在手部手势分类、手部运动学回归、语音生成和识别等领域的泛化能力，其性能与现有最佳方法相当或更优，同时模型规模低于5M参数。在NinaPro DB5（89.4%）、UCI-EMG（97.56%）以及EPN-612（96.74%）数据集上相比之前的基于基础模型的工作，我们的结果达到了现有最佳水平。此外，我们首次将一个EMG基础模型部署到了超低功耗微控制器（GAP9）上，在推理时间仅为0.785秒、能耗为44.91 mJ和功率上限为57.18 mW的情况下实现了这一目标。 结论：TinyMyo表明，紧凑的自监督EMG基础模型能够在多种下游任务中保证强大的泛化能力，并且兼容低功耗边缘设备。意义：TinyMyo是首个应用于超低功耗边缘设备的EMG基础模型，这使得运动意图解码、神经肌肉评估及生物信号驱动的人机交互能够实现可扩展和节能的目标感测技术。
URL
https://arxiv.org/abs/2512.15729
PDF
-
When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions
2026-01-13 15:21:08Mingyu Zhu, Jiangong Chen, Bin LiAbstract
Extended Reality (XR), including virtual, augmented, and mixed reality, provides immersive and interactive experiences across diverse applications, from VR-based education to AR-based assistance and MR-based training. However, widespread XR adoption remains limited due to two key challenges: 1) the high cost and complexity of authoring 3D content, especially for large-scale environments or complex interactions; and 2) the steep learning curve associated with non-intuitive interaction methods like handheld controllers or scripted gestures. Generative AI (GenAI) presents a promising solution by enabling intuitive, language-driven interaction and automating content generation. Leveraging vision-language models and diffusion-based generation, GenAI can interpret ambiguous instructions, understand physical scenes, and generate or manipulate 3D content, significantly lowering barriers to XR adoption. This paper explores the integration of XR and GenAI through three concrete use cases, showing how they address key obstacles in scalability and natural interaction, and identifying technical challenges that must be resolved to enable broader adoption.
Abstract (translated)
扩展现实（XR），包括虚拟现实（VR）、增强现实（AR）和混合现实（MR），为各种应用场景提供了沉浸式和互动体验，从基于 VR 的教育到基于 AR 的辅助功能以及基于 MR 的培训。然而，由于两个关键挑战，广泛采用 XR 仍然受到限制：1) 创作三维内容的高成本和复杂性，特别是对于大规模环境或复杂的交互；2) 手持控制器或脚本化手势等非直观互动方法的学习曲线陡峭。 生成式人工智能（GenAI）通过支持直觉化的语言驱动互动以及自动内容生成提供了一种有前景的解决方案。利用视觉-语言模型和扩散生成技术，GenAI 可以解读模糊指令、理解物理场景，并生成或操作三维内容，显著降低 XR 采用的门槛。本文探讨了通过三个具体用例将 XR 和 GenAI 集成的方法，展示了它们如何解决可扩展性和自然互动的关键障碍，并确定了必须解决的技术挑战，以便实现更广泛的采用。
URL
https://arxiv.org/abs/2601.15308
PDF
-
Teaching Robots Like Dogs: Learning Agile Navigation from Luring, Gesture, and Speech
2026-01-13 10:47:12Taerim Yoon, Dongho Kang, Jin Cheng, Fatemeh Zargarbashi, Yijiang Huang, Minsung Ahn, Stelian Coros, Sungjoon ChoiAbstract
In this work, we aim to enable legged robots to learn how to interpret human social cues and produce appropriate behaviors through physical human guidance. However, learning through physical engagement can place a heavy burden on users when the process requires large amounts of human-provided data. To address this, we propose a human-in-the-loop framework that enables robots to acquire navigational behaviors in a data-efficient manner and to be controlled via multimodal natural human inputs, specifically gestural and verbal commands. We reconstruct interaction scenes using a physics-based simulation and aggregate data to mitigate distributional shifts arising from limited demonstration data. Our progressive goal cueing strategy adaptively feeds appropriate commands and navigation goals during training, leading to more accurate navigation and stronger alignment between human input and robot behavior. We evaluate our framework across six real-world agile navigation scenarios, including jumping over or avoiding obstacles. Our experimental results show that our proposed method succeeds in almost all trials across these scenarios, achieving a 97.15% task success rate with less than 1 hour of demonstration data in total.
Abstract (translated)
在这项工作中，我们旨在使腿部机器人通过物理人类指导学会解读人类社交线索并产生适当的行为。然而，通过身体互动进行学习可能会给用户带来沉重的负担，特别是当过程需要大量的人类提供的数据时。为了解决这个问题，我们提出了一种人机交互框架，使得机器人能够在数据高效的方式下获取导航行为，并能够接受多模态自然人类输入（具体来说是手势和口头命令）。我们使用基于物理的模拟重建互动场景，并汇总数据以缓解由于演示数据有限而产生的分布变化。我们的渐进式目标提示策略在训练过程中适应性地提供适当的指令和导航目标，从而导致更准确的导航以及人机交互与机器人行为之间更强的一致性。 我们在六个现实世界中的敏捷导航场景中评估了该框架，包括跳跃或避开障碍物的情况。实验结果显示，在这些场景下，我们的方法几乎都在所有试验中取得成功，并且在总演示数据不到1小时的情况下达到了97.15%的任务成功率。
URL
https://arxiv.org/abs/2601.08422
PDF
