Gesture
-
MSF-Mamba: Motion-aware State Fusion Mamba for Efficient Micro-Gesture Recognition
2025-10-12 07:16:58Deng Li, Jun Shao, Bohao Xing, Rong Gao, Bihan Wen, Heikki K\"alvi\"ainen, Xin LiuAbstract
Micro-gesture recognition (MGR) targets the identification of subtle and fine-grained human motions and requires accurate modeling of both long-range and local spatiotemporal dependencies. While CNNs are effective at capturing local patterns, they struggle with long-range dependencies due to their limited receptive fields. Transformer-based models address this limitation through self-attention mechanisms but suffer from high computational costs. Recently, Mamba has shown promise as an efficient model, leveraging state space models (SSMs) to enable linear-time processing However, directly applying the vanilla Mamba to MGR may not be optimal. This is because Mamba processes inputs as 1D sequences, with state updates relying solely on the previous state, and thus lacks the ability to model local spatiotemporal dependencies. In addition, previous methods lack a design of motion-awareness, which is crucial in MGR. To overcome these limitations, we propose motion-aware state fusion mamba (MSF-Mamba), which enhances Mamba with local spatiotemporal modeling by fusing local contextual neighboring states. Our design introduces a motion-aware state fusion module based on central frame difference (CFD). Furthermore, a multiscale version named MSF-Mamba+ has been proposed. Specifically, MSF-Mamba supports multiscale motion-aware state fusion, as well as an adaptive scale weighting module that dynamically weighs the fused states across different scales. These enhancements explicitly address the limitations of vanilla Mamba by enabling motion-aware local spatiotemporal modeling, allowing MSF-Mamba and MSF-Mamba to effectively capture subtle motion cues for MGR. Experiments on two public MGR datasets demonstrate that even the lightweight version, namely, MSF-Mamba, achieves SoTA performance, outperforming existing CNN-, Transformer-, and SSM-based models while maintaining high efficiency.
Abstract (translated)
微手势识别（Micro-gesture Recognition，MGR）的目标是识别细微且精细的人体动作，并需要准确建模长距离和局部的时空依赖关系。尽管卷积神经网络（CNNs）在捕捉局部模式方面表现出色，但由于其有限的感受野，它们难以处理长程依赖性。基于Transformer的模型通过自注意力机制解决了这一局限性，但面临着高计算成本的问题。最近，Mamba作为一种高效模型显示出潜力，它利用状态空间模型（State Space Models, SSMs）实现线性时间处理。然而，直接将标准的Mamba应用于微手势识别可能并非最佳选择，因为Mamba以一维序列方式处理输入，并且状态更新仅依赖于前一个状态，因此无法建模局部时空依赖关系。此外，先前的方法缺乏运动感知设计，在微手势识别中这一点至关重要。 为克服这些限制，我们提出了一种基于运动感知的状态融合mamba（Motion-aware State Fusion mamba, MSF-Mamba），通过融合本地上下文邻近状态来增强Mamba的局部时空建模能力。我们的设计方案引入了一个基于中央帧差（Central Frame Difference, CFD）的运动感知状态融合模块。此外，还提出了一种多尺度版本MSF-Mamba+，它支持多尺度的运动感知状态融合以及一个自适应缩放权重模块，在不同尺度之间动态加权融合后的状态。 这些改进明确解决了标准Mamba模型的局限性，通过启用基于运动的状态融合实现了局部时空建模。这使得MSF-Mamba及其扩展版本能够有效捕捉细微的动作线索，用于微手势识别任务中。在两个公开的微手势数据集上的实验表明，即便是轻量级版本的MSF-Mamba也达到了最先进的性能水平，并且优于现有的CNN、Transformer和SSM模型，同时保持了高效率。 总结来说，所提出的MSF-Mamba及其多尺度版本通过引入运动感知机制显著提高了在长程与局部时空依赖建模中的有效性，展示了微手势识别任务中卓越的性能与效率。
URL
https://arxiv.org/abs/2510.10478
PDF
-
VividAnimator: An End-to-End Audio and Pose-driven Half-Body Human Animation Framework
2025-10-11 16:04:56Donglin Huang, Yongyuan Li, Tianhang Liu, Junming Huang, Xiaoda Yang, Chi Wang, Weiwei XuAbstract
Existing for audio- and pose-driven human animation methods often struggle with stiff head movements and blurry hands, primarily due to the weak correlation between audio and head movements and the structural complexity of hands. To address these issues, we propose VividAnimator, an end-to-end framework for generating high-quality, half-body human animations driven by audio and sparse hand pose conditions. Our framework introduces three key innovations. First, to overcome the instability and high cost of online codebook training, we pre-train a Hand Clarity Codebook (HCC) that encodes rich, high-fidelity hand texture priors, significantly mitigating hand degradation. Second, we design a Dual-Stream Audio-Aware Module (DSAA) to model lip synchronization and natural head pose dynamics separately while enabling interaction. Third, we introduce a Pose Calibration Trick (PCT) that refines and aligns pose conditions by relaxing rigid constraints, ensuring smooth and natural gesture transitions. Extensive experiments demonstrate that Vivid Animator achieves state-of-the-art performance, producing videos with superior hand detail, gesture realism, and identity consistency, validated by both quantitative metrics and qualitative evaluations.
Abstract (translated)
现有的音频和姿态驱动的人体动画方法通常在头部僵硬运动和手部模糊方面存在问题，主要是由于音频与头部动作之间的相关性较弱以及手部结构的复杂性。为了解决这些问题，我们提出了VividAnimator，这是一个端到端框架，用于生成高质量、半身人体动画，这些动画由音频和稀疏的手部姿态条件驱动。 我们的框架引入了三个关键创新点： 1. 为了克服在线代码库训练中的不稳定性和高成本问题，我们预训练了一个手清晰度代码库（HCC），它编码了丰富的高保真手部纹理先验知识，大大减少了手部的退化现象。 2. 我们设计了一种双流音频感知模块（DSAA），能够分别建模唇同步和自然头部姿态动态变化，并且使它们可以相互作用。 3. 我们引入了一个姿态校准技巧（PCT），通过放松刚性约束来细化并对齐姿态条件，从而确保手势过渡的平滑性和自然度。 广泛的实验表明，Vivid Animator 达到了最先进的性能水平，在手部细节、动作现实感和身份一致性方面表现出优越的表现，并且这些结果不仅由定量指标验证，还通过定性评估得到了确认。
URL
https://arxiv.org/abs/2510.10269
PDF
-
Multi Class Parkinsons Disease Detection Based on Finger Tapping Using Attention-Enhanced CNN BiLSTM
2025-10-11 09:02:14Abu Saleh Musa Miah, Najmul Hassan, Md Maruf Al Hossain, Yuichi Okuyama, Jungpil ShinAbstract
Effective clinical management and intervention development depend on accurate evaluation of Parkinsons disease (PD) severity. Many researchers have worked on developing gesture-based PD recognition systems; however, their performance accuracy is not satisfactory. In this study, we propose a multi-class Parkinson Disease detection system based on finger tapping using an attention-enhanced CNN BiLSTM. We collected finger tapping videos and derived temporal, frequency, and amplitude based features from wrist and hand movements. Then, we proposed a hybrid deep learning framework integrating CNN, BiLSTM, and attention mechanisms for multi-class PD severity classification from video-derived motion features. First, the input sequence is reshaped and passed through a Conv1D MaxPooling block to capture local spatial dependencies. The resulting feature maps are fed into a BiLSTM layer to model temporal dynamics. An attention mechanism focuses on the most informative temporal features, producing a context vector that is further processed by a second BiLSTM layer. CNN-derived features and attention-enhanced BiLSTM outputs are concatenated, followed by dense and dropout layers, before the final softmax classifier outputs the predicted PD severity level. The model demonstrated strong performance in distinguishing between the five severity classes, suggesting that integrating spatial temporal representations with attention mechanisms can improve automated PD severity detection, making it a promising non-invasive tool to support clinicians in PD monitoring and progression tracking.
Abstract (translated)
有效的临床管理和干预措施开发依赖于对帕金森病（PD）严重程度的准确评估。许多研究人员致力于开发基于手势的PD识别系统，但它们的表现准确性不尽如人意。本研究提出了一种基于手指敲击动作，并结合注意力增强的CNN BiLSTM的多类帕金森病检测系统。 我们收集了手指敲击视频，并从手腕和手部的动作中提取出时域、频域和幅度相关特征。然后，我们提出了一个集成CNN（卷积神经网络）、BiLSTM（双向长短时记忆网络）和注意力机制的混合深度学习框架，用于基于视频动作特征进行多类PD严重程度分类。 首先，输入序列被重新塑形并通过Conv1D MaxPooling块传递以捕捉局部空间依赖性。生成的特征图通过一个BiLSTM层来建模时间动态特性。注意力机制聚焦于最具有信息量的时间特征，产生上下文向量，并进一步由第二层BiLSTM处理。随后将CNN衍生出的特征与增强后的BiLSTM输出进行拼接，之后经过密集和dropout层，最后通过softmax分类器输出预测的PD严重程度等级。 该模型在区分五个严重程度类别方面表现出了强大的性能，表明结合空间时间表示和注意力机制可以提高自动化PD严重度检测能力。这使其成为支持临床医生用于PD监控及病情跟踪的一项有前景的非侵入性工具。
URL
https://arxiv.org/abs/2510.10121
PDF
-
mmJoints: Expanding Joint Representations Beyond in mmWave-Based 3D Pose Estimation
2025-10-10 03:24:56Zhenyu Wang, Mahathir Monjur, Shahriar NirjonAbstract
In mmWave-based pose estimation, sparse signals and weak reflections often cause models to infer body joints from statistical priors rather than sensor data. While prior knowledge helps in learning meaningful representations, over-reliance on it degrades performance in downstream tasks like gesture and activity recognition. In this paper, we introduce mmJoints, a framework that augments a pre-trained, black-box mmWave-based 3D pose estimator's output with additional joint descriptors. Rather than mitigating bias, mmJoints makes it explicit by estimating the likelihood of a joint being sensed and the reliability of its predicted location. These descriptors enhance interpretability and improve downstream task accuracy. Through extensive evaluations using over 115,000 signal frames across 13 pose estimation settings, we show that mmJoints estimates descriptors with an error rate below 4.2%. mmJoints also improves joint position accuracy by up to 12.5% and boosts activity recognition by up to 16% over state-of-the-art methods.
Abstract (translated)
在基于毫米波（mmWave）的姿态估计中，稀疏信号和弱反射经常导致模型从统计先验而非传感器数据来推断人体关节位置。虽然先验知识有助于学习有意义的表示，但过度依赖它会降低手势识别和活动识别等下游任务的表现。本文介绍了 mmJoints 框架，该框架通过添加额外的关节描述符来增强预训练、黑盒 mmWave 基于 3D 姿势估计器的输出。mmJoints 不是减轻偏见，而是使其明确化，通过对一个关节被感知的概率和其预测位置的可靠性进行评估来体现这一点。这些描述符增强了模型的可解释性，并提高了下游任务的准确性。 通过使用超过115,000个信号帧，在13种姿态估计设置下进行了广泛的实验评估，结果表明 mmJoints 估算的描述符错误率低于4.2%。此外，mmJoints 还将关节位置精度提高了最多12.5%，并将活动识别性能提升了高达16%，优于最先进的方法。
URL
https://arxiv.org/abs/2510.08970
PDF
-
Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agents
2025-10-06 09:41:37Zeyi Zhang, Yanju Zhou, Heyuan Yao, Tenglong Ao, Xiaohang Zhan, Libin LiuAbstract
We present Social Agent, a novel framework for synthesizing realistic and contextually appropriate co-speech nonverbal behaviors in dyadic conversations. In this framework, we develop an agentic system driven by a Large Language Model (LLM) to direct the conversation flow and determine appropriate interactive behaviors for both participants. Additionally, we propose a novel dual-person gesture generation model based on an auto-regressive diffusion model, which synthesizes coordinated motions from speech signals. The output of the agentic system is translated into high-level guidance for the gesture generator, resulting in realistic movement at both the behavioral and motion levels. Furthermore, the agentic system periodically examines the movements of interlocutors and infers their intentions, forming a continuous feedback loop that enables dynamic and responsive interactions between the two participants. User studies and quantitative evaluations show that our model significantly improves the quality of dyadic interactions, producing natural, synchronized nonverbal behaviors.
Abstract (translated)
我们介绍了Social Agent，这是一个用于生成真实且符合情境的对话语音非言语行为的创新框架。在该框架中，我们开发了一个由大型语言模型（LLM）驱动的代理系统来引导对话流程并确定双方参与者适当的互动行为。此外，我们提出了一种基于自回归扩散模型的新颖双人手势生成模型，该模型可根据语音信号合成协调的动作。代理系统的输出被转换为对动作生成器的高层次指导，从而在行为和运动层面产生真实感的动作。此外，代理系统会定期检查对话双方的动作并推断其意图，形成一个持续反馈循环，使两者的互动更加动态且响应迅速。用户研究和定量评估表明，我们的模型显著提升了双边交互的质量，产生了自然、同步的非言语行为。
URL
https://arxiv.org/abs/2510.04637
PDF
-
Evaluating High-Resolution Piano Sustain Pedal Depth Estimation with Musically Informed Metrics
2025-10-04 09:29:52Hanwen Zhang, Kun Fang, Ziyu Wang, Ichiro FujinagaAbstract
Evaluation for continuous piano pedal depth estimation tasks remains incomplete when relying only on conventional frame-level metrics, which overlook musically important features such as direction-change boundaries and pedal curve contours. To provide more interpretable and musically meaningful insights, we propose an evaluation framework that augments standard frame-level metrics with an action-level assessment measuring direction and timing using segments of press/hold/release states and a gesture-level analysis that evaluates contour similarity of each press-release cycle. We apply this framework to compare an audio-only baseline with two variants: one incorporating symbolic information from MIDI, and another trained in a binary-valued setting, all within a unified architecture. Results show that the MIDI-informed model significantly outperforms the others at action and gesture levels, despite modest frame-level gains. These findings demonstrate that our framework captures musically relevant improvements indiscernible by traditional metrics, offering a more practical and effective approach to evaluating pedal depth estimation models.
Abstract (translated)
仅依赖传统的帧级指标来评估连续钢琴踏板深度估计任务是不完整的，因为这些指标忽略了诸如方向变化边界和踏板曲线轮廓等重要的音乐特征。为了提供更可解释且具有音乐意义的见解，我们提出了一种评估框架，该框架在标准帧级度量的基础上增加了动作级别评估（通过按压/保持/释放状态的片段来衡量方向和时间）以及手势级别的分析（评估每个按压-释放周期中的轮廓相似性）。我们将此框架应用于比较仅基于音频的方法与两个变体：一个结合了来自MIDI的符号信息，另一个在二进制值设置中训练，并且所有模型均采用了统一架构。实验结果表明，尽管帧级性能提升不大，但MIDI增强型模型在动作和手势级别上显著优于其他模型。这些发现证明我们的框架能够捕捉到传统度量无法察觉的重要音乐改进，从而提供了一种更实用有效的评估踏板深度估计模型的方法。
URL
https://arxiv.org/abs/2510.03750
PDF
-
Towards Intuitive Human-Robot Interaction through Embodied Gesture-Driven Control with Woven Tactile Skins
2025-09-30 08:46:17ChunPing Lam, Xiangjia Chen, Chenming Wu, Hao Chen, Binzhi Sun, Guoxin Fang, Charlie C. L. Wang, Chengkai Dai, Yeung YamAbstract
This paper presents a novel human-robot interaction (HRI) framework that enables intuitive gesture-driven control through a capacitance-based woven tactile skin. Unlike conventional interfaces that rely on panels or handheld devices, the woven tactile skin integrates seamlessly with curved robot surfaces, enabling embodied interaction and narrowing the gap between human intent and robot response. Its woven design combines fabric-like flexibility with structural stability and dense multi-channel sensing through the interlaced conductive threads. Building on this capability, we define a gesture-action mapping of 14 single- and multi-touch gestures that cover representative robot commands, including task-space motion and auxiliary functions. A lightweight convolution-transformer model designed for gesture recognition in real time achieves an accuracy of near-100%, outperforming prior baseline approaches. Experiments on robot arm tasks, including pick-and-place and pouring, demonstrate that our system reduces task completion time by up to 57% compared with keyboard panels and teach pendants. Overall, our proposed framework demonstrates a practical pathway toward more natural and efficient embodied HRI.
Abstract (translated)
本文提出了一种新颖的人机交互（HRI）框架，该框架通过基于电容的编织触觉皮肤实现了直观的手势驱动控制。与依赖于面板或手持设备的传统界面不同，这种编织触觉皮肤可以无缝集成到机器人曲面表面中，从而实现具身化互动，并缩小了人类意图和机器人响应之间的差距。其编织设计结合了类似织物的灵活性、结构稳定性以及通过交织导电线获得的密集多通道感应能力。 在此基础上，我们定义了一个由14种单触点及多触点手势组成的动作映射，涵盖了包括任务空间运动和辅助功能在内的代表性机器人指令。为了实现实时的手势识别，我们设计了一种轻量级卷积-变压器模型，并且该模型达到了接近百分之百的准确率，在实时性能上优于之前的方法。 通过在机器人手臂任务中的实验（如取放物品和倒水），我们的系统与键盘面板以及教学挂件相比，在完成这些任务的时间上最多减少了57%。总的来说，我们提出的框架为实现更加自然且高效的具身化HRI提供了一条实用途径。
URL
https://arxiv.org/abs/2509.25951
PDF
-
FinCap: Topic-Aligned Captions for Short-Form Financial YouTube Videos
2025-09-30 04:04:41Siddhant Sukhani, Yash Bhardwaj, Riya Bhadani, Veer Kejriwal, Michael Galarnyk, Sudheer ChavaAbstract
We evaluate multimodal large language models (MLLMs) for topic-aligned captioning in financial short-form videos (SVs) by testing joint reasoning over transcripts (T), audio (A), and video (V). Using 624 annotated YouTube SVs, we assess all seven modality combinations (T, A, V, TA, TV, AV, TAV) across five topics: main recommendation, sentiment analysis, video purpose, visual analysis, and financial entity recognition. Video alone performs strongly on four of five topics, underscoring its value for capturing visual context and effective cues such as emotions, gestures, and body language. Selective pairs such as TV or AV often surpass TAV, implying that too many modalities may introduce noise. These results establish the first baselines for financial short-form video captioning and illustrate the potential and challenges of grounding complex visual cues in this domain. All code and data can be found on our Github under the CC-BY-NC-SA 4.0 license.
Abstract (translated)
我们通过测试跨转录文本（T）、音频（A）和视频（V）的联合推理，评估了多模态大型语言模型（MLLMs）在金融短视频（SVs）中进行主题一致字幕生成的能力。使用624个标注过的YouTube SV数据集，我们在五个话题上对所有七种模态组合（T、A、V、TA、TV、AV、TAV）进行了评估：主要推荐、情感分析、视频目的、视觉分析和金融实体识别。单独的视频在五项中的四项表现突出，强调了捕捉视觉环境和有效线索（如情绪、手势和肢体语言）的重要性。而选择性的模态组合，例如TV或AV，经常超过TAV的表现，表明过多的模态可能引入噪声。这些结果为金融短视频字幕生成建立了首个基准，并展示了在该领域将复杂视觉提示转化为实际应用的潜力与挑战。所有代码和数据可以在我们的GitHub上找到（遵循CC-BY-NC-SA 4.0许可）。
URL
https://arxiv.org/abs/2509.25745
PDF
-
Fast Real-Time Pipeline for Robust Arm Gesture Recognition
2025-09-29 16:57:56Mil\'an Zsolt Bagladi, L\'aszl\'o Guly\'as, Gerg\H{o} SzalayAbstract
This paper presents a real-time pipeline for dynamic arm gesture recognition based on OpenPose keypoint estimation, keypoint normalization, and a recurrent neural network classifier. The 1 x 1 normalization scheme and two feature representations (coordinate- and angle-based) are presented for the pipeline. In addition, an efficient method to improve robustness against camera angle variations is also introduced by using artificially rotated training data. Experiments on a custom traffic-control gesture dataset demonstrate high accuracy across varying viewing angles and speeds. Finally, an approach to calculate the speed of the arm signal (if necessary) is also presented.
Abstract (translated)
本文提出了一种基于OpenPose关键点估计、关键点归一化和循环神经网络分类器的实时动态手臂手势识别流水线。该流水线提出了1x1归一化方案以及两种特征表示（基于坐标的和基于角度的）。此外，通过使用人工旋转的训练数据，还介绍了一种提高对抗相机角度变化鲁棒性的有效方法。在自定义交通控制手势数据集上的实验表明，在不同的观察角度和速度下具有高准确性。最后，本文还提出了一种计算手臂信号速度（如有必要）的方法。
URL
https://arxiv.org/abs/2509.25042
PDF
-
SIG-Chat: Spatial Intent-Guided Conversational Gesture Generation Involving How, When and Where
2025-09-28 12:43:09Yiheng Huang, Junran Peng, Silei Shen, Jingwei Yang, ZeJi Wei, ChenCheng Bai, Yonghao He, Wei Sui, Muyi Sun, Yan Liu, Xu-Cheng Yin, Man Zhang, Zhaoxiang Zhang, Chuanchen LuoAbstract
The accompanying actions and gestures in dialogue are often closely linked to interactions with the environment, such as looking toward the interlocutor or using gestures to point to the described target at appropriate moments. Speech and semantics guide the production of gestures by determining their timing (WHEN) and style (HOW), while the spatial locations of interactive objects dictate their directional execution (WHERE). Existing approaches either rely solely on descriptive language to generate motions or utilize audio to produce non-interactive gestures, thereby lacking the characterization of interactive timing and spatial intent. This significantly limits the applicability of conversational gesture generation, whether in robotics or in the fields of game and animation production. To address this gap, we present a full-stack solution. We first established a unique data collection method to simultaneously capture high-precision human motion and spatial intent. We then developed a generation model driven by audio, language, and spatial data, alongside dedicated metrics for evaluating interaction timing and spatial accuracy. Finally, we deployed the solution on a humanoid robot, enabling rich, context-aware physical interactions.
Abstract (translated)
对话中的伴随动作和手势通常与环境互动紧密相关，例如朝对方看或在适当时刻用手势指向所描述的目标。言语和语义通过确定手势的时间（何时）和风格（如何）来指导手势的产生，而交互对象的空间位置则决定了其执行的方向（何处）。现有的方法要么仅依赖于描述性语言生成动作，要么利用音频生成非互动性手势，因此缺乏对互动时机和空间意图的刻画。这大大限制了对话中手势生成的应用范围，无论是机器人领域还是游戏与动画制作行业。 为解决这一问题，我们提出了一整套解决方案。首先，我们建立了一个独特的方法来同时捕捉高质量的人体动作和空间意图数据。接着，开发了一种由音频、语言和空间数据驱动的生成模型，并制定了评估互动时机和空间准确性专门指标。最后，我们将该方案部署在人形机器人上，实现了丰富且具备情境意识的物理交互。
URL
https://arxiv.org/abs/2509.23852
PDF
-
SlotFM: A Motion Foundation Model with Slot Attention for Diverse Downstream Tasks
2025-09-25 22:41:43Junyong Park, Oron Levy, Rebecca Adaimi, Asaf Liberman, Gierad Laput, Abdelkareem BedriAbstract
Wearable accelerometers are used for a wide range of applications, such as gesture recognition, gait analysis, and sports monitoring. Yet most existing foundation models focus primarily on classifying common daily activities such as locomotion and exercise, limiting their applicability to the broader range of tasks that rely on other signal characteristics. We present SlotFM, an accelerometer foundation model that generalizes across diverse downstream tasks. SlotFM uses Time-Frequency Slot Attention, an extension of Slot Attention that processes both time and frequency representations of the raw signals. It generates multiple small embeddings (slots), each capturing different signal components, enabling task-specific heads to focus on the most relevant parts of the data. We also introduce two loss regularizers that capture local structure and frequency patterns, which improve reconstruction of fine-grained details and helps the embeddings preserve task-relevant information. We evaluate SlotFM on 16 classification and regression downstream tasks that extend beyond standard human activity recognition. It outperforms existing self-supervised approaches on 13 of these tasks and achieves comparable results to the best performing approaches on the remaining tasks. On average, our method yields a 4.5% performance gain, demonstrating strong generalization for sensing foundation models.
Abstract (translated)
可穿戴加速度计被广泛应用于多种应用场景，例如手势识别、步态分析和运动监测。然而，现有的大多数基础模型主要关注于分类日常活动（如行走和锻炼），这限制了它们在依赖其他信号特征的更广泛的任务中的适用性。我们提出了SlotFM，这是一种适用于各种下游任务的加速度计基础模型。 SlotFM采用了时频槽注意力机制（Time-Frequency Slot Attention），它是槽注意力机制的一种扩展形式，可以处理原始信号的时间和频率表示。它生成多个小型嵌入（slot），每个嵌入捕捉不同的信号组成部分，从而使特定于任务的头部能够专注于数据中最相关的部分。我们还引入了两个损失正则化器来捕获局部结构和频域模式，这有助于重建细粒度细节并帮助嵌入保留与任务相关的信息。 我们在16个分类和回归下游任务上评估了SlotFM，这些任务超出了标准的人体活动识别范围。在其中的13项任务中，SlotFM的表现优于现有的自监督方法，并且在其余的任务中取得了接近最佳的方法的成绩。总体而言，我们的方法平均提高了4.5%的性能，这表明对于感知基础模型来说，具有强大的泛化能力。
URL
https://arxiv.org/abs/2509.21673
PDF
-
mmHSense: Multi-Modal and Distributed mmWave ISAC Datasets for Human Sensing
2025-09-24 11:49:08Nabeel Nisar Bhat, Maksim Karnaukh, Stein Vandenbroeke, Wouter Lemoine, Jakob Struye, Jesus Omar Lacruz, Siddhartha Kumar, Mohammad Hossein Moghaddam, Joerg Widmer, Rafael Berkvens, Jeroen FamaeyAbstract
This article presents mmHSense, a set of open labeled mmWave datasets to support human sensing research within Integrated Sensing and Communication (ISAC) systems. The datasets can be used to explore mmWave ISAC for various end applications such as gesture recognition, person identification, pose estimation, and localization. Moreover, the datasets can be used to develop and advance signal processing and deep learning research on mmWave ISAC. This article describes the testbed, experimental settings, and signal features for each dataset. Furthermore, the utility of the datasets is demonstrated through validation on a specific downstream task. In addition, we demonstrate the use of parameter-efficient fine-tuning to adapt ISAC models to different tasks, significantly reducing computational complexity while maintaining performance on prior tasks.
Abstract (translated)
本文介绍了mmHSense，这是一个包含开放标记的毫米波数据集的集合，旨在支持集成传感和通信（ISAC）系统中的人体感应研究。这些数据集可用于探索毫米波ISAC在各种终端应用中的用途，例如手势识别、人员识别、姿态估计和定位。此外，还可以利用这些数据集来开发和推进毫米波ISAC的信号处理和深度学习研究。本文描述了每个数据集的测试平台、实验设置和信号特征。此外，通过在一个特定的下游任务上进行验证，展示了数据集的应用价值。另外，我们还演示了如何使用参数高效微调方法将ISAC模型适应不同的任务，在减少计算复杂度的同时保持在先前任务上的性能水平。
URL
https://arxiv.org/abs/2509.21396
PDF
-
A Bimanual Gesture Interface for ROS-Based Mobile Manipulators Using TinyML and Sensor Fusion
2025-09-23 19:39:00Najeeb Ahmed Bhuiyan, M. Nasimul Huq, Sakib H. Chowdhury, Rahul MangharamAbstract
Gesture-based control for mobile manipulators faces persistent challenges in reliability, efficiency, and intuitiveness. This paper presents a dual-hand gesture interface that integrates TinyML, spectral analysis, and sensor fusion within a ROS framework to address these limitations. The system uses left-hand tilt and finger flexion, captured using accelerometer and flex sensors, for mobile base navigation, while right-hand IMU signals are processed through spectral analysis and classified by a lightweight neural network. This pipeline enables TinyML-based gesture recognition to control a 7-DOF Kinova Gen3 manipulator. By supporting simultaneous navigation and manipulation, the framework improves efficiency and coordination compared to sequential methods. Key contributions include a bimanual control architecture, real-time low-power gesture recognition, robust multimodal sensor fusion, and a scalable ROS-based implementation. The proposed approach advances Human-Robot Interaction (HRI) for industrial automation, assistive robotics, and hazardous environments, offering a cost-effective, open-source solution with strong potential for real-world deployment and further optimization.
Abstract (translated)
基于手势控制的移动操作器面临可靠性、效率和直观性等方面的持续挑战。本文提出了一种双手持界面，结合了TinyML（轻量级机器学习）、频谱分析以及多传感器融合技术，在ROS框架内运行以解决这些问题。该系统使用左手倾斜及手指弯曲动作来导航移动基座，这些动作通过加速度计和柔性传感器捕捉；而右手的IMU信号则经过频谱分析，并由轻量化神经网络进行分类。这一流程使得基于TinyML的手势识别能够控制具有7个自由度的Kinova Gen3操作器。通过支持同时进行导航与操作，该框架在效率和协调性上相比顺序方法有所提升。 关键贡献包括： - 双手控制系统架构 - 实时低功耗手势识别技术 - 稳健多模式传感器融合能力 - 基于ROS的可扩展实现 所提出的这种方法推进了人机交互（HRI）在工业自动化、辅助机器人和危险环境中的应用，提供了一个成本效益高且开源的解决方案，具有显著的实际部署潜力以及进一步优化的空间。
URL
https://arxiv.org/abs/2509.19521
PDF
-
Incorporating the Refractory Period into Spiking Neural Networks through Spike-Triggered Threshold Dynamics
2025-09-22 13:33:31Yang Li, Xinyi Zeng, Zhe Xue, Pinxian Zeng, Zikai Zhang, Yan WangAbstract
As the third generation of neural networks, spiking neural networks (SNNs) have recently gained widespread attention for their biological plausibility, energy efficiency, and effectiveness in processing neuromorphic datasets. To better emulate biological neurons, various models such as Integrate-and-Fire (IF) and Leaky Integrate-and-Fire (LIF) have been widely adopted in SNNs. However, these neuron models overlook the refractory period, a fundamental characteristic of biological neurons. Research on excitable neurons reveal that after firing, neurons enter a refractory period during which they are temporarily unresponsive to subsequent stimuli. This mechanism is critical for preventing over-excitation and mitigating interference from aberrant signals. Therefore, we propose a simple yet effective method to incorporate the refractory period into spiking LIF neurons through spike-triggered threshold dynamics, termed RPLIF. Our method ensures that each spike accurately encodes neural information, effectively preventing neuron over-excitation under continuous inputs and interference from anomalous inputs. Incorporating the refractory period into LIF neurons is seamless and computationally efficient, enhancing robustness and efficiency while yielding better performance with negligible overhead. To the best of our knowledge, RPLIF achieves state-of-the-art performance on Cifar10-DVS(82.40%) and N-Caltech101(83.35%) with fewer timesteps and demonstrates superior performance on DVS128 Gesture(97.22%) at low latency.
Abstract (translated)
作为第三代神经网络，脉冲神经网络（SNNs）因其生物学合理性、能源效率以及在处理类脑数据集方面的有效性而最近受到了广泛关注。为了更好地模拟生物神经元，各种模型如积分放电（IF）和漏导积分放电（LIF）被广泛应用于SNN中。然而，这些神经元模型忽略了复现期，这是生物神经元的一个基本特征。研究表明，在放电之后，神经元会进入一个暂时对后续刺激无反应的复现期。这一机制对于防止过度兴奋以及减少异常信号干扰至关重要。 因此，我们提出了一种简单而有效的方法，通过脉冲触发阈值动力学将复现期纳入到脉冲LIF神经元中，并将其命名为RPLIF。我们的方法确保每个脉冲能够准确编码神经信息，在连续输入和异常输入干扰下有效地防止神经元过度兴奋。 将复现期融入LIF神经元是无缝且计算效率高的，增强了稳健性和效率，并在几乎不增加额外开销的情况下提高了性能。据我们所知，RPLIF在Cifar10-DVS（82.40%）和N-Caltech101（83.35%）数据集上实现了最先进的性能，在更少的时间步下表现出色，并且在DVS128手势识别任务中以低延迟取得了97.22%的卓越表现。
URL
https://arxiv.org/abs/2509.17769
PDF
-
DA-Mamba: Dialogue-aware selective state-space model for multimodal engagement estimation
2025-09-22 12:48:42Shenwei Kang, Xin Zhang, Wen Liu, Bin Li, Yujie Liu, Bo GaoAbstract
Human engagement estimation in conversational scenarios is essential for applications such as adaptive tutoring, remote healthcare assessment, and socially aware human--computer interaction. Engagement is a dynamic, multimodal signal conveyed by facial expressions, speech, gestures, and behavioral cues over time. In this work we introduce DA-Mamba, a dialogue-aware multimodal architecture that replaces attention-heavy dialogue encoders with Mamba-based selective state-space processing to achieve linear time and memory complexity while retaining expressive cross-modal reasoning. We design a Mamba dialogue-aware selective state-space model composed of three core modules: a Dialogue-Aware Encoder, and two Mamba-based fusion mechanisms: Modality-Group Fusion and Partner-Group Fusion, these modules achieve expressive dialogue understanding. Extensive experiments on three standard benchmarks (NoXi, NoXi-Add, and MPIIGI) show that DA-Mamba surpasses prior state-of-the-art (SOTA) methods in concordance correlation coefficient (CCC), while reducing training time and peak memory; these gains enable processing much longer sequences and facilitate real-time deployment in resource-constrained, multi-party conversational settings. The source code will be available at: this https URL.
Abstract (translated)
在对话场景中的人类参与度估计对于自适应辅导、远程医疗评估和社会意识型人机交互等应用至关重要。参与度是一个随时间变化的多模态信号，通过面部表情、言语、手势和行为线索传达。 在这项工作中，我们引入了DA-Mamba（Dialogue-Aware Multimodal Architecture with Mamba-based Selective State-Space Processing），这是一种对话感知的多模式架构，它用基于Mamba的选择性状态空间处理方法替代了注重注意力的对话编码器。这种架构在保持跨模态推理能力的同时实现了线性的时间和内存复杂度。我们设计了一个由三个核心模块组成的DA-Mamba对话感知选择性状态空间模型：一个对话感知编解码器，以及两种基于Mamba的融合机制——模式组融合（Modality-Group Fusion）和伙伴组融合（Partner-Group Fusion）。这些模块共同实现了具有表达性的对话理解。 在三个标准基准数据集（NoXi、NoXi-Add 和 MPIIGI）上的广泛实验表明，DA-Mamba 在一致性相关系数 (CCC) 方面超越了先前的最先进方法，并且减少了训练时间和峰值内存使用量。这种改进使得处理更长序列成为可能，并有助于在资源受限的多方对话场景中实现实时部署。 该研究的源代码将在以下网址提供：this https URL.
URL
https://arxiv.org/abs/2509.17711
PDF
-
The Rhythm In Anything: Audio-Prompted Drums Generation with Masked Language Modeling
2025-09-19 05:41:51Patrick O'Reilly, Julia Barnett, Hugo Flores Garc\'ia, Annie Chu, Nathan Pruyne, Prem Seetharaman, Bryan PardoAbstract
Musicians and nonmusicians alike use rhythmic sound gestures, such as tapping and beatboxing, to express drum patterns. While these gestures effectively communicate musical ideas, realizing these ideas as fully-produced drum recordings can be time-consuming, potentially disrupting many creative workflows. To bridge this gap, we present TRIA (The Rhythm In Anything), a masked transformer model for mapping rhythmic sound gestures to high-fidelity drum recordings. Given an audio prompt of the desired rhythmic pattern and a second prompt to represent drumkit timbre, TRIA produces audio of a drumkit playing the desired rhythm (with appropriate elaborations) in the desired timbre. Subjective and objective evaluations show that a TRIA model trained on less than 10 hours of publicly-available drum data can generate high-quality, faithful realizations of sound gestures across a wide range of timbres in a zero-shot manner.
Abstract (translated)
无论是音乐人还是非音乐人都会使用诸如敲击和节奏口技等有节奏的声音手势来表达鼓点模式。虽然这些手势能够有效地传达音乐构思，但将这些想法转化为完整的鼓录音可能需要耗费大量时间，从而扰乱许多创意工作流程。为了解决这个问题，我们提出了TRIA（The Rhythm In Anything），这是一种掩码变压器模型，用于将有节奏的声音手势映射到高保真的鼓录音上。给定一个所需节拍模式的音频提示和另一个表示鼓组音色的提示，TRIA可以生成所需的节奏在指定音色下演奏的鼓组音频（并进行适当的扩展）。主观和客观评估表明，使用不到10小时公开可用的鼓数据训练出的TRIA模型可以在零样本条件下，在广泛的音色范围内产生高质量且忠实的声音手势实现。
URL
https://arxiv.org/abs/2509.15625
PDF
-
Doppler Radiance Field-Guided Antenna Selection for Improved Generalization in Multi-Antenna Wi-Fi-based Human Activity Recognition
2025-09-18 16:40:14Navid Hasanzadeh, Shahrokh ValaeeAbstract
With the IEEE 802.11bf Task Group introducing amendments to the WLAN standard for advanced sensing, interest in using Wi-Fi Channel State Information (CSI) for remote sensing has surged. Recent findings indicate that learning a unified three-dimensional motion representation through Doppler Radiance Fields (DoRFs) derived from CSI significantly improves the generalization capabilities of Wi-Fi-based human activity recognition (HAR). Despite this progress, CSI signals remain affected by asynchronous access point (AP) clocks and additive noise from environmental and hardware sources. Consequently, even with existing preprocessing techniques, both the CSI data and Doppler velocity projections used in DoRFs are still susceptible to noise and outliers, limiting HAR performance. To address this challenge, we propose a novel framework for multi-antenna APs to suppress noise and identify the most informative antennas based on DoRF fitting errors, which capture inconsistencies among Doppler velocity projections. Experimental results on a challenging small-scale hand gesture recognition dataset demonstrate that the proposed DoRF-guided Wi-Fi-based HAR approach significantly improves generalization capability, paving the way for robust real-world sensing deployments.
Abstract (translated)
随着IEEE 802.11bf工作组引入针对高级传感的无线局域网（WLAN）标准修正案，人们对利用Wi-Fi信道状态信息（CSI）进行远程传感的兴趣激增。近期的研究发现表明，通过从CSI中导出多普勒辐射场（DoRFs），学习统一的三维运动表示可以显著提升基于Wi-Fi的人体活动识别（HAR）的一般化能力。尽管取得了这一进展，但CSI信号仍然受到接入点（AP）时钟异步和环境及硬件源产生的加性噪声的影响。因此，即使采用了现有的预处理技术，用于DoRFs的CSI数据和多普勒速度投影依然容易受噪声和异常值的影响，从而限制了HAR性能。 为了解决这一挑战，我们提出了一种新的框架，利用多天线AP来抑制噪声，并根据DoRF拟合误差识别最具有信息量的天线，这些误差捕捉到了多普勒速度投影之间的不一致性。在一项针对小型手部手势识别数据集的实验中，我们的DoRF引导的基于Wi-Fi的人体活动识别方法显示出显著提升了泛化能力，为稳健的实际传感部署铺平了道路。
URL
https://arxiv.org/abs/2509.15129
PDF
-
GestOS: Advanced Hand Gesture Interpretation via Large Language Models to control Any Type of Robot
2025-09-17 20:27:11Artem Lykov, Oleg Kobzarev, Dzmitry TsetserukouAbstract
We present GestOS, a gesture-based operating system for high-level control of heterogeneous robot teams. Unlike prior systems that map gestures to fixed commands or single-agent actions, GestOS interprets hand gestures semantically and dynamically distributes tasks across multiple robots based on their capabilities, current state, and supported instruction sets. The system combines lightweight visual perception with large language model (LLM) reasoning: hand poses are converted into structured textual descriptions, which the LLM uses to infer intent and generate robot-specific commands. A robot selection module ensures that each gesture-triggered task is matched to the most suitable agent in real time. This architecture enables context-aware, adaptive control without requiring explicit user specification of targets or commands. By advancing gesture interaction from recognition to intelligent orchestration, GestOS supports scalable, flexible, and user-friendly collaboration with robotic systems in dynamic environments.
Abstract (translated)
我们介绍了GestOS，这是一种基于手势的操作系统，用于对异构机器人团队进行高级控制。与以前将手势映射到固定命令或单一代理动作的系统不同，GestOS 通过考虑机器人的能力、当前状态和可支持指令集来语义化地解释手部姿势，并动态分配任务给多个机器人。该系统结合了轻量级视觉感知技术和大型语言模型（LLM）推理：将手势转换为结构化的文本描述，然后由 LLM 推断意图并生成特定于机器人的命令。一个机器人选择模块确保每次通过手势触发的任务都能实时匹配最合适的代理。这种架构使上下文感知、自适应控制成为可能，而无需用户明确指定目标或命令。通过将手势交互从识别提升到智能编排，GestOS 支持在动态环境中与机器人系统进行可扩展的、灵活且用户友好的协作。
URL
https://arxiv.org/abs/2509.14412
PDF
-
Towards Context-Aware Human-like Pointing Gestures with RL Motion Imitation
2025-09-16 09:30:42Anna Deichler, Siyang Wang, Simon Alexanderson, Jonas BeskowAbstract
Pointing is a key mode of interaction with robots, yet most prior work has focused on recognition rather than generation. We present a motion capture dataset of human pointing gestures covering diverse styles, handedness, and spatial targets. Using reinforcement learning with motion imitation, we train policies that reproduce human-like pointing while maximizing precision. Results show our approach enables context-aware pointing behaviors in simulation, balancing task performance with natural dynamics.
Abstract (translated)
指向是与机器人互动的关键方式，然而大多数先前的研究主要集中在识别而非生成上。我们提出了一套人类指向手势的动作捕捉数据集，涵盖了多样化的风格、左右手使用偏好及空间目标。通过结合运动模仿的强化学习方法，我们训练出了能够再现类人指向动作并最大化精度的策略。实验结果显示，我们的方法能够在模拟环境中实现情境感知的指向行为，在保持任务性能的同时兼顾自然动态表现。
URL
https://arxiv.org/abs/2509.12880
PDF
-
A Novel Skill Modeling Approach: Integrating Vergnaud's Scheme with Cognitive Architectures
2025-09-16 09:09:15Antoine L\'enat (Cetim, LS2N, LS2N - \'equipe RoMas, LS2N - \'equipe PACCE), Olivier Cheminat (Cetim), Damien Chablat (LS2N - \'equipe RoMas, LS2N), Camilo Charron (LS2N, UR2, LS2N - \'equipe PACCE)Abstract
Human-machine interaction is increasingly important in industry, and this trend will only intensify with the rise of Industry 5.0. Human operators have skills that need to be adapted when using machines to achieve the best results. It is crucial to highlight the operator's skills and understand how they use and adapt them [18]. A rigorous description of these skills is necessary to compare performance with and without robot assistance. Predicate logic, used by Vergnaud within Piaget's scheme concept, offers a promising approach. However, this theory doesn't account for cognitive system constraints, such as the timing of actions, the limitation of cognitive resources, the parallelization of tasks, or the activation of automatic gestures contrary to optimal knowledge. Integrating these constraints is essential for representing agent skills understanding skill transfer between biological and mechanical structures. Cognitive architectures models [2] address these needs by describing cognitive structure and can be combined with the scheme for mutual benefit. Welding provides a relevant case study, as it highlights the challenges faced by operators, even highly skilled ones. Welding's complexity stems from the need for constant skill adaptation to variable parameters like part position and process. This adaptation is crucial, as weld quality, a key factor, is only assessed afterward via destructive testing. Thus, the welder is confronted with a complex perception-decision-action cycle, where the evaluation of the impact of his actions is delayed and where errors are definitive. This dynamic underscores the importance of understanding and modeling the skills of operators.
Abstract (translated)
人机交互在工业领域日益重要，随着Industry 5.0的到来，这一趋势只会加剧。人类操作员具备的技能需要适应机器使用以达到最佳效果。突出操作者的技能并理解他们如何运用和调整这些技能至关重要[18]。为了比较有无机器人协助时的表现，详细描述这些技能是必要的。维尔格诺在皮亚杰方案概念中使用的谓词逻辑提供了一个很有前途的方法。然而，该理论没有考虑认知系统的限制，如动作时间、认知资源的局限性、任务并行化以及与最优知识相悖的手动习惯被激活的情况。将这些约束整合进来对于表示代理技能和理解生物结构向机械结构的技能转移至关重要。认知架构模型[2]通过描述认知结构来满足这一需求，并且可以结合方案以实现互利。 焊接提供了一个相关的案例研究，它突显了操作者面临的挑战，即使是高度熟练的操作者也不例外。焊接复杂性的根源在于需要不断调整技能以适应变化的因素，如零件位置和工艺过程。这种调整至关重要，因为焊缝质量——一个关键因素——只能通过破坏性测试来评估。因此，焊工面临着复杂的感知-决策-行动循环，在这个过程中，他们对自身行为影响的评价被延迟了，并且错误是不可逆转的。这一动态强调了理解并建模操作员技能的重要性。
URL
https://arxiv.org/abs/2509.12851
PDF
