Gesture
-
Multimodal Quantitative Measures for Multiparty Behaviour Evaluation
2025-08-01 13:46:12Ojas Shirekar, Wim Pouw, Chenxu Hao, Vrushank Phadnis, Thabo Beeler, Chirag RamanAbstract
Digital humans are emerging as autonomous agents in multiparty interactions, yet existing evaluation metrics largely ignore contextual coordination dynamics. We introduce a unified, intervention-driven framework for objective assessment of multiparty social behaviour in skeletal motion data, spanning three complementary dimensions: (1) synchrony via Cross-Recurrence Quantification Analysis, (2) temporal alignment via Multiscale Empirical Mode Decompositionbased Beat Consistency, and (3) structural similarity via Soft Dynamic Time Warping. We validate metric sensitivity through three theory-driven perturbations -- gesture kinematic dampening, uniform speech-gesture delays, and prosodic pitch-variance reduction-applied to $\approx 145$ 30-second thin slices of group interactions from the DnD dataset. Mixed-effects analyses reveal predictable, joint-independent shifts: dampening increases CRQA determinism and reduces beat consistency, delays weaken cross-participant coupling, and pitch flattening elevates F0 Soft-DTW costs. A complementary perception study ($N=27$) compares judgments of full-video and skeleton-only renderings to quantify representation effects. Our three measures deliver orthogonal insights into spatial structure, timing alignment, and behavioural variability. Thereby forming a robust toolkit for evaluating and refining socially intelligent agents. Code available on \href{this https URL}{GitHub}.
Abstract (translated)
数字人类正在作为多方互动中的自主代理出现，然而现有的评估指标大多忽略了情境协调动态。我们引入了一个统一的、以干预驱动为主的框架，用于基于骨骼运动数据客观地评估多方社交行为，涵盖三个互补维度：（1）通过交叉回归量化分析(CRQA)来衡量同步性；（2）通过多尺度经验模式分解(MEMD)基础节拍一致性来衡量时间对齐；以及（3）通过软动态时间规整(Soft-DTW)来衡量结构相似度。我们通过对从DnD数据集中提取的约145个30秒长度的小组互动片段进行三种理论驱动干扰——手势动力学减弱、统一的语言-动作延迟和音调变异性减少——验证了这些指标的敏感性。混合效应分析揭示了可预测且独立于关节变化的趋势：动力减弱增加了CRQA决定论并降低了节拍一致性，延迟削弱了参与者之间的耦合，而音调扁平化提高了F0 Soft-DTW成本。 一项补充感知研究（N=27）比较了全视频和骨骼动画渲染的评判结果，以量化表示效果。我们的三项度量分别提供了空间结构、时间对齐和行为变异性方面的独立见解。因此形成了一套评估并改进社交智能代理的稳健工具包。代码可在GitHub上获得。 原文链接：[此处应为一个有效的GitHub链接]
URL
https://arxiv.org/abs/2508.10916
PDF
-
Cued-Agent: A Collaborative Multi-Agent System for Automatic Cued Speech Recognition
2025-08-01 07:40:39Guanjie Huang, Danny H. K. Tsang, Shan Yang, Guangzhi Lei, Li LiuAbstract
Cued Speech (CS) is a visual communication system that combines lip-reading with hand coding to facilitate communication for individuals with hearing impairments. Automatic CS Recognition (ACSR) aims to convert CS hand gestures and lip movements into text via AI-driven methods. Traditionally, the temporal asynchrony between hand and lip movements requires the design of complex modules to facilitate effective multimodal fusion. However, constrained by limited data availability, current methods demonstrate insufficient capacity for adequately training these fusion mechanisms, resulting in suboptimal performance. Recently, multi-agent systems have shown promising capabilities in handling complex tasks with limited data availability. To this end, we propose the first collaborative multi-agent system for ACSR, named Cued-Agent. It integrates four specialized sub-agents: a Multimodal Large Language Model-based Hand Recognition agent that employs keyframe screening and CS expert prompt strategies to decode hand movements, a pretrained Transformer-based Lip Recognition agent that extracts lip features from the input video, a Hand Prompt Decoding agent that dynamically integrates hand prompts with lip features during inference in a training-free manner, and a Self-Correction Phoneme-to-Word agent that enables post-process and end-to-end conversion from phoneme sequences to natural language sentences for the first time through semantic refinement. To support this study, we expand the existing Mandarin CS dataset by collecting data from eight hearing-impaired cuers, establishing a mixed dataset of fourteen subjects. Extensive experiments demonstrate that our Cued-Agent performs superbly in both normal and hearing-impaired scenarios compared with state-of-the-art methods. The implementation is available at this https URL.
Abstract (translated)
Cued Speech（CS）是一种结合唇读和手码的视觉沟通系统，旨在为听力受损的人士提供有效的交流方式。自动化的Cued Speech识别（ACSR）目标是利用人工智能技术将手部动作和口型变化转换成文本信息。传统方法中，由于手势与口型间的时序不同步问题，需要设计复杂的模块来实现有效多模态融合。然而，受限于数据量的不足，现有方法在训练这些融合机制方面显得力不从心，导致性能不佳。 最近的研究显示，多智能体系统（multi-agent systems）在处理数据有限的任务中展现出巨大潜力。基于此，我们提出了一种用于ACSR的首个协作多智能体系统——Cued-Agent。该系统集成了四个专业化子代理：一个多模态大型语言模型驱动的手部识别代理，它通过关键帧筛选和CS专家提示策略解码手势；一个预训练的Transformer架构口型识别代理，从输入视频中提取口唇特征；一个手部提示解码代理，在推理过程中以无需训练的方式动态融合手部提示与口唇特征；以及一个自我纠错音素到单词转换代理，首次通过语义精炼实现从音素序列到自然语言句子的后处理和端到端转化。 为了支持这项研究，我们扩展了现有的普通话CS数据集，收集来自八位听力受损的Cued Speech用户的资料，并建立了一个由十四名用户组成的混合数据集。实验结果表明，我们的Cued-Agent在正常人和听力障碍场景下均优于现有最先进的方法。该系统的代码可在提供的链接中获取。
URL
https://arxiv.org/abs/2508.00391
PDF
-
Indian Sign Language Detection for Real-Time Translation using Machine Learning
2025-07-31 08:51:49Rajat Singhal, Jatin Gupta, Akhil Sharma, Anushka Gupta, Navya SharmaAbstract
Gestural language is used by deaf & mute communities to communicate through hand gestures & body movements that rely on visual-spatial patterns known as sign languages. Sign languages, which rely on visual-spatial patterns of hand gestures & body movements, are the primary mode of communication for deaf & mute communities worldwide. Effective communication is fundamental to human interaction, yet individuals in these communities often face significant barriers due to a scarcity of skilled interpreters & accessible translation technologies. This research specifically addresses these challenges within the Indian context by focusing on Indian Sign Language (ISL). By leveraging machine learning, this study aims to bridge the critical communication gap for the deaf & hard-of-hearing population in India, where technological solutions for ISL are less developed compared to other global sign languages. We propose a robust, real-time ISL detection & translation system built upon a Convolutional Neural Network (CNN). Our model is trained on a comprehensive ISL dataset & demonstrates exceptional performance, achieving a classification accuracy of 99.95%. This high precision underscores the model's capability to discern the nuanced visual features of different signs. The system's effectiveness is rigorously evaluated using key performance metrics, including accuracy, F1 score, precision & recall, ensuring its reliability for real-world applications. For real-time implementation, the framework integrates MediaPipe for precise hand tracking & motion detection, enabling seamless translation of dynamic gestures. This paper provides a detailed account of the model's architecture, the data preprocessing pipeline & the classification methodology. The research elaborates the model architecture, preprocessing & classification methodologies for enhancing communication in deaf & mute communities.
Abstract (translated)
手势语言是由聋哑社区通过手部动作和身体姿态的视觉空间模式来交流的一种方式，这种模式被称为手语。手语依靠手部动作和身体姿态的视觉空间模式，是全球聋哑人群的主要沟通方式。有效的沟通对人类互动至关重要，然而由于熟练翻译人员和可访问翻译技术稀缺，这些群体常面临重大障碍。 本研究针对印度语境下的这些问题，特别关注印度手语（ISL），通过机器学习来填补听障人士的交流缺口。在印度，相对于全球其他手语来说，ISL的技术解决方案发展相对滞后。我们提出了一种基于卷积神经网络（CNN）的强大实时ISL检测与翻译系统。我们的模型经过全面的ISL数据集训练，并展示了卓越性能，达到了99.95%的分类准确率。这一高精度表明了该模型能够识别不同手势之间的细微视觉特征差异。 为了评估系统的有效性，我们使用了一系列关键性能指标进行严格测试，包括准确性、F1分数、精确度和召回率，确保其在实际应用中的可靠性。对于实时实施，框架整合了MediaPipe以实现精准的手部追踪及运动检测，使得动态手势的无缝翻译得以实现。 本文详细描述了模型架构、数据预处理管道以及分类方法，并深入探讨了增强聋哑人群体沟通的方法和策略。
URL
https://arxiv.org/abs/2507.20414
PDF
-
CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding
2025-07-29 15:00:21Fevziye Irem Eyiokur, Dogucan Yaman, Haz{\i}m Kemal Ekenel, Alexander WaibelAbstract
We address the problem of Embodied Reference Understanding, which involves predicting the object that a person in the scene is referring to through both pointing gesture and language. Accurately identifying the referent requires multimodal understanding: integrating textual instructions, visual pointing, and scene context. However, existing methods often struggle to effectively leverage visual clues for disambiguation. We also observe that, while the referent is often aligned with the head-to-fingertip line, it occasionally aligns more closely with the wrist-to-fingertip line. Therefore, relying on a single line assumption can be overly simplistic and may lead to suboptimal performance. To address this, we propose a dual-model framework, where one model learns from the head-to-fingertip direction and the other from the wrist-to-fingertip direction. We further introduce a Gaussian ray heatmap representation of these lines and use them as input to provide a strong supervisory signal that encourages the model to better attend to pointing cues. To combine the strengths of both models, we present the CLIP-Aware Pointing Ensemble module, which performs a hybrid ensemble based on CLIP features. Additionally, we propose an object center prediction head as an auxiliary task to further enhance referent localization. We validate our approach through extensive experiments and analysis on the benchmark YouRefIt dataset, achieving an improvement of approximately 4 mAP at the 0.25 IoU threshold.
Abstract (translated)
我们解决的是具身指代理解问题，这个问题涉及通过指向手势和语言来预测场景中的人所指的是哪个物体。准确地识别出被指代的物体需要多模态的理解能力：结合文本指令、视觉指向以及场景上下文信息。然而，现有的方法往往难以有效地利用视觉线索来进行消歧化处理。我们还观察到，虽然参照物通常与头部至指尖的方向线对齐，但也偶尔会更接近于腕部至指尖的方向线。因此，依赖单一方向线的假设可能会过于简单，并可能导致性能不佳。 为了应对这一挑战，我们提出了一种双模型框架：一个模型从头部至指尖的方向学习，另一个则从腕部至指尖的方向学习。我们进一步引入了这两种线条的高斯射线热图表示方法，并将其用作输入以提供强有力的监督信号，鼓励模型更好地关注指向线索。 为了结合两个模型的优点，我们提出了CLIP-Aware Pointing Ensemble模块，该模块基于CLIP特征执行混合集成操作。此外，我们还提出了一项辅助任务——物体中心预测头，进一步增强参照物的定位能力。 通过在YouRefIt基准数据集上进行广泛实验和分析，我们的方法在0.25 IoU阈值下实现了约4 mAP的性能提升。
URL
https://arxiv.org/abs/2507.21888
PDF
-
Evaluating Interactions between Automated Vehicles and Cyclists using a coupled In-the-Loop Test Environment
2025-07-29 14:32:33Michael Kaiser, Clemens Gro{\ss}, Lisa Marie Otto, Steffen M\"ullerAbstract
Testing and evaluating automated driving systems (ADS) in interactions with vulnerable road users (VRUs), such as cyclists, are essential for improving the safety of VRUs, but often lack realism. This paper presents and validates a coupled in-the-loop test environment that integrates a Cyclist-in-the Loop test bench with a Vehicle-in-the-Loop test bench via a virtual environment (VE) developed in Unreal Engine 5. The setup enables closed-loop, bidirectional interaction between a real human cyclist and a real automated vehicle under safe and controllable conditions. The automated vehicle reacts to cyclist gestures via stimulated camera input, while the cyclist, riding a stationary bicycle, perceives and reacts to the vehicle in the VE in real time. Validation experiments are conducted using a real automated shuttle bus with a track-and-follow function, performing three test maneuvers - straight-line driving with stop, circular track driving, and double lane change - on a proving ground and in the coupled in-the-loop test environment. The performance is evaluated by comparing the resulting vehicle trajectories in both environments. Additionally, the introduced latencies of individual components in the test setup are measured. The results demonstrate the feasibility of the approach and highlight its strengths and limitations for realistic ADS evaluation.
Abstract (translated)
测试和评估自动驾驶系统（ADS）与弱势道路使用者（VRU，如骑自行车的人）互动的安全性至关重要，但这类评估往往缺乏真实性。本文提出并验证了一种集成的闭环测试环境，该环境通过使用虚幻引擎5开发的虚拟环境将“骑行者在环”测试台和“车辆在环”测试台连接起来。这种设置能够在安全且可控的情况下实现真实人类骑自行车人与真实自动驾驶汽车之间的闭环双向互动。具体来说，自动驾驶车辆通过模拟相机输入对骑车人的手势作出反应，而骑车人在虚拟环境中实时感知并回应这辆车的动作。 为了验证这一方法的有效性，研究团队使用了一辆具有跟随功能的真实自动穿梭巴士，在测试场地上进行三种不同的机动操作：直线行驶带停车、圆形赛道驾驶和双车道变换。这些操作既在实际的测试场地中完成，也在耦合的闭环测试环境中进行。通过比较两种环境下生成的车辆轨迹来评估性能，并测量了测试设备各个组件引入的时间延迟。 研究结果表明，该方法具有可行性，并揭示了其用于现实世界自动驾驶系统评估的优势和局限性。这种方法为提高弱势道路使用者的安全性提供了一个有价值的工具。
URL
https://arxiv.org/abs/2507.21859
PDF
-
emg2tendon: From sEMG Signals to Tendon Control in Musculoskeletal Hands
Abstract
Tendon-driven robotic hands offer unparalleled dexterity for manipulation tasks, but learning control policies for such systems presents unique challenges. Unlike joint-actuated robotic hands, tendon-driven systems lack a direct one-to-one mapping between motion capture (mocap) data and tendon controls, making the learning process complex and expensive. Additionally, visual tracking methods for real-world applications are prone to occlusions and inaccuracies, further complicating joint tracking. Wrist-wearable surface electromyography (sEMG) sensors present an inexpensive, robust alternative to capture hand motion. However, mapping sEMG signals to tendon control remains a significant challenge despite the availability of EMG-to-pose data sets and regression-based models in the existing literature. We introduce the first large-scale EMG-to-Tendon Control dataset for robotic hands, extending the emg2pose dataset, which includes recordings from 193 subjects, spanning 370 hours and 29 stages with diverse gestures. This dataset incorporates tendon control signals derived using the MyoSuite MyoHand model, addressing limitations such as invalid poses in prior methods. We provide three baseline regression models to demonstrate emg2tendon utility and propose a novel diffusion-based regression model for predicting tendon control from sEMG recordings. This dataset and modeling framework marks a significant step forward for tendon-driven dexterous robotic manipulation, laying the groundwork for scalable and accurate tendon control in robotic hands. this https URL
Abstract (translated)
肌腱驱动的机器人手在操控任务中提供了无与伦比的灵巧性，但为其系统学习控制策略却面临着独特的挑战。与关节驱动的机器人手不同，肌腱驱动系统缺乏运动捕捉（mocap）数据和肌腱控制之间的一对一映射关系，使得学习过程变得复杂且成本高昂。此外，用于实际应用中的视觉跟踪方法易受遮挡和不准确的影响，进一步增加了关节追踪的难度。手腕穿戴式的表面肌电图（sEMG）传感器提供了一种低成本且稳健的方式来捕捉手部动作，然而将sEMG信号映射到肌腱控制仍然是一项重大挑战，尽管现有的文献中已有EMG至姿态数据集和基于回归模型的方法。 我们首次推出了大规模的EMG至肌腱控制数据集，适用于机器人手，并扩展了emg2pose数据集，该数据集收录了193名受试者的记录，总计时长为370小时，涵盖了29个不同手势阶段。此数据集包含通过MyoSuite MyoHand模型导出的肌腱控制信号，克服了先前方法中存在的无效姿态等问题。我们提供了三种基础回归模型来展示emg2tendon工具的有效性，并提出了一种基于扩散的新颖回归模型，用于从sEMG记录中预测肌腱控制。这一数据集和建模框架标志着肌腱驱动的灵巧机器人操控技术向前迈出的一大步，为在机器人手中实现可扩展且准确的肌腱控制奠定了基础。 相关链接：[请在此处插入实际网址](https://this.url)
URL
https://arxiv.org/abs/2508.08269
PDF
-
AutoSign: Direct Pose-to-Text Translation for Continuous Sign Language Recognition
2025-07-26 07:28:33Samuel Ebimobowei Johnny, Blessed Guda, Andrew Blayama Stephen, Assane GueyeAbstract
Continuously recognizing sign gestures and converting them to glosses plays a key role in bridging the gap between the hearing and hearing-impaired communities. This involves recognizing and interpreting the hands, face, and body gestures of the signer, which pose a challenge as it involves a combination of all these features. Continuous Sign Language Recognition (CSLR) methods rely on multi-stage pipelines that first extract visual features, then align variable-length sequences with target glosses using CTC or HMM-based approaches. However, these alignment-based methods suffer from error propagation across stages, overfitting, and struggle with vocabulary scalability due to the intermediate gloss representation bottleneck. To address these limitations, we propose AutoSign, an autoregressive decoder-only transformer that directly translates pose sequences to natural language text, bypassing traditional alignment mechanisms entirely. The use of this decoder-only approach allows the model to directly map between the features and the glosses without the need for CTC loss while also directly learning the textual dependencies in the glosses. Our approach incorporates a temporal compression module using 1D CNNs to efficiently process pose sequences, followed by AraGPT2, a pre-trained Arabic decoder, to generate text (glosses). Through comprehensive ablation studies, we demonstrate that hand and body gestures provide the most discriminative features for signer-independent CSLR. By eliminating the multi-stage pipeline, AutoSign achieves substantial improvements on the Isharah-1000 dataset, achieving an improvement of up to 6.1\% in WER score compared to the best existing method.
Abstract (translated)
持续识别手势并将其转换为语符（glosses）在连接听觉正常人群和听力障碍人群之间发挥着关键作用。这涉及到识别和解释手语者的双手、面部及身体动作，由于这些特征的组合使用而带来了一定挑战。连续手语识别（Continuous Sign Language Recognition, CSLR）方法依赖于多阶段流水线技术，该技术首先提取视觉特征，然后通过CTC或HMM基方法将不同长度的时间序列与目标语符对齐。然而，基于这种方法存在跨阶段的错误传播、过拟合以及由于中间语符表示瓶颈而导致词汇量扩展困难的问题。 为了解决这些问题，我们提出了一种名为AutoSign的方法，这是一种自回归解码器专用的Transformer模型，它直接将姿态序列翻译成自然语言文本，完全绕过了传统的对齐机制。使用这种解码器专有方法使得该模型能够在不依赖CTC损失的情况下直接映射特征与语符，并且还可以直接学习语符中的文本依赖关系。我们的方法利用了一维CNN的时间压缩模块来高效处理姿态序列，随后采用预训练的阿拉伯语言解码器AraGPT2生成文本（语符）。通过全面的消融研究，我们证明了手部和身体动作提供了对说话人无关CSLR而言最具区分性的特征。通过消除多阶段管道，AutoSign在Isharah-1000数据集上实现了显著改进，与现有最佳方法相比，WER得分提高了高达6.1%。 简而言之，这项工作展示了使用自回归解码器专用Transformer模型进行直接姿态到文本翻译的潜力，并通过有效的时间压缩模块解决了连续手语识别中的关键问题。这种方法不仅简化了传统CSLR流程，还提升了系统的整体性能和灵活性。
URL
https://arxiv.org/abs/2507.19840
PDF
-
SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning
2025-07-25 15:10:15Lanmiao Liu, Esam Ghaleb, Asl{\i} \"Ozy\"urek, Zerrin YumakAbstract
Creating a virtual avatar with semantically coherent gestures that are aligned with speech is a challenging task. Existing gesture generation research mainly focused on generating rhythmic beat gestures, neglecting the semantic context of the gestures. In this paper, we propose a novel approach for semantic grounding in co-speech gesture generation that integrates semantic information at both fine-grained and global levels. Our approach starts with learning the motion prior through a vector-quantized variational autoencoder. Built on this model, a second-stage module is applied to automatically generate gestures from speech, text-based semantics and speaker identity that ensures consistency between the semantic relevance of generated gestures and co-occurring speech semantics through semantic coherence and relevance modules. Experimental results demonstrate that our approach enhances the realism and coherence of semantic gestures. Extensive experiments and user studies show that our method outperforms state-of-the-art approaches across two benchmarks in co-speech gesture generation in both objective and subjective metrics. The qualitative results of our model, code, dataset and pre-trained models can be viewed at this https URL.
Abstract (translated)
创建一个与说话内容语义一致、动作连贯的虚拟化身是一项具有挑战性的任务。现有的手势生成研究主要集中在产生节奏性拍打手势上，而忽视了手势的语义背景。在本文中，我们提出了一种新颖的方法来解决伴随言语的手势生成中的语义关联问题，这种方法在整个层次结构（从细粒度到全局）整合了语义信息。我们的方法首先通过向量量化变分自动编码器学习运动先验知识。在此模型基础上，第二阶段模块会根据语音、文本语义和说话者身份自动生成手势，并确保生成的手势与伴随出现的言语语义在语境一致性和相关性方面保持一致性。实验结果表明，我们的方法提升了语义手势的真实感和连贯性。广泛的实验证明，在两个基准数据集上，我们提出的方法无论是在客观指标还是主观评价中均优于当前最先进的方法。我们的模型、代码、数据集及预训练模型的相关定性分析可以在[这个链接](https://this-url.com)查看。
URL
https://arxiv.org/abs/2507.19359
PDF
-
Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations
2025-07-23 06:34:58Xiaoan Liu, Difan Jia, Xianhao Carton Liu, Mar Gonzalez-Franco, Chen Zhu-TianAbstract
Interacting with real-world objects in Mixed Reality (MR) often proves difficult when they are crowded, distant, or partially occluded, hindering straightforward selection and manipulation. We observe that these difficulties stem from performing interaction directly on physical objects, where input is tightly coupled to their physical constraints. Our key insight is to decouple interaction from these constraints by introducing proxies-abstract representations of real-world objects. We embody this concept in Reality Proxy, a system that seamlessly shifts interaction targets from physical objects to their proxies during selection. Beyond facilitating basic selection, Reality Proxy uses AI to enrich proxies with semantic attributes and hierarchical spatial relationships of their corresponding physical objects, enabling novel and previously cumbersome interactions in MR - such as skimming, attribute-based filtering, navigating nested groups, and complex multi object selections - all without requiring new gestures or menu systems. We demonstrate Reality Proxy's versatility across diverse scenarios, including office information retrieval, large-scale spatial navigation, and multi-drone control. An expert evaluation suggests the system's utility and usability, suggesting that proxy-based abstractions offer a powerful and generalizable interaction paradigm for future MR systems.
Abstract (translated)
在混合现实（MR）中与真实世界对象进行交互常常因为这些对象过于拥挤、距离较远或部分被遮挡而变得困难，这阻碍了直接的选择和操作。我们观察到这些问题的根源在于直接在物理对象上执行交互时输入与其物理约束紧密耦合。我们的关键见解是通过引入代理——即现实世界对象的抽象表示来解除这种约束，从而使交互更加灵活。 Reality Proxy 是一个实现这一概念的系统，它可以在选择过程中将互动目标从实际物体无缝转移到它们对应的代理（proxies）。除了简化基本的选择操作之外，Reality Proxy 还使用人工智能技术为这些代理添加与相应物理对象相关的语义属性和空间层次关系。这使得在 MR 中进行诸如浏览、基于属性筛选、导航嵌套组以及复杂多对象选择等新型且以前难以实现的交互成为可能，并且无需引入新的手势或菜单系统。 我们展示了 Reality Proxy 在各种场景中的灵活性，包括办公信息检索、大规模空间导航和多无人机控制。专家评估表明该系统的实用性和易用性，并暗示基于代理（proxies）的抽象为未来的 MR 系统提供了一种强大而通用化的交互方式。
URL
https://arxiv.org/abs/2507.17248
PDF
-
Falconry-like palm landing by a flapping-wing drone based on the human gesture interaction and distance-aware flight planning
2025-07-23 02:25:03Kazuki Numazato, Keiichiro Kan, Masaki Kitagawa, Yunong Li, Johannes Kubel, Moju ZhaoAbstract
Flapping-wing drones have attracted significant attention due to their biomimetic flight. They are considered more human-friendly due to their characteristics such as low noise and flexible wings, making them suitable for human-drone interactions. However, few studies have explored the practical interaction between humans and flapping-wing drones. On establishing a physical interaction system with flapping-wing drones, we can acquire inspirations from falconers who guide birds of prey to land on their arms. This interaction interprets the human body as a dynamic landing platform, which can be utilized in various scenarios such as crowded or spatially constrained environments. Thus, in this study, we propose a falconry-like interaction system in which a flapping-wing drone performs a palm landing motion on a human hand. To achieve a safe approach toward humans, we design a trajectory planning method that considers both physical and psychological factors of the human safety such as the drone's velocity and distance from the user. We use a commercial flapping platform with our implemented motion planning and conduct experiments to evaluate the palm landing performance and safety. The results demonstrate that our approach enables safe and smooth hand landing interactions. To the best of our knowledge, it is the first time to achieve a contact-based interaction between flapping-wing drones and humans.
Abstract (translated)
扑翼无人机因其仿生飞行特性而吸引了广泛的关注。由于其低噪音和灵活翅膀等特点，被认为更有利于与人类友好互动，适合进行人机交互。然而，很少有研究探讨人类与扑翼无人机之间实际的互动方式。建立一个物理互动系统时，我们可以借鉴猎鹰驯养者引导猛禽停落在手臂上的方法。这种方法将人体视为动态着陆平台，在拥挤或空间受限环境中具有多种应用场景。因此，本研究提出了一种类似猎人与鹰互动的系统，使扑翼无人机能够在人的手掌上进行安全着陆。 为了实现安全接近人类的目标，我们设计了一个轨迹规划方法，考虑了物理和心理因素的安全性，如无人机的速度和距离等。我们使用一个商业化的扑翼平台并实施了该运动计划，在实验中评估了手部着陆性能与安全性。结果表明，我们的方法能够实现安全、流畅的手部接触互动。 据我们所知，这是首次实现了扑翼无人机与人类之间的接触式交互。
URL
https://arxiv.org/abs/2507.17144
PDF
-
Hybrid-supervised Hypergraph-enhanced Transformer for Micro-gesture Based Emotion Recognition
2025-07-20 08:27:56Zhaoqiang Xia, Hexiang Huang, Haoyu Chen, Xiaoyi Feng, Guoying ZhaoAbstract
Micro-gestures are unconsciously performed body gestures that can convey the emotion states of humans and start to attract more research attention in the fields of human behavior understanding and affective computing as an emerging topic. However, the modeling of human emotion based on micro-gestures has not been explored sufficiently. In this work, we propose to recognize the emotion states based on the micro-gestures by reconstructing the behavior patterns with a hypergraph-enhanced Transformer in a hybrid-supervised framework. In the framework, hypergraph Transformer based encoder and decoder are separately designed by stacking the hypergraph-enhanced self-attention and multiscale temporal convolution modules. Especially, to better capture the subtle motion of micro-gestures, we construct a decoder with additional upsampling operations for a reconstruction task in a self-supervised learning manner. We further propose a hypergraph-enhanced self-attention module where the hyperedges between skeleton joints are gradually updated to present the relationships of body joints for modeling the subtle local motion. Lastly, for exploiting the relationship between the emotion states and local motion of micro-gestures, an emotion recognition head from the output of encoder is designed with a shallow architecture and learned in a supervised way. The end-to-end framework is jointly trained in a one-stage way by comprehensively utilizing self-reconstruction and supervision information. The proposed method is evaluated on two publicly available datasets, namely iMiGUE and SMG, and achieves the best performance under multiple metrics, which is superior to the existing methods.
Abstract (translated)
微手势是人类无意识中执行的身体动作，能够传达人的情感状态，在人类行为理解和情感计算领域作为一个新兴话题开始吸引更多的研究关注。然而，基于微手势的人类情绪建模尚未被充分探索。在本工作中，我们提出了一种通过使用增强的超图Transformer在一个混合监督框架内重构行为模式来识别基于微手势的情绪状态的方法。在这个框架中，超图Transformer编码器和解码器分别由堆叠的超图增强自注意力模块和多尺度时间卷积模块设计而成。 特别地，为了更好地捕捉微手势中的细微运动，我们在解码器中增加了额外的上采样操作以在自我监督学习模式下进行重构任务。我们还提出了一种增强的超图自注意模块，在该模块中骨架关节之间的超边逐渐更新，以呈现身体关节之间的关系，从而用于建模局部细小动作。 最后，为了利用情绪状态与微手势局部运动之间关系，我们在编码器输出处设计了一个情感识别头，并采用浅层架构并通过监督方式学习。整个端到端框架通过全面利用自我重构和监督信息在一次阶段联合训练中进行训练。所提出的方法在两个公开可用的数据集iMiGUE和SMG上进行了评估，在多个指标下达到了最佳性能，优于现有方法。 这段文字详细介绍了基于微手势的情感状态识别新方法，包括其新颖的超图增强Transformer框架、用于捕获细微运动的解码器设计以及如何通过编码器输出来学习情绪与动作之间的关系。该方法在两个公开数据集上的评估结果表明了它相对于当前技术的优势。
URL
https://arxiv.org/abs/2507.14867
PDF
-
Real Time Captioning of Sign Language Gestures in Video Meetings
2025-07-19 09:01:59Sharanya Mukherjee, Md Hishaam Akhtar, Kannadasan RAbstract
It has always been a rather tough task to communicate with someone possessing a hearing impairment. One of the most tested ways to establish such a communication is through the use of sign based languages. However, not many people are aware of the smaller intricacies involved with sign language. Sign language recognition using computer vision aims at eliminating the communication barrier between deaf-mute and ordinary people so that they can properly communicate with others. Recently the pandemic has left the whole world shaken up and has transformed the way we communicate. Video meetings have become essential for everyone, even people with a hearing disability. In recent studies, it has been found that people with hearing disabilities prefer to sign over typing during these video calls. In this paper, we are proposing a browser extension that will automatically translate sign language to subtitles for everyone else in the video call. The Large-scale dataset which contains more than 2000 Word-Level ASL videos, which were performed by over 100 signers will be used.
Abstract (translated)
与听障人士沟通一直是一项相当艰巨的任务。建立这种沟通的最常见方式之一是使用基于手势的语言（手语）。然而，许多人并不了解手语中包含的一些细微差别。计算机视觉中的手语识别技术旨在消除聋哑人和普通人之间的交流障碍，使他们能够更好地与其他人群进行沟通。 最近，疫情让全世界都受到了冲击，并且改变了我们的交流方式。视频会议已成为每个人的必需品，即使对于有听觉障碍的人也不例外。近期的研究表明，在这些视频通话中，听力受损的人更倾向于使用手语而不是打字。在这篇论文中，我们提出了一种浏览器扩展程序，该程序将自动把手语翻译成字幕供视频通话中的其他人观看。 为此，我们将采用一个大型数据集，其中包含超过2000个单词级别的ASL（美国手语）视频，这些视频由100多名手势使用者完成。
URL
https://arxiv.org/abs/2507.14543
PDF
-
Camera-based implicit mind reading by capturing higher-order semantic dynamics of human gaze within environmental context
2025-07-17 08:17:35Mengke Song, Yuge Xie, Qi Cui, Luming Li, Xinyu Liu, Guotao Wang, Chenglizhao Chen, Shanchen PangAbstract
Emotion recognition,as a step toward mind reading,seeks to infer internal states from external this http URL existing methods rely on explicit signals-such as facial expressions,speech,or gestures-that reflect only bodily responses and overlook the influence of environmental this http URL cues are often voluntary,easy to mask,and insufficient for capturing deeper,implicit emotions. Physiological signal-based approaches offer more direct access to internal states but require complex sensors that compromise natural behavior and limit this http URL-based methods typically rely on static fixation analysis and fail to capture the rich,dynamic interactions between gaze and the environment,and thus cannot uncover the deep connection between emotion and implicit this http URL address these limitations,we propose a novel camera-based,user-unaware emotion recognition approach that integrates gaze fixation patterns with environmental semantics and temporal this http URL standard HD cameras,our method unobtrusively captures users'eye appearance and head movements in natural settings-without the need for specialized hardware or active user this http URL these visual cues,the system estimates gaze trajectories over time and space, providing the basis for modeling the spatial, semantic,and temporal dimensions of gaze behavior. This allows us to capture the dynamic interplay between visual attention and the surrounding environment,revealing that emotions are not merely physiological responses but complex outcomes of human-environment this http URL proposed approach enables user-unaware,real-time,and continuous emotion recognition,offering high generalizability and low deployment cost.
Abstract (translated)
情绪识别作为一种迈向读心术的步骤，旨在从外部信号推断出内部状态。现有方法依赖于如面部表情、言语或手势等明确信号，这些信号仅反映了身体反应，并且忽略了环境线索的影响。由于这些提示往往是自愿产生的，容易被掩盖，并不足以捕捉深层的隐性情绪。 基于生理信号的方法提供了更直接访问内心状态的方式，但需要复杂的传感器，这会干扰自然行为并限制其应用范围。而基于视觉的方法通常依赖于静态注视分析，无法捕获凝视与环境之间的丰富、动态互动，因此也无法揭示情绪和隐性认知之间的深层联系。 为了解决这些局限性，我们提出了一种新颖的相机为基础的情绪识别方法，该方法不需用户知情且集成了注视模式与环境语义及时间线索。通过使用标准高清摄像机，我们的方法可以在自然环境中无须专用硬件或主动用户参与的情况下捕捉用户的外观和头部动作。 利用这些视觉提示，系统可以估计出随着时间及空间变化的凝视轨迹，从而为建模注视行为的空间、语义和时间维度提供了基础。这使我们能够捕捉到视觉注意力与周围环境之间的动态互动，揭示情绪不仅仅是一种生理反应，而是人类与环境复杂交互的结果。 所提出的这种方法实现了无意识用户参与、实时且连续的情绪识别，并具有高泛化能力和低部署成本。
URL
https://arxiv.org/abs/2507.12889
PDF
-
Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light
2025-07-15 16:53:14Mani Hamidi, Terrence W. DeaconAbstract
Three core tenets of reinforcement learning (RL)--concerning the definition of agency, the objective of learning, and the scope of the reward hypothesis--have been highlighted as key targets for conceptual revision, with major implications for theory and application. We propose a framework, inspired by open-ended evolutionary theory, to reconsider these three "dogmas." We revisit each assumption and address related concerns raised alongside them. To make our arguments relevant to RL as a model of biological learning, we first establish that evolutionary dynamics can plausibly operate within living brains over an individual's lifetime, and are not confined to cross-generational processes. We begin by revisiting the second dogma, drawing on evolutionary insights to enrich the "adaptation-rather-than-search" view of learning. We then address the third dogma regarding the limits of the reward hypothesis, using analogies from evolutionary fitness to illuminate the scalar reward vs. multi-objective debate. After discussing practical implications for exploration in RL, we turn to the first--and arguably most fundamental--issue: the absence of a formal account of agency. We argue that unlike the other two problems, the evolutionary paradigm alone cannot resolve the agency question, though it gestures in a productive direction. We advocate integrating ideas from origins-of-life theory, where the thermodynamics of sustenance and replication offer promising foundations for understanding agency and resource-constrained reinforcement learning in biological systems.
Abstract (translated)
强化学习（RL）的三个核心原则——关于代理定义、学习目标以及奖励假设范围的问题——已被提出需要概念上的重新评估，这对理论和应用都有重大影响。我们借鉴开放进化理论提出了一个框架来重新考虑这三条“信条”。我们将回顾每个假设，并解决与其相关的关切问题。 为了使我们的论点对强化学习作为生物学习模型的相关性得到认可，我们首先建立了一个观点：进化动力学可以在个体一生中在活脑内合理运作，并不仅仅局限于跨代过程。我们将从回溯第二个信条开始，利用进化视角来丰富“适应而非搜索”的学习观念。 接着，我们将讨论第三个信条中的奖励假设的局限性问题，通过类比于进化的适配度概念来阐明标量回报与多目标之间的辩论。在探讨强化学习中探索的实际影响后，我们转向最根本的问题：代理定义的缺乏。我们认为，与其他两个问题不同的是，仅仅依靠进化范式无法解决代理问题，尽管它指明了一个富有成效的方向。 我们提倡将起源理论的思想整合到这一框架内，在这些理论中，维持与复制的热力学基础为理解生物系统中的代理性和资源受限的强化学习提供了有希望的基础。
URL
https://arxiv.org/abs/2507.11482
PDF
-
Hand Gesture Recognition for Collaborative Robots Using Lightweight Deep Learning in Real-Time Robotic Systems
2025-07-14 08:40:24Muhtadin, I Wayan Agus Darmawan, Muhammad Hilmi Rusydiansyah, I Ketut Eddy Purnama, Chastine Fatichah, Mauridhi Hery PurnomoAbstract
Direct and natural interaction is essential for intuitive human-robot collaboration, eliminating the need for additional devices such as joysticks, tablets, or wearable sensors. In this paper, we present a lightweight deep learning-based hand gesture recognition system that enables humans to control collaborative robots naturally and efficiently. This model recognizes eight distinct hand gestures with only 1,103 parameters and a compact size of 22 KB, achieving an accuracy of 93.5%. To further optimize the model for real-world deployment on edge devices, we applied quantization and pruning using TensorFlow Lite, reducing the final model size to just 7 KB. The system was successfully implemented and tested on a Universal Robot UR5 collaborative robot within a real-time robotic framework based on ROS2. The results demonstrate that even extremely lightweight models can deliver accurate and responsive hand gesture-based control for collaborative robots, opening new possibilities for natural human-robot interaction in constrained environments.
Abstract (translated)
直接且自然的互动对于直观的人机协作至关重要，可以消除操纵杆、平板电脑或可穿戴传感器等额外设备的需求。本文提出了一种基于轻量级深度学习的手势识别系统，使人类能够自然高效地控制协作机器人。该模型仅使用1,103个参数和22KB的紧凑体积就能识别出8种不同的手部手势，并达到了93.5%的准确率。为了进一步优化模型以适应边缘设备上的实际部署，我们利用TensorFlow Lite进行了量化和裁剪处理，最终将模型大小减少到仅7KB。 该系统已在基于ROS2的实时机器人框架中成功实施并应用于Universal Robot UR5协作机器人上。结果表明，即使是极其轻量级的模型也能提供准确且响应迅速的手势控制功能，为受限环境中的自然人机交互开启了新的可能性。
URL
https://arxiv.org/abs/2507.10055
PDF
-
EHPE: A Segmented Architecture for Enhanced Hand Pose Estimation
2025-07-13 10:09:23Bolun Zheng, Xinjie Liu, Qianyu Zhang, Canjin Wang, Fangni Chen, Mingen XuAbstract
3D hand pose estimation has garnered great attention in recent years due to its critical applications in human-computer interaction, virtual reality, and related fields. The accurate estimation of hand joints is essential for high-quality hand pose estimation. However, existing methods neglect the importance of Distal Phalanx Tip (TIP) and Wrist in predicting hand joints overall and often fail to account for the phenomenon of error accumulation for distal joints in gesture estimation, which can cause certain joints to incur larger errors, resulting in misalignments and artifacts in the pose estimation and degrading the overall reconstruction quality. To address this challenge, we propose a novel segmented architecture for enhanced hand pose estimation (EHPE). We perform local extraction of TIP and wrist, thus alleviating the effect of error accumulation on TIP prediction and further reduce the predictive errors for all joints on this basis. EHPE consists of two key stages: In the TIP and Wrist Joints Extraction stage (TW-stage), the positions of the TIP and wrist joints are estimated to provide an initial accurate joint configuration; In the Prior Guided Joints Estimation stage (PG-stage), a dual-branch interaction network is employed to refine the positions of the remaining joints. Extensive experiments on two widely used benchmarks demonstrate that EHPE achieves state-of-the-arts performance. Code is available at this https URL.
Abstract (translated)
近年来，3D手部姿态估计因其在人机交互、虚拟现实及相关领域的关键应用而备受关注。准确地估算手关节位置对于高质量的手部姿态估计至关重要。然而，现有方法往往忽略了远端指骨顶端（TIP）和手腕在预测整体手关节时的重要性，并且通常未能考虑到手势估计中远端关节误差累积的现象，这会导致某些关节出现更大的误差，从而导致姿态估计中的错位和伪影，降低整体重建质量。 为了解决这一挑战，我们提出了一种新型分段架构的增强型手部姿态估计算法（EHPE）。我们在局部提取TIP和手腕位置，以此减轻误差累积对TIP预测的影响，并在此基础上进一步减少所有关节的预测错误。EHPE由两个关键阶段组成：在第一阶段（TIP和腕关节抽取阶段，TW阶段），估算TIP和腕关节的位置以提供初步准确的手部关节配置；在第二阶段（先验引导关节估计阶段，PG阶段），采用双分支交互网络来优化剩余关节的位置。 在两项广泛使用的基准测试中进行的大量实验表明，EHPE达到了最先进的性能。代码可以在提供的链接处获取。
URL
https://arxiv.org/abs/2507.09560
PDF
-
Online Micro-gesture Recognition Using Data Augmentation and Spatial-Temporal Attention
2025-07-13 06:38:17Pengyu Liu, Kun Li, Fei Wang, Yanyan Wei, Junhui She, Dan GuoAbstract
In this paper, we introduce the latest solution developed by our team, HFUT-VUT, for the Micro-gesture Online Recognition track of the IJCAI 2025 MiGA Challenge. The Micro-gesture Online Recognition task is a highly challenging problem that aims to locate the temporal positions and recognize the categories of multiple micro-gesture instances in untrimmed videos. Compared to traditional temporal action detection, this task places greater emphasis on distinguishing between micro-gesture categories and precisely identifying the start and end times of each instance. Moreover, micro-gestures are typically spontaneous human actions, with greater differences than those found in other human actions. To address these challenges, we propose hand-crafted data augmentation and spatial-temporal attention to enhance the model's ability to classify and localize micro-gestures more accurately. Our solution achieved an F1 score of 38.03, outperforming the previous state-of-the-art by 37.9%. As a result, our method ranked first in the Micro-gesture Online Recognition track.
Abstract (translated)
在这篇论文中，我们介绍了由我们团队HFUT-VUT为IJCAI 2025 MiGA挑战赛中的Micro-gesture Online Recognition（微手势在线识别）赛道开发的最新解决方案。微手势在线识别任务是一个极具挑战性的难题，其目标是在未剪辑的视频中定位多个微手势实例的时间位置并识别它们的类别。与传统的时序动作检测相比，该任务更侧重于区分不同的微手势类别，并精确地确定每个实例的开始和结束时间。此外，微手势通常是自发的人类行为，在细微差别上比其他人类动作更为不同。为了应对这些挑战，我们提出了手工设计的数据增强方法以及空间-时间注意力机制来提升模型对微手势分类与定位的能力。我们的解决方案取得了38.03的F1分数，超过了之前的最先进水平37.9%。因此，在微手势在线识别赛道上，我们的方法排名第一。
URL
https://arxiv.org/abs/2507.09512
PDF
-
MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion
2025-07-11 06:45:42Jihao Gu, Fei Wang, Kun Li, Yanyan Wei, Zhiliang Wu, Dan GuoAbstract
In this paper, we present MM-Gesture, the solution developed by our team HFUT-VUT, which ranked 1st in the micro-gesture classification track of the 3rd MiGA Challenge at IJCAI 2025, achieving superior performance compared to previous state-of-the-art methods. MM-Gesture is a multimodal fusion framework designed specifically for recognizing subtle and short-duration micro-gestures (MGs), integrating complementary cues from joint, limb, RGB video, Taylor-series video, optical-flow video, and depth video modalities. Utilizing PoseConv3D and Video Swin Transformer architectures with a novel modality-weighted ensemble strategy, our method further enhances RGB modality performance through transfer learning pre-trained on the larger MA-52 dataset. Extensive experiments on the iMiGUE benchmark, including ablation studies across different modalities, validate the effectiveness of our proposed approach, achieving a top-1 accuracy of 73.213%.
Abstract (translated)
在这篇论文中，我们介绍了MM-Gesture，这是由我们的团队HFUT-VUT开发的解决方案。该方案在IJCAI 2025年举行的第三届MiGA挑战赛微手势分类赛道上排名首位，并且其性能优于之前的最先进的方法。MM-Gesture是一个专门为识别细微且短时长的手势（简称MGs）而设计的多模态融合框架，它结合了关节、肢体、RGB视频、泰勒级数视频、光流视频和深度视频等多种模式中的互补线索。 我们的方法利用了PoseConv3D和Video Swin Transformer架构，并采用了一种新颖的模态加权集成策略。通过在更大的MA-52数据集上进行预训练，我们的方法进一步提升了RGB模态的表现。我们在iMiGUE基准测试上的广泛实验，包括不同模态下的消融研究，验证了我们提出的方法的有效性，实现了73.213%的顶级准确率。
URL
https://arxiv.org/abs/2507.08344
PDF
-
Democratizing High-Fidelity Co-Speech Gesture Video Generation
2025-07-09 13:02:12Xu Yang, Shaoli Huang, Shenbo Xie, Xuelin Chen, Yifei Liu, Changxing DingAbstract
Co-speech gesture video generation aims to synthesize realistic, audio-aligned videos of speakers, complete with synchronized facial expressions and body gestures. This task presents challenges due to the significant one-to-many mapping between audio and visual content, further complicated by the scarcity of large-scale public datasets and high computational demands. We propose a lightweight framework that utilizes 2D full-body skeletons as an efficient auxiliary condition to bridge audio signals with visual outputs. Our approach introduces a diffusion model conditioned on fine-grained audio segments and a skeleton extracted from the speaker's reference image, predicting skeletal motions through skeleton-audio feature fusion to ensure strict audio coordination and body shape consistency. The generated skeletons are then fed into an off-the-shelf human video generation model with the speaker's reference image to synthesize high-fidelity videos. To democratize research, we present CSG-405-the first public dataset with 405 hours of high-resolution videos across 71 speech types, annotated with 2D skeletons and diverse speaker demographics. Experiments show that our method exceeds state-of-the-art approaches in visual quality and synchronization while generalizing across speakers and contexts.
Abstract (translated)
演讲伴随的手势视频生成旨在合成与音频同步的、包含面部表情和身体动作的真实人物视频。这项任务面临着挑战，因为语音与视觉内容之间存在显著的一对多映射关系，并且由于大规模公开数据集的缺乏以及高昂的计算需求而变得更加复杂。我们提出了一种轻量级框架，该框架利用2D全身骨架作为高效的辅助条件来连接音频信号和视觉输出。我们的方法引入了一个基于细粒度音频片段并结合从演讲者参考图像中提取的骨骼信息进行条件化的扩散模型，通过融合骨骼-音频特征预测骨骼运动以确保严格的音视频同步和身体形状的一致性。生成的骨架随后被输入到现有的人脸视频生成模型中，并与演讲者的参考图像一起使用来合成高保真度的视频。 为了促进研究民主化，我们提供了CSG-405——这是第一个公开的数据集，包含71种不同类型的高质量语音，总时长为405小时，并且标注有2D骨架和多样化的说话人人口统计信息。实验表明，我们的方法在视觉质量和同步性方面超过了现有最佳的方法，并且能够跨说话者和上下文进行泛化。
URL
https://arxiv.org/abs/2507.06812
PDF
-
EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation
Abstract
We propose Expotion (Facial Expression and Motion Control for Multimodal Music Generation), a generative model leveraging multimodal visual controls - specifically, human facial expressions and upper-body motion - as well as text prompts to produce expressive and temporally accurate music. We adopt parameter-efficient fine-tuning (PEFT) on the pretrained text-to-music generation model, enabling fine-grained adaptation to the multimodal controls using a small dataset. To ensure precise synchronization between video and music, we introduce a temporal smoothing strategy to align multiple modalities. Experiments demonstrate that integrating visual features alongside textual descriptions enhances the overall quality of generated music in terms of musicality, creativity, beat-tempo consistency, temporal alignment with the video, and text adherence, surpassing both proposed baselines and existing state-of-the-art video-to-music generation models. Additionally, we introduce a novel dataset consisting of 7 hours of synchronized video recordings capturing expressive facial and upper-body gestures aligned with corresponding music, providing significant potential for future research in multimodal and interactive music generation.
Abstract (translated)
我们提出了一种名为Expotion（面部表情和动作控制的多模态音乐生成）的生成模型，该模型利用多种视觉控制手段——具体为人类面部表情及上半身动作，并结合文本提示来生成具有表现力且时间准确的音乐。我们在预训练的文本到音乐生成模型基础上采用参数高效的微调方法（PEFT），使得在使用小规模数据集的情况下也能进行细致调整以适应多模态控制。为了确保视频与音乐之间的时间同步，我们引入了一种时序平滑策略来对齐多种模式。 实验结果表明，将视觉特征与文本描述相结合能够显著提升生成音乐的整体质量，在音乐性、创意性、节奏一致性、时间上的视频同步以及遵循文本的程度方面均超越了现有的基线模型和最先进的视频到音乐的生成模型。此外，我们还引入了一个新的数据集，包含7小时的同步视频记录，这些记录捕捉了具有表现力的面部和上半身动作，并且与对应的音乐相匹配。这个数据集为未来多模态和交互式音乐生成的研究提供了巨大的潜力。
URL
https://arxiv.org/abs/2507.04955
PDF
