Gesture
-
My Emotion on your face: The use of Facial Keypoint Detection to preserve Emotions in Latent Space Editing
2025-05-09 21:10:27Jingrui He, Andrew Stephen McGoughAbstract
Generative Adversarial Network approaches such as StyleGAN/2 provide two key benefits: the ability to generate photo-realistic face images and possessing a semantically structured latent space from which these images are created. Many approaches have emerged for editing images derived from vectors in the latent space of a pre-trained StyleGAN/2 models by identifying semantically meaningful directions (e.g., gender or age) in the latent space. By moving the vector in a specific direction, the ideal result would only change the target feature while preserving all the other features. Providing an ideal data augmentation approach for gesture research as it could be used to generate numerous image variations whilst keeping the facial expressions intact. However, entanglement issues, where changing one feature inevitably affects other features, impacts the ability to preserve facial expressions. To address this, we propose the use of an addition to the loss function of a Facial Keypoint Detection model to restrict changes to the facial expressions. Building on top of an existing model, adding the proposed Human Face Landmark Detection (HFLD) loss, provided by a pre-trained Facial Keypoint Detection model, to the original loss function. We quantitatively and qualitatively evaluate the existing and our extended model, showing the effectiveness of our approach in addressing the entanglement issue and maintaining the facial expression. Our approach achieves up to 49% reduction in the change of emotion in our experiments. Moreover, we show the benefit of our approach by comparing with state-of-the-art models. By increasing the ability to preserve the facial gesture and expression during facial transformation, we present a way to create human face images with fixed expression but different appearances, making it a reliable data augmentation approach for Facial Gesture and Expression research.
Abstract (translated)
生成对抗网络（如StyleGAN/2）提供了两个关键优势：能够生成逼真的面部图像以及拥有语义结构化的潜在空间，从中可以创建这些图像。许多方法已经出现，通过识别预训练的StyleGAN/2模型中的潜在空间中具有特定语义意义的方向（例如性别或年龄），来编辑从该空间向量衍生出的图片。理想的结果是移动向量以仅改变目标特征而不影响其他所有特征。这种数据增强方法为手势研究提供了理想的途径，因为它可以生成许多图像变化的同时保持面部表情不变。然而，纠缠问题的存在使得在改变一个特性时不可避免地会影响到其它特性，从而难以保留面部表情。 为了应对这一挑战，我们建议通过向面部关键点检测模型的损失函数中添加一项来限制对表情的变化。基于现有的模型，在原始损失函数的基础上增加我们提出的“人类面部地标检测”（HFLD）损失项，该损失项来自预训练的关键点检测模型。通过对现有模型和扩展后的模型进行定量和定性评估，证明了我们的方法在解决纠缠问题并保持面部表情方面是有效的。实验结果表明，在我们的实验中，情感变化减少了多达49%。 此外，我们通过与最先进的模型进行比较展示了我们方法的优势。通过增强在面部转换过程中保留面部手势和表达的能力，我们可以创建具有固定表情但不同外观的人脸图像，从而为面部手势和表情研究提供可靠的增广数据方法。
URL
https://arxiv.org/abs/2505.06436
PDF
-
Multimodal Emotion Coupling via Speech-to-Facial and Bodily Gestures in Dyadic Interaction
2025-05-08 10:55:54Von Ralph Dane Marquez Herbuela, Yukie NagaiAbstract
Human emotional expression emerges through coordinated vocal, facial, and gestural signals. While speech face alignment is well established, the broader dynamics linking emotionally expressive speech to regional facial and hand motion remains critical for gaining a deeper insight into how emotional and behavior cues are communicated in real interactions. Further modulating the coordination is the structure of conversational exchange like sequential turn taking, which creates stable temporal windows for multimodal synchrony, and simultaneous speech, often indicative of high arousal moments, disrupts this alignment and impacts emotional clarity. Understanding these dynamics enhances realtime emotion detection by improving the accuracy of timing and synchrony across modalities in both human interactions and AI systems. This study examines multimodal emotion coupling using region specific motion capture from dyadic interactions in the IEMOCAP corpus. Speech features included low level prosody, MFCCs, and model derived arousal, valence, and categorical emotions (Happy, Sad, Angry, Neutral), aligned with 3D facial and hand marker displacements. Expressive activeness was quantified through framewise displacement magnitudes, and speech to gesture prediction mapped speech features to facial and hand movements. Nonoverlapping speech consistently elicited greater activeness particularly in the lower face and mouth. Sadness showed increased expressivity during nonoverlap, while anger suppressed gestures during overlaps. Predictive mapping revealed highest accuracy for prosody and MFCCs in articulatory regions while arousal and valence had lower and more context sensitive correlations. Notably, hand speech synchrony was enhanced under low arousal and overlapping speech, but not for valence.
Abstract (translated)
人类情感表达通过协调的语音、面部和手势信号实现。虽然言语与面部表情的一致性已经得到了很好的研究，但将情感丰富的言语与特定区域的面部运动及手部动作联系起来的动力学特性对于深入理解情感和行为线索在实际互动中的传达方式至关重要。 进一步影响这种协调的是对话结构，例如轮换发言模式，在这些模式中，多模态同步会形成稳定的时序窗口。而同时说话的情况通常表明高度兴奋的状态，这会破坏这种一致性和影响情绪清晰度。理解这些动力学有助于提高实时情感检测的准确性，特别是在人类互动和人工智能系统中的不同模态之间的时间和同步性方面。 这项研究使用了IEMOCAP语料库中一对多交互的区域特定运动捕捉数据来考察跨模式情感耦合。语音特征包括低层声调、梅尔频率倒谱系数（MFCCs）以及通过模型得到的情绪唤醒度、效价及分类情绪（快乐、悲伤、愤怒和中立），与3D面部和手部标记位移对齐。表达积极性是通过对帧级别的位移幅度进行量化来测量的，同时将语音特征映射到面部和手部运动以预测手势。 研究发现，在非重叠说话期间，下颌和嘴唇区域表现出更高的活性。悲伤情绪在非重叠时间窗口中表现为更强的情感表现力；而愤怒则会抑制重叠说话时的手势。预测映射显示声调和MFCCs在发音区域具有最高的准确性，而唤醒度和效价的相关性较低且更依赖于具体语境。 值得注意的是，在低唤醒状态下以及言语重叠期间，手部运动与语音同步的增强效果明显；而在效价值方面则没有观察到类似效应。
URL
https://arxiv.org/abs/2506.10010
PDF
-
Inter-Diffusion Generation Model of Speakers and Listeners for Effective Communication
2025-05-08 07:00:58Jinhe Huang, Yongkang Cheng, Yuming Hang, Gaoge Han, Jinewei Li, Jing Zhang, Xingjian GuAbstract
Full-body gestures play a pivotal role in natural interactions and are crucial for achieving effective communication. Nevertheless, most existing studies primarily focus on the gesture generation of speakers, overlooking the vital role of listeners in the interaction process and failing to fully explore the dynamic interaction between them. This paper innovatively proposes an Inter-Diffusion Generation Model of Speakers and Listeners for Effective Communication. For the first time, we integrate the full-body gestures of listeners into the generation framework. By devising a novel inter-diffusion mechanism, this model can accurately capture the complex interaction patterns between speakers and listeners during communication. In the model construction process, based on the advanced diffusion model architecture, we innovatively introduce interaction conditions and the GAN model to increase the denoising step size. As a result, when generating gesture sequences, the model can not only dynamically generate based on the speaker's speech information but also respond in realtime to the listener's feedback, enabling synergistic interaction between the two. Abundant experimental results demonstrate that compared with the current state-of-the-art gesture generation methods, the model we proposed has achieved remarkable improvements in the naturalness, coherence, and speech-gesture synchronization of the generated gestures. In the subjective evaluation experiments, users highly praised the generated interaction scenarios, believing that they are closer to real life human communication situations. Objective index evaluations also show that our model outperforms the baseline methods in multiple key indicators, providing more powerful support for effective communication.
Abstract (translated)
全身手势在自然互动中扮演着关键角色，并且对于实现有效沟通至关重要。然而，大多数现有研究主要关注说话者的手势生成，忽略了倾听者在互动过程中的重要性，并未能充分探索双方之间的动态交互作用。本文创新地提出了一种用于有效沟通的演讲者与倾听者之间相互扩散生成模型（Inter-Diffusion Generation Model of Speakers and Listeners）。首次将听众的全身手势融入到生成框架中，通过设计一种新颖的互扩散机制，该模型能够准确捕捉说话人和倾听人在交流过程中复杂的交互模式。在模型构建过程中，基于先进的扩散模型架构，我们创新性地引入了互动条件以及GAN（生成对抗网络）模型，以增加去噪步骤大小。因此，在生成手势序列时，该模型不仅能根据发言人的语言信息动态生成，还能实时回应听者的反馈，从而实现两者之间的协同交互作用。 大量的实验结果表明，与当前最先进的手势生成方法相比，我们提出的模型在所生成手势的自然性、连贯性和话语-手势同步性方面取得了显著改进。在主观评价实验中，用户对生成的互动场景给予了高度赞扬，认为它们更接近现实生活中的交流情况。客观指标评估也显示，我们的模型在多个关键指标上超越了基准方法，为有效沟通提供了更为强大的支持。
URL
https://arxiv.org/abs/2505.04996
PDF
-
Input-Specific and Universal Adversarial Attack Generation for Spiking Neural Networks in the Spiking Domain
2025-05-07 19:49:18Spyridon Raptis, Haralampos-G. StratigopoulosAbstract
As Spiking Neural Networks (SNNs) gain traction across various applications, understanding their security vulnerabilities becomes increasingly important. In this work, we focus on the adversarial attacks, which is perhaps the most concerning threat. An adversarial attack aims at finding a subtle input perturbation to fool the network's decision-making. We propose two novel adversarial attack algorithms for SNNs: an input-specific attack that crafts adversarial samples from specific dataset inputs and a universal attack that generates a reusable patch capable of inducing misclassification across most inputs, thus offering practical feasibility for real-time deployment. The algorithms are gradient-based operating in the spiking domain proving to be effective across different evaluation metrics, such as adversarial accuracy, stealthiness, and generation time. Experimental results on two widely used neuromorphic vision datasets, NMNIST and IBM DVS Gesture, show that our proposed attacks surpass in all metrics all existing state-of-the-art methods. Additionally, we present the first demonstration of adversarial attack generation in the sound domain using the SHD dataset.
Abstract (translated)
随着尖峰神经网络（SNNs）在各种应用中的普及，了解其安全漏洞变得越来越重要。在这项工作中，我们重点关注对抗性攻击，这是最令人担忧的威胁之一。对抗性攻击旨在找到微小的输入扰动来误导网络的决策过程。我们提出了两种针对SNN的新颖对抗性攻击算法：一种是特定于输入的攻击，该攻击从特定数据集输入中创建对抗样本；另一种是通用攻击，它生成一个可重复使用的补丁，在大多数输入上都能引发分类错误，从而为实时部署提供实际可行性。这些算法基于梯度并在尖峰域内操作，证明在各种评估指标（如对抗准确性、隐蔽性和生成时间）下都非常有效。 我们在两个广泛使用的人工神经形态视觉数据集NMNIST和IBM DVS手势数据集上的实验结果显示，我们提出的攻击方法在所有指标上都超越了现有的最先进方法。此外，我们还首次展示了使用SHD数据集在声音域中生成对抗性攻击的方法。
URL
https://arxiv.org/abs/2505.06299
PDF
-
FoodTrack: Estimating Handheld Food Portions with Egocentric Video
Abstract
Accurately tracking food consumption is crucial for nutrition and health monitoring. Traditional approaches typically require specific camera angles, non-occluded images, or rely on gesture recognition to estimate intake, making assumptions about bite size rather than directly measuring food volume. We propose the FoodTrack framework for tracking and measuring the volume of hand-held food items using egocentric video which is robust to hand occlusions and flexible with varying camera and object poses. FoodTrack estimates food volume directly, without relying on intake gestures or fixed assumptions about bite size, offering a more accurate and adaptable solution for tracking food consumption. We achieve absolute percentage loss of approximately 7.01% on a handheld food object, improving upon a previous approach that achieved a 16.40% mean absolute percentage error in its best case, under less flexible conditions.
Abstract (translated)
准确追踪食物摄入量对于营养和健康监测至关重要。传统方法通常需要特定的拍摄角度、非遮挡图像或依赖手势识别来估计摄取量，它们会假设咬食大小而非直接测量食物体积。我们提出了一种名为FoodTrack的框架，用于使用第一人称视角视频（egocentric video）追踪并测量手持食物物品的体积，该方法能够应对手部遮挡问题，并且对不同相机和对象姿态变化具有灵活性。FoodTrack直接估计食物体积，而不依赖于摄取手势或固定咬食大小假设，提供了一种更准确、更具适应性的解决方案来追踪食物摄入量。我们在手持食品物体上实现了约7.01%的绝对百分比损失，在之前一种方法的最佳情况下也仅达成了16.40%的平均绝对百分比误差（在更为严格的条件下）。 该框架具有更高的准确性，并且能够在更加复杂的使用环境中有效运作，从而为食物摄入量的精确追踪和监测提供了新的可能。
URL
https://arxiv.org/abs/2505.04055
PDF
-
Robust Understanding of Human-Robot Social Interactions through Multimodal Distillation
2025-05-06 19:52:18Tongfei Bian, Mathieu Chollet, Tanaya GuhaAbstract
The need for social robots and agents to interact and assist humans is growing steadily. To be able to successfully interact with humans, they need to understand and analyse socially interactive scenes from their (robot's) perspective. Works that model social situations between humans and agents are few; and even those existing ones are often too computationally intensive to be suitable for deployment in real time or on real world scenarios with limited available information. We propose a robust knowledge distillation framework that models social interactions through various multimodal cues, yet is robust against incomplete and noisy information during inference. Our teacher model is trained with multimodal input (body, face and hand gestures, gaze, raw images) that transfers knowledge to a student model that relies solely on body pose. Extensive experiments on two publicly available human-robot interaction datasets demonstrate that the our student model achieves an average accuracy gain of 14.75\% over relevant baselines on multiple downstream social understanding task even with up to 51\% of its input being corrupted. The student model is highly efficient: it is $<1$\% in size of the teacher model in terms of parameters and uses $\sim 0.5$\textperthousand~FLOPs of that in the teacher model. Our code will be made public during publication.
Abstract (translated)
社会机器人和代理与人类互动并提供帮助的需求正在稳步增长。为了能够成功地与人类交互，它们需要从自身的（机器人的）视角理解并分析社交场景。然而，模拟人机之间社交情景的作品并不多；即使现有的作品也往往计算复杂度高，不适合在实际场景中实时部署或处理信息有限的情况。我们提出了一种鲁棒的知识蒸馏框架，该框架通过多种多模态线索来建模社会互动，并且能够应对推理过程中的不完整和噪音信息。我们的教师模型使用多模态输入（身体、面部及手部动作、视线以及原始图像）进行训练，并将知识传递给仅依赖于身体姿态的学徒模型。在两个公开的人机交互数据集上的大量实验表明，即使有高达51%的输入被破坏的情况下，我们的学徒模型在多个下游社交理解任务上仍比相关基准提高了平均14.75%的准确度。该学生模型非常高效：它的参数量仅为教师模型的不到1%，并且使用的计算资源（FLOPs）约为教师模型的0.5‰左右。我们将在论文发表时公开代码。
URL
https://arxiv.org/abs/2505.06278
PDF
-
Beyond the Monitor: Mixed Reality Visualization and AI for Enhanced Digital Pathology Workflow
2025-05-05 16:46:53Jai Prakash Veerla, Partha Sai Guttikonda, Helen H. Shang, Mohammad Sadegh Nasr, Cesar Torres, Jacob M. LuberAbstract
Pathologists rely on gigapixel whole-slide images (WSIs) to diagnose diseases like cancer, yet current digital pathology tools hinder diagnosis. The immense scale of WSIs, often exceeding 100,000 X 100,000 pixels, clashes with the limited views traditional monitors offer. This mismatch forces constant panning and zooming, increasing pathologist cognitive load, causing diagnostic fatigue, and slowing pathologists' adoption of digital methods. PathVis, our mixed-reality visualization platform for Apple Vision Pro, addresses these challenges. It transforms the pathologist's interaction with data, replacing cumbersome mouse-and-monitor navigation with intuitive exploration using natural hand gestures, eye gaze, and voice commands in an immersive workspace. PathVis integrates AI to enhance diagnosis. An AI-driven search function instantly retrieves and displays the top five similar patient cases side-by-side, improving diagnostic precision and efficiency through rapid comparison. Additionally, a multimodal conversational AI assistant offers real-time image interpretation support and aids collaboration among pathologists across multiple Apple devices. By merging the directness of traditional pathology with advanced mixed-reality visualization and AI, PathVis improves diagnostic workflows, reduces cognitive strain, and makes pathology practice more effective and engaging. The PathVis source code and a demo video are publicly available at: this https URL
Abstract (translated)
病理学家依赖于巨像素全切片图像（WSIs）来诊断癌症等疾病，然而现有的数字病理工具却妨碍了这一过程。由于WSI的规模庞大，通常超过10万乘以10万像素，与传统显示器提供的有限视角形成了冲突。这种不匹配迫使病理学家不断地进行平移和缩放操作，增加了认知负担，导致诊断疲劳，并减缓了他们采用数字方法的速度。 PathVis是我们为苹果Vision Pro设计的混合现实可视化平台，旨在解决这些问题。它通过在沉浸式工作空间中使用自然的手势、视线以及语音命令来替代笨拙的老鼠与显示器导航方式，从而改变了病理学家对数据的交互方式。PathVis将人工智能集成到诊断过程中：一个基于AI的搜索功能能够立即检索并展示前五名类似患者的病例，实现快速比较以提高诊断精度和效率。 此外，一个多模态对话式AI助手为实时图像解读提供支持，并帮助多台苹果设备上的病理学家进行协作。通过结合传统病理学的直接性和先进的混合现实可视化以及人工智能技术，PathVis改进了诊断工作流程，减少了认知负担，并使病理实践更加有效和吸引人。 PathVis源代码及演示视频公开链接：[此 URL](this%20https%20URL)（请将“this https URL”替换为实际提供的链接）。
URL
https://arxiv.org/abs/2505.02780
PDF
-
A Multimodal Framework for Explainable Evaluation of Soft Skills in Educational Environments
2025-05-03 11:54:35Jared D. T. Guerrero-Sosa, Francisco P. Romero, V\'ictor Hugo Men\'endez-Dom\'inguez, Jesus Serrano-Guerrero, Andres Montoro-Montarroso, Jose A. OlivasAbstract
In the rapidly evolving educational landscape, the unbiased assessment of soft skills is a significant challenge, particularly in higher education. This paper presents a fuzzy logic approach that employs a Granular Linguistic Model of Phenomena integrated with multimodal analysis to evaluate soft skills in undergraduate students. By leveraging computational perceptions, this approach enables a structured breakdown of complex soft skill expressions, capturing nuanced behaviours with high granularity and addressing their inherent uncertainties, thereby enhancing interpretability and reliability. Experiments were conducted with undergraduate students using a developed tool that assesses soft skills such as decision-making, communication, and creativity. This tool identifies and quantifies subtle aspects of human interaction, such as facial expressions and gesture recognition. The findings reveal that the framework effectively consolidates multiple data inputs to produce meaningful and consistent assessments of soft skills, showing that integrating multiple modalities into the evaluation process significantly improves the quality of soft skills scores, making the assessment work transparent and understandable to educational stakeholders.
Abstract (translated)
在迅速变化的教育环境中，对软技能进行公正评估是一个重要的挑战，尤其是在高等教育领域。本文提出了一种采用现象颗粒语言模型结合多模态分析的模糊逻辑方法，用于评估本科生的软技能。通过利用计算感知，这种方法能够系统地分解复杂的软技能表达，捕捉细微的行为特征，并解决其内在不确定性，从而提高解释性和可靠性。 研究团队使用开发的一款工具对本科学生进行了实验，该工具可以评估决策能力、沟通能力和创造力等软技能。这款工具能识别并量化人类互动中的微妙方面，如面部表情和手势识别。研究结果表明，所提出的框架能够有效整合多种数据输入，生成有意义且一致的软技能评估结果。这证明将多模态融入评估过程显著提高了软技能评分的质量，使评估工作对教育相关方更加透明易懂。
URL
https://arxiv.org/abs/2505.01794
PDF
-
Co$^{3}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion
2025-05-03 08:51:19Xingqun Qi, Yatian Wang, Hengyuan Zhang, Jiahao Pan, Wei Xue, Shanghang Zhang, Wenhan Luo, Qifeng Liu, Yike GuoAbstract
Generating gestures from human speech has gained tremendous progress in animating virtual avatars. While the existing methods enable synthesizing gestures cooperated by individual self-talking, they overlook the practicality of concurrent gesture modeling with two-person interactive conversations. Moreover, the lack of high-quality datasets with concurrent co-speech gestures also limits handling this issue. To fulfill this goal, we first construct a large-scale concurrent co-speech gesture dataset that contains more than 7M frames for diverse two-person interactive posture sequences, dubbed GES-Inter. Additionally, we propose Co$^3$Gesture, a novel framework that enables coherent concurrent co-speech gesture synthesis including two-person interactive movements. Considering the asymmetric body dynamics of two speakers, our framework is built upon two cooperative generation branches conditioned on separated speaker audio. Specifically, to enhance the coordination of human postures with respect to corresponding speaker audios while interacting with the conversational partner, we present a Temporal Interaction Module (TIM). TIM can effectively model the temporal association representation between two speakers' gesture sequences as interaction guidance and fuse it into the concurrent gesture generation. Then, we devise a mutual attention mechanism to further holistically boost learning dependencies of interacted concurrent motions, thereby enabling us to generate vivid and coherent gestures. Extensive experiments demonstrate that our method outperforms the state-of-the-art models on our newly collected GES-Inter dataset. The dataset and source code are publicly available at \href{this https URL}{\textit{this https URL}}.
Abstract (translated)
从人类言语生成手势在虚拟角色动画领域取得了重大进展。尽管现有的方法能够合成个人自言自语时的手势，但它们忽视了同时模拟双人互动对话中的手势模型的实际应用性。此外，缺乏高质量的包含同步共话语音手势的数据集也限制了这一问题的处理能力。为了实现这一目标，我们首先构建了一个大规模的并发共话语音手势数据集——GES-Inter，该数据集中包含了超过700万帧用于多样化的双人互动姿势序列。另外，我们提出了Co$^3$Gesture，这是一个全新的框架，能够生成包括两人互动动作在内的连贯的同步共话语音手势。 考虑到两位说话者身体动态的不对称性，我们的框架基于两个根据各自语音条件化生成的合作分支建立而成。具体而言，为了增强人体姿态与相应讲话人音频之间的协调，在与对话伙伴互动时，我们提出了一种时间交互模块（TIM）。TIM可以有效建模两名演讲者手势序列之间的时间关联表示作为交互指导，并将其融入到同步手势生成中。此外，我们设计了一个互注意机制，进一步整体提升相互作用的并发动作的学习依赖性，从而能够生成生动且连贯的手势。 广泛的实验表明，我们的方法在我们新收集的GES-Inter数据集上优于当前最佳模型。该数据集和源代码可以在\href{this https URL}{此链接}公开获取。
URL
https://arxiv.org/abs/2505.01746
PDF
-
Towards Cognitive Collaborative Robots: Semantic-Level Integration and Explainable Control for Human-Centric Cooperation
2025-05-02 17:08:23Jaehong OhAbstract
This is a preprint of a review article that has not yet undergone peer review. The content is intended for early dissemination and academic discussion. The final version may differ upon formal publication. As the Fourth Industrial Revolution reshapes industrial paradigms, human-robot collaboration (HRC) has transitioned from a desirable capability to an operational necessity. In response, collaborative robots (Cobots) are evolving beyond repetitive tasks toward adaptive, semantically informed interaction with humans and environments. This paper surveys five foundational pillars enabling this transformation: semantic-level perception, cognitive action planning, explainable learning and control, safety-aware motion design, and multimodal human intention recognition. We examine the role of semantic mapping in transforming spatial data into meaningful context, and explore cognitive planning frameworks that leverage this context for goal-driven decision-making. Additionally, we analyze explainable reinforcement learning methods, including policy distillation and attention mechanisms, which enhance interpretability and trust. Safety is addressed through force-adaptive control and risk-aware trajectory planning, while seamless human interaction is supported via gaze and gesture-based intent recognition. Despite these advancements, challenges such as perception-action disjunction, real-time explainability limitations, and incomplete human trust persist. To address these, we propose a unified Cognitive Synergy Architecture, integrating all modules into a cohesive framework for truly human-centric cobot collaboration.
Abstract (translated)
这是一篇尚未经过同行评审的综述文章的预印本。内容旨在早期传播和学术讨论，正式发表时可能会有所更改。随着第四次工业革命重塑产业范式，人机协作（HRC）已经从一种可取的能力转变为运营上的必要性。为此，协作机器人（Cobots）正在超越重复性的任务，迈向适应性强、语义驱动的人机及环境互动。本文概述了实现这一转型的五大基础支柱：语义级感知、认知行为规划、解释性学习和控制、安全意识的动作设计以及多模态人类意图识别。 我们探讨了语义映射在将空间数据转化为有意义上下文中的作用，并研究了利用这种背景信息进行目标导向决策的认知计划框架。此外，本文还分析了解释性强化学习方法（包括策略蒸馏和注意力机制），这些方法增强了解释性和信任度。安全问题通过力适应控制及风险感知轨迹规划得到解决；而无缝的人机互动则由注视与手势意图识别来支持。 尽管取得了上述进展，但诸如感知-行动断层、实时可解释性限制以及人类信任不足等问题仍然存在挑战。为应对这些挑战，我们提出了一种统一的认知协同架构，将所有模块整合进一个连贯的整体框架中，从而实现真正以人为本的协作机器人合作。
URL
https://arxiv.org/abs/2505.03815
PDF
-
Confidence-based Intent Prediction for Teleoperation in Bimanual Robotic Suturing
2025-04-29 13:41:37Zhaoyang Jacopo Hu, Haozheng Xu, Sion Kim, Yanan Li, Ferdinando Rodriguez y Baena, Etienne BurdetAbstract
Robotic-assisted procedures offer enhanced precision, but while fully autonomous systems are limited in task knowledge, difficulties in modeling unstructured environments, and generalisation abilities, fully manual teleoperated systems also face challenges such as delay, stability, and reduced sensory information. To address these, we developed an interactive control strategy that assists the human operator by predicting their motion plan at both high and low levels. At the high level, a surgeme recognition system is employed through a Transformer-based real-time gesture classification model to dynamically adapt to the operator's actions, while at the low level, a Confidence-based Intention Assimilation Controller adjusts robot actions based on user intent and shared control paradigms. The system is built around a robotic suturing task, supported by sensors that capture the kinematics of the robot and task dynamics. Experiments across users with varying skill levels demonstrated the effectiveness of the proposed approach, showing statistically significant improvements in task completion time and user satisfaction compared to traditional teleoperation.
Abstract (translated)
机器人辅助程序提供了更高的精确度，但完全自主系统在任务知识、模拟非结构化环境的难度以及泛化能力方面存在局限。与此同时，全手动遥操作系统也面临着延迟、稳定性降低和感觉信息减少等问题。为解决这些问题，我们开发了一种互动控制策略，该策略通过预测操作员的动作计划，在高层次和低层次上辅助人类操作员。 在高层次上，采用基于Transformer的实时手势分类模型来识别手术动作（surgeme），使系统能够根据操作者的行动动态适应。在低层次上，一种基于信心度的意图融合控制器会根据用户意图及共享控制范式调整机器人的行为。 该系统围绕一个机器人缝合任务构建，并利用传感器捕捉机器人的运动学和任务动力学。经过不同技能水平用户的实验测试，结果显示所提出的方法在任务完成时间和用户满意度方面相比传统遥操作有统计显著的改进。
URL
https://arxiv.org/abs/2504.20761
PDF
-
A Real-Time Gesture-Based Control Framework
Abstract
We introduce a real-time, human-in-the-loop gesture control framework that can dynamically adapt audio and music based on human movement by analyzing live video input. By creating a responsive connection between visual and auditory stimuli, this system enables dancers and performers to not only respond to music but also influence it through their movements. Designed for live performances, interactive installations, and personal use, it offers an immersive experience where users can shape the music in real time. The framework integrates computer vision and machine learning techniques to track and interpret motion, allowing users to manipulate audio elements such as tempo, pitch, effects, and playback sequence. With ongoing training, it achieves user-independent functionality, requiring as few as 50 to 80 samples to label simple gestures. This framework combines gesture training, cue mapping, and audio manipulation to create a dynamic, interactive experience. Gestures are interpreted as input signals, mapped to sound control commands, and used to naturally adjust music elements, showcasing the seamless interplay between human interaction and machine response.
Abstract (translated)
我们介绍了一种实时的人机交互姿态控制框架，该框架能够通过分析实时视频输入动态地根据人体动作调整音频和音乐。通过建立视觉与听觉刺激之间的响应联系，此系统使舞者和表演者不仅能够回应音乐，还能通过他们的动作影响音乐。专为现场演出、互动装置和个人使用设计，它提供了一种沉浸式体验，在这种体验中用户可以实时塑造音乐。该框架整合了计算机视觉和机器学习技术来跟踪和解释运动，让用户能够操控音频元素如节奏、音高、效果以及播放顺序。 通过持续训练，此系统实现了独立于用户的功能，仅需50到80个样本就可以标记简单的手势。这个框架结合了姿态培训、提示映射和音频操作，以创建一种动态且互动的体验。姿势被解读为输入信号，并将其映射至声音控制命令，用于自然地调整音乐元素，展示了人机交互与机器响应之间的无缝融合。
URL
https://arxiv.org/abs/2504.19460
PDF
-
Generative AI for Character Animation: A Comprehensive Survey of Techniques, Applications, and Future Directions
2025-04-27 00:09:31Mohammad Mahdi Abootorabi, Omid Ghahroodi, Pardis Sadat Zahraei, Hossein Behzadasl, Alireza Mirrokni, Mobina Salimipanah, Arash Rasouli, Bahar Behzadipour, Sara Azarnoush, Benyamin Maleki, Erfan Sadraiye, Kiarash Kiani Feriz, Mahdi Teymouri Nahad, Ali Moghadasi, Abolfazl Eshagh Abianeh, Nizi Nazar, Hamid R. Rabiee, Mahdieh Soleymani Baghshah, Meisam Ahmadi, Ehsaneddin AsgariAbstract
Generative AI is reshaping art, gaming, and most notably animation. Recent breakthroughs in foundation and diffusion models have reduced the time and cost of producing animated content. Characters are central animation components, involving motion, emotions, gestures, and facial expressions. The pace and breadth of advances in recent months make it difficult to maintain a coherent view of the field, motivating the need for an integrative review. Unlike earlier overviews that treat avatars, gestures, or facial animation in isolation, this survey offers a single, comprehensive perspective on all the main generative AI applications for character animation. We begin by examining the state-of-the-art in facial animation, expression rendering, image synthesis, avatar creation, gesture modeling, motion synthesis, object generation, and texture synthesis. We highlight leading research, practical deployments, commonly used datasets, and emerging trends for each area. To support newcomers, we also provide a comprehensive background section that introduces foundational models and evaluation metrics, equipping readers with the knowledge needed to enter the field. We discuss open challenges and map future research directions, providing a roadmap to advance AI-driven character-animation technologies. This survey is intended as a resource for researchers and developers entering the field of generative AI animation or adjacent fields. Resources are available at: this https URL.
Abstract (translated)
生成式AI正在重塑艺术、游戏，尤其是动画领域。近期在基础模型和扩散模型方面的突破减少了制作动画内容所需的时间和成本。角色是动画中的关键组成部分，涉及到动作、情感、姿态和面部表情等元素。最近几个月进步的速度与广度使得保持对该领域的全面理解变得困难，因此有必要进行一次综合性的回顾。不同于以往将化身（avatar）、手势或面部动画单独处理的综述，本次调查提供了一个对生成式AI在角色动画中的主要应用进行全面审视的独特视角。 我们首先探讨了面部动画、表情渲染、图像合成、化身创建、手势建模、动作合成、物体生成和纹理合成等领域的最新进展。我们将突出领先的研究成果、实际部署案例、常用的数据集以及每个领域内的新兴趋势。为了帮助新入门者，我们也提供了一个全面的背景部分，介绍了基础模型和评估指标，为读者提供了进入该领域的知识准备。 此外，我们还讨论了当前面临的挑战，并指出了未来研究的方向，为推进AI驱动的角色动画技术绘制了一条路线图。这篇综述旨在成为正在进入生成式AI动画领域或相关领域的研究人员和技术开发者的一个资源。更多相关信息可在以下网址找到：[此URL]（请将“this https URL”替换为实际提供的链接）。
URL
https://arxiv.org/abs/2504.19056
PDF
-
POET: Prompt Offset Tuning for Continual Human Action Adaptation
2025-04-25 04:11:24Prachi Garg, Joseph K J, Vineeth N Balasubramanian, Necati Cihan Camgoz, Chengde Wan, Kenrick Kin, Weiguang Si, Shugao Ma, Fernando De La TorreAbstract
As extended reality (XR) is redefining how users interact with computing devices, research in human action recognition is gaining prominence. Typically, models deployed on immersive computing devices are static and limited to their default set of classes. The goal of our research is to provide users and developers with the capability to personalize their experience by adding new action classes to their device models continually. Importantly, a user should be able to add new classes in a low-shot and efficient manner, while this process should not require storing or replaying any of user's sensitive training data. We formalize this problem as privacy-aware few-shot continual action recognition. Towards this end, we propose POET: Prompt-Offset Tuning. While existing prompt tuning approaches have shown great promise for continual learning of image, text, and video modalities; they demand access to extensively pretrained transformers. Breaking away from this assumption, POET demonstrates the efficacy of prompt tuning a significantly lightweight backbone, pretrained exclusively on the base class data. We propose a novel spatio-temporal learnable prompt offset tuning approach, and are the first to apply such prompt tuning to Graph Neural Networks. We contribute two new benchmarks for our new problem setting in human action recognition: (i) NTU RGB+D dataset for activity recognition, and (ii) SHREC-2017 dataset for hand gesture recognition. We find that POET consistently outperforms comprehensive benchmarks. Source code at this https URL.
Abstract (translated)
随着扩展现实（XR）重新定义了用户与计算设备的交互方式，人类动作识别领域的研究正变得越来越重要。传统上，在沉浸式计算设备中部署的模型是静态的，并且局限于其默认的一组类。我们的研究目标是为用户提供能力，让他们能够通过向设备模型持续添加新的动作类别来个性化他们的体验。至关重要的是，用户应当能够在低样本和高效的方式下添加新类别，而这一过程不应涉及存储或重放任何用户的敏感训练数据。我们将这个问题形式化为隐私感知的少样本连续动作识别问题。 为此，我们提出了POET：Prompt-Offset Tuning（提示偏移微调）。尽管现有的提示微调方法在图像、文本和视频模式的持续学习中显示出巨大的潜力；但它们要求访问大规模预训练的变压器。POET打破了这一假设，展示了对仅基于基础类数据进行预先训练的轻量级骨干网络进行提示微调的有效性。 我们提出了一种新颖的空间-时间可学习提示偏移微调方法，并且首次将这种提示微调应用于图神经网络（Graph Neural Networks, GNN）中。为了我们的新问题设置，我们在人类动作识别领域贡献了两个新的基准测试：(i) NTU RGB+D 数据集用于活动识别；以及 (ii) SHREC-2017 数据集用于手部姿态识别。 我们发现POET在全面的基准测试中表现一致出色。源代码可在提供的链接地址获取。
URL
https://arxiv.org/abs/2504.18059
PDF
-
SSLR: A Semi-Supervised Learning Method for Isolated Sign Language Recognition
2025-04-23 11:59:52Hasan Algafri, Hamzah Luqman, Sarah Alyami, Issam LaradjiAbstract
Sign language is the primary communication language for people with disabling hearing loss. Sign language recognition (SLR) systems aim to recognize sign gestures and translate them into spoken language. One of the main challenges in SLR is the scarcity of annotated datasets. To address this issue, we propose a semi-supervised learning (SSL) approach for SLR (SSLR), employing a pseudo-label method to annotate unlabeled samples. The sign gestures are represented using pose information that encodes the signer's skeletal joint points. This information is used as input for the Transformer backbone model utilized in the proposed approach. To demonstrate the learning capabilities of SSL across various labeled data sizes, several experiments were conducted using different percentages of labeled data with varying numbers of classes. The performance of the SSL approach was compared with a fully supervised learning-based model on the WLASL-100 dataset. The obtained results of the SSL model outperformed the supervised learning-based model with less labeled data in many cases.
Abstract (translated)
手语是听觉障碍人士的主要交流语言。手语识别（SLR）系统旨在识别手势并将其翻译成口语。在SLR中，主要挑战之一是标注数据的稀缺性。为了解决这个问题，我们提出了一种针对SLR的半监督学习（SSL）方法（SSLR），采用伪标签技术来注释未标记的数据样本。手语通过包含表演者骨骼关节点信息的姿态表示进行编码，并将其作为Transformer骨干模型输入。为了展示不同标注数据量下SSL的学习能力，使用WLASL-100数据集进行了多项实验，在这些实验中采用了具有不同类别数量的各种比例的标注数据。将所提出的半监督学习方法与基于完全有监督学习的方法在相同数据集上进行对比。结果显示，在许多情况下，当可用标记数据较少时，SSLR模型的表现优于基于监督学习的模型。
URL
https://arxiv.org/abs/2504.16640
PDF
-
Time Frequency Analysis of EMG Signal for Gesture Recognition using Fine grained Features
2025-04-20 18:51:10Parshuram N. Aarotale, Ajita RattaniAbstract
Electromyography (EMG) based hand gesture recognition converts forearm muscle activity into control commands for prosthetics, rehabilitation, and human computer interaction. This paper proposes a novel approach to EMG-based hand gesture recognition that uses fine-grained classification and presents XMANet, which unifies low-level local and high level semantic cues through cross layer mutual attention among shallow to deep CNN experts. Using stacked spectrograms and scalograms derived from the Short Time Fourier Transform (STFT) and Wavelet Transform (WT), we benchmark XMANet against ResNet50, DenseNet-121, MobileNetV3, and EfficientNetB0. Experimental results on the Grabmyo dataset indicate that, using STFT, the proposed XMANet model outperforms the baseline ResNet50, EfficientNetB0, MobileNetV3, and DenseNet121 models with improvement of approximately 1.72%, 4.38%, 5.10%, and 2.53%, respectively. When employing the WT approach, improvements of around 1.57%, 1.88%, 1.46%, and 2.05% are observed over the same baselines. Similarly, on the FORS EMG dataset, the XMANet(ResNet50) model using STFT shows an improvement of about 5.04% over the baseline ResNet50. In comparison, the XMANet(DenseNet121) and XMANet(MobileNetV3) models yield enhancements of approximately 4.11% and 2.81%, respectively. Moreover, when using WT, the proposed XMANet achieves gains of around 4.26%, 9.36%, 5.72%, and 6.09% over the baseline ResNet50, DenseNet121, MobileNetV3, and EfficientNetB0 models, respectively. These results confirm that XMANet consistently improves performance across various architectures and signal processing techniques, demonstrating the strong potential of fine grained features for accurate and robust EMG classification.
Abstract (translated)
肌电图（EMG）基于的手势识别技术将前臂肌肉活动转化为假肢控制、康复和人机交互的指令。本文提出了一种新的基于EMG的手势识别方法，该方法采用细粒度分类，并提出了XMANet模型，通过浅层到深层卷积神经网络专家之间的跨层相互注意力机制统一低级局部特征与高级语义线索。 使用从短时傅里叶变换（STFT）和小波变换（WT）导出的堆叠频谱图和斯卡洛格拉姆进行基准测试，我们将XMANet与ResNet50、DenseNet-121、MobileNetV3和EfficientNetB0进行了对比。在Grabmyo数据集上的实验结果表明，在使用STFT时，提出的XMANet模型比基线模型ResNet50、EfficientNetB0、MobileNetV3和DenseNet121分别提高了约1.72%、4.38%、5.10%和2.53%。同样，在使用WT方法时，与这些基准相比，XMANet的改进约为1.57%、1.88%、1.46%和2.05%。 在FORS EMG数据集上，当使用STFT时，基于ResNet50的XMANet模型比基线ResNet50提高了约5.04%。相比之下，基于DenseNet121和MobileNetV3的XMANet模型分别提供了大约4.11%和2.81%的改进。当使用WT时，提出的XMANet在基准ResNet50、DenseNet121、MobileNetV3和EfficientNetB0上分别获得了约4.26%、9.36%、5.72%和6.09%的增益。 这些结果确认了XMANet能够在多种架构和信号处理技术中持续提高性能，展示了细粒度特征在准确且鲁棒的EMG分类中的强大潜力。
URL
https://arxiv.org/abs/2504.14708
PDF
-
Multi-Sensor Fusion-Based Mobile Manipulator Remote Control for Intelligent Smart Home Assistance
2025-04-17 22:33:38Xiao Jin, Bo Xiao, Huijiang Wang, Wendong Wang, Zhenhua YuAbstract
This paper proposes a wearable-controlled mobile manipulator system for intelligent smart home assistance, integrating MEMS capacitive microphones, IMU sensors, vibration motors, and pressure feedback to enhance human-robot interaction. The wearable device captures forearm muscle activity and converts it into real-time control signals for mobile manipulation. The wearable device achieves an offline classification accuracy of 88.33\%\ across six distinct movement-force classes for hand gestures by using a CNN-LSTM model, while real-world experiments involving five participants yield a practical accuracy of 83.33\%\ with an average system response time of 1.2 seconds. In Human-Robot synergy in navigation and grasping tasks, the robot achieved a 98\%\ task success rate with an average trajectory deviation of only 3.6 cm. Finally, the wearable-controlled mobile manipulator system achieved a 93.3\%\ gripping success rate, a transfer success of 95.6\%\, and a full-task success rate of 91.1\%\ during object grasping and transfer tests, in which a total of 9 object-texture combinations were evaluated. These three experiments' results validate the effectiveness of MEMS-based wearable sensing combined with multi-sensor fusion for reliable and intuitive control of assistive robots in smart home scenarios.
Abstract (translated)
这篇论文提出了一种用于智能智能家居辅助的可穿戴设备控制移动机械臂系统，集成了MEMS电容式麦克风、IMU传感器、振动电机和压力反馈来增强人机交互。该可穿戴装置捕捉前臂肌肉活动，并将其转换为实时控制信号以进行移动操作。通过使用CNN-LSTM模型，该可穿戴装置在六种不同的手势力度分类上实现了88.33%的离线分类准确率，而涉及五名参与者的实际实验中获得了83.33%的实际准确性，平均系统响应时间为1.2秒。在导航和抓取任务中的人机协同作用中，机器人达到了98%的任务成功率，并且平均轨迹偏差仅为3.6厘米。最后，在物体抓取和转移测试中，可穿戴控制的移动机械臂系统实现了93.3%的成功夹持率、95.6%的成功传输率以及91.1%的整体任务成功率，共评估了九种不同的物体纹理组合。这三项实验的结果验证了基于MEMS的可穿戴传感技术结合多传感器融合在智能家庭场景中实现辅助机器人可靠且直观控制的有效性。
URL
https://arxiv.org/abs/2504.13370
PDF
-
ADAT: Time-Series-Aware Adaptive Transformer Architecture for Sign Language Translation
2025-04-16 10:20:11Nada Shahin, Leila IsmailAbstract
Current sign language machine translation systems rely on recognizing hand movements, facial expressions and body postures, and natural language processing, to convert signs into text. Recent approaches use Transformer architectures to model long-range dependencies via positional encoding. However, they lack accuracy in recognizing fine-grained, short-range temporal dependencies between gestures captured at high frame rates. Moreover, their high computational complexity leads to inefficient training. To mitigate these issues, we propose an Adaptive Transformer (ADAT), which incorporates components for enhanced feature extraction and adaptive feature weighting through a gating mechanism to emphasize contextually relevant features while reducing training overhead and maintaining translation accuracy. To evaluate ADAT, we introduce MedASL, the first public medical American Sign Language dataset. In sign-to-gloss-to-text experiments, ADAT outperforms the encoder-decoder transformer, improving BLEU-4 accuracy by 0.1% while reducing training time by 14.33% on PHOENIX14T and 3.24% on MedASL. In sign-to-text experiments, it improves accuracy by 8.7% and reduces training time by 2.8% on PHOENIX14T and achieves 4.7% higher accuracy and 7.17% faster training on MedASL. Compared to encoder-only and decoder-only baselines in sign-to-text, ADAT is at least 6.8% more accurate despite being up to 12.1% slower due to its dual-stream structure.
Abstract (translated)
当前的手语机器翻译系统依赖于识别手部动作、面部表情和身体姿态，并通过自然语言处理将手势转换为文本。近期的方法采用了Transformer架构，利用位置编码来建模长距离依赖关系。然而，它们在捕捉高帧率下细微且短时间内的手势依赖关系方面缺乏准确性。此外，其计算复杂度很高，导致训练效率低下。 为了缓解这些问题，我们提出了一种自适应Transformer（ADAT），该模型通过引入增强特征提取和自适应特征加权的组件来解决这一问题，并通过门控机制强调上下文相关的特征，同时减少训练开销并保持翻译准确性。为了评估ADAT的效果，我们推出了MedASL，这是首个公开的医学美国手语数据集。 在手势到文字（经由词符）的实验中，在PHOENIX14T和MedASL上，ADAT的表现优于编码器-解码器Transformer模型，BLEU-4精度提升了0.1%，训练时间分别缩短了14.33%和3.24%。在直接手势到文字的实验中，在PHOENIX14T数据集上，ADAT提高了8.7%的准确率，并减少了2.8%的训练时间；而在MedASL上，其表现更是提升了4.7%的准确度并加快了7.17%的训练速度。 与手势到文字任务中的编码器和解码器基线相比，尽管ADAT由于其双流结构最多慢至多12.1%，但在准确性方面至少提高了6.8%。
URL
https://arxiv.org/abs/2504.11942
PDF
-
Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehicles
2025-04-15 05:04:25Tonko E. W. Bossen, Andreas M{\o}gelmose, Ross GreerAbstract
In autonomous driving, it is crucial to correctly interpret traffic gestures (TGs), such as those of an authority figure providing orders or instructions, or a pedestrian signaling the driver, to ensure a safe and pleasant traffic environment for all road users. This study investigates the capabilities of state-of-the-art vision-language models (VLMs) in zero-shot interpretation, focusing on their ability to caption and classify human gestures in traffic contexts. We create and publicly share two custom datasets with varying formal and informal TGs, such as 'Stop', 'Reverse', 'Hail', etc. The datasets are "Acted TG (ATG)" and "Instructive TG In-The-Wild (ITGI)". They are annotated with natural language, describing the pedestrian's body position and gesture. We evaluate models using three methods utilizing expert-generated captions as baseline and control: (1) caption similarity, (2) gesture classification, and (3) pose sequence reconstruction similarity. Results show that current VLMs struggle with gesture understanding: sentence similarity averages below 0.59, and classification F1 scores reach only 0.14-0.39, well below the expert baseline of 0.70. While pose reconstruction shows potential, it requires more data and refined metrics to be reliable. Our findings reveal that although some SOTA VLMs can interpret zero-shot human traffic gestures, none are accurate and robust enough to be trustworthy, emphasizing the need for further research in this domain.
Abstract (translated)
在自动驾驶领域，正确解读交通手势（TG）至关重要。这些手势可能来自指挥交通的执法人员或行人向驾驶员发出信号的情况，以确保所有道路使用者的安全和顺畅环境。本研究探讨了目前最先进的视觉-语言模型（VLMs）在零样本解释中的能力，重点关注其对交通环境中人类手势进行描述和分类的能力。我们创建并公开分享了两个定制数据集，包含不同形式的正式和非正式TG，例如“停止”、“倒车”、“招手”等。这些数据集分别是“表演TG（ATG）”和“实际指示TG（ITGI）”，它们用自然语言注释，描述行人的身体位置和手势。 我们采用三种方法评估模型性能，并以专家生成的说明作为基准和控制： 1. 句子相似度 2. 手势分类 3. 姿态序列重建相似度 结果显示：目前的VLM在理解手势方面存在困难。句子相似性的平均值低于0.59，分类F1分数仅为0.14-0.39，远低于专家基准的0.70。虽然姿态重构显示出潜力，但需要更多的数据和更精细的度量标准才能变得可靠。 我们的研究结果表明，尽管一些最先进的VLM能够解释零样本的人类交通手势，但是没有一种模型在准确性和可靠性方面足够强大以确保信任，这突显了该领域进一步研究的需求。
URL
https://arxiv.org/abs/2504.10873
PDF
-
IlluSign: Illustrating Sign Language Videos by Leveraging the Attention Mechanism
2025-04-15 02:53:32Janna Bruner, Amit Moryossef, Lior WolfAbstract
Sign languages are dynamic visual languages that involve hand gestures, in combination with non manual elements such as facial expressions. While video recordings of sign language are commonly used for education and documentation, the dynamic nature of signs can make it challenging to study them in detail, especially for new learners and educators. This work aims to convert sign language video footage into static illustrations, which serve as an additional educational resource to complement video content. This process is usually done by an artist, and is therefore quite costly. We propose a method that illustrates sign language videos by leveraging generative models' ability to understand both the semantic and geometric aspects of images. Our approach focuses on transferring a sketch like illustration style to video footage of sign language, combining the start and end frames of a sign into a single illustration, and using arrows to highlight the hand's direction and motion. While many style transfer methods address domain adaptation at varying levels of abstraction, applying a sketch like style to sign languages, especially for hand gestures and facial expressions, poses a significant challenge. To tackle this, we intervene in the denoising process of a diffusion model, injecting style as keys and values into high resolution attention layers, and fusing geometric information from the image and edges as queries. For the final illustration, we use the attention mechanism to combine the attention weights from both the start and end illustrations, resulting in a soft combination. Our method offers a cost effective solution for generating sign language illustrations at inference time, addressing the lack of such resources in educational materials.
Abstract (translated)
手语是一种动态的视觉语言，通过手势结合面部表情等非手动元素来表达。虽然视频录制的手语在教育和记录中被广泛使用，但由于其动态特性，在细节上进行研究具有挑战性，尤其是对于新手学习者和教师来说更为困难。本项目旨在将手语视频片段转化为静态插图，作为补充视频内容的教育资源。这一过程通常由艺术家完成，因此成本较高。我们提出了一种方法，利用生成模型理解图像的语义和几何方面的能力来绘制手语视频。我们的方法重点在于将类似素描的风格转移到手语视频上，并结合手势开始帧和结束帧以形成单一插图，同时使用箭头突出双手的方向和运动。 虽然许多风格迁移的方法在不同程度上解决了领域适应的问题，但对手势语言特别是对于手部动作和面部表情进行素描化处理仍然具有挑战性。为此，我们对扩散模型的去噪过程进行了干预，在高分辨率注意力层中注入样式作为键值，并融合图像和边缘中的几何信息作为查询。最后，通过注意机制结合开始帧和结束帧插图的关注权重来产生最终的插图，形成柔和组合。 我们的方法提供了一种成本效益高的解决方案，可以在推断时生成手语插图，弥补了教育材料中此类资源的不足。
URL
https://arxiv.org/abs/2504.10822
PDF
