Abstract
Sign Language Translation (SLT) is a complex cross-modal task requiring the integration of Manual Signals (MS) and Non-Manual Signals (NMS). While recent gloss-free SLT methods have made strides in translating manual gestures, they frequently overlook the semantic criticality of facial expressions, resulting in ambiguity when distinct concepts share identical manual articulations. To address this, we present **EASLT** (**E**motion-**A**ware **S**ign **L**anguage **T**ranslation), a framework that treats facial affect not as auxiliary information, but as a robust semantic anchor. Unlike methods that relegate facial expressions to a secondary role, EASLT incorporates a dedicated emotional encoder to capture continuous affective dynamics. These representations are integrated via a novel *Emotion-Aware Fusion* (EAF) module, which adaptively recalibrates spatio-temporal sign features based on affective context to resolve semantic ambiguities. Extensive evaluations on the PHOENIX14T and CSL-Daily benchmarks demonstrate that EASLT establishes advanced performance among gloss-free methods, achieving BLEU-4 scores of 26.15 and 22.80, and BLEURT scores of 61.0 and 57.8, respectively. Ablation studies confirm that explicitly modeling emotion effectively decouples affective semantics from manual dynamics, significantly enhancing translation fidelity. Code is available at this https URL.
Abstract (translated)
手语翻译（SLT）是一项复杂的跨模态任务，需要整合手势信号（MS）和非手动信号（NMS）。尽管最近的无助词符SLT方法在翻译手动动作方面取得了进展，但它们常常忽视面部表情所包含的重要语义信息，在不同概念的手势相同的情况下导致理解上的模糊性。为解决这一问题，我们提出了**EASLT** (**E**motion-**A**ware **S**ign **L**anguage **T**ranslation)，这是一个将面部情感视为重要语义锚点而非辅助信息的框架。与那些将面部表情置于次要地位的方法不同，EASLT引入了一个专用的情感编码器来捕捉连续的情感动态。通过一种新颖的*情感感知融合*（EAF）模块，这些表示被整合在一起，该模块根据情感背景自适应地重新校准空间-时间手势特征以解决语义模糊问题。 在PHOENIX14T和CSL-Daily基准测试上进行的广泛评估表明，EASLT在无助词符的方法中表现出色，在这两个数据集上分别实现了BLEU-4得分为26.15和22.80，以及BLEURT得分分别为61.0和57.8。消融实验证实了显式建模情感能够有效分离情感语义与手动动态，并显著提升翻译的准确性。 代码可在以下链接获取：[提供链接处]（请将[提供链接处]替换为实际提供的代码链接）。
URL
https://arxiv.org/abs/2601.03549
