Abstract
Sign language video retrieval plays a key role in facilitating information access for the deaf community. Despite significant advances in video-text retrieval, the complexity and inherent uncertainty of sign language preclude the direct application of these techniques. Previous methods achieve the mapping between sign language video and text through fine-grained modal alignment. However, due to the scarcity of fine-grained annotation, the uncertainty inherent in sign language video is underestimated, limiting the further development of sign language retrieval tasks. To address this challenge, we propose a novel Uncertainty-aware Probability Distribution Retrieval (UPRet), that conceptualizes the mapping process of sign language video and text in terms of probability distributions, explores their potential interrelationships, and enables flexible mappings. Experiments on three benchmarks demonstrate the effectiveness of our method, which achieves state-of-the-art results on How2Sign (59.1%), PHOENIX-2014T (72.0%), and CSL-Daily (78.4%).
Abstract (translated)
手语视频检索在方便听觉社区的信息访问中扮演着关键角色。尽管在视频-文本检索方面取得了显著的进步，但是手语视频的复杂性和固有不确定性使得这些技术无法直接应用。以前的方法通过精细语义模态对齐在形式化模型中实现了手语视频和文本之间的映射。然而，由于精细标注的稀缺性，手语视频中的不确定性被低估，限制了手语检索任务的进一步发展。为了应对这一挑战，我们提出了一个名为不确定性感知概率分布检索（UPRet）的新方法，将手语视频和文本的映射过程置于概率分布下，探索它们之间的潜在关系，并实现灵活的映射。在三个基准上的实验证明了我们方法的有效性，实现了How2Sign（59.1%）、PHOENIX-2014T（72.0%）和CSL-Daily（78.4%）等最先进的结果。
URL
https://arxiv.org/abs/2405.19689
