Abstract
Animated movies are captivating for their unique character designs and imaginative storytelling, yet they pose significant challenges for existing recognition systems. Unlike the consistent visual patterns detected by conventional face recognition methods, animated characters exhibit extreme diversity in their appearance, motion, and deformation. In this work, we propose an audio-visual pipeline to enable automatic and robust animated character recognition, and thereby enhance character-centric understanding of animated movies. Central to our approach is the automatic construction of an audio-visual character bank from online sources. This bank contains both visual exemplars and voice (audio) samples for each character, enabling subsequent multi-modal character recognition despite long-tailed appearance distributions. Building on accurate character recognition, we explore two downstream applications: Audio Description (AD) generation for visually impaired audiences, and character-aware subtitling for the hearing impaired. To support research in this domain, we introduce CMD-AM, a new dataset of 75 animated movies with comprehensive annotations. Our character-centric pipeline demonstrates significant improvements in both accessibility and narrative comprehension for animated content over prior face-detection-based approaches. For the code and dataset, visit this https URL.
Abstract (translated)
动画电影因其独特的角色设计和富有想象力的叙事而引人入胜，但现有的识别系统在处理这些内容时面临重大挑战。与传统面部识别方法检测到的一致性视觉模式不同，动画角色在外貌、动作及变形方面展现出了极大的多样性。为此，我们提出了一种音频-视觉管道来实现自动且稳健的动画人物识别，并借此增强对动画电影中以角色为中心的理解。 我们的核心方法是通过在线资源自动构建一个集视觉范例和声音（音频）样本于一体的音频-视觉角色库。这使得即便是在出现长尾分布的情况下，也能够进行多模态的角色识别。在此基础上，我们探讨了两个下游应用：为视障观众生成音轨描述(AD)以及针对听障观众的具有角色意识的字幕。 为了支持该领域的研究，我们引入了一个名为CMD-AM的新数据集，其中包括75部动画电影及其全面注释。我们的以角色为中心的管道在提高无障碍访问和叙事理解方面相对于传统的基于面部检测的方法展示出了显著的进步。 有关代码和数据集，请访问提供的URL链接。
URL
https://arxiv.org/abs/2509.12204
