Abstract
Animal vocalizations provide crucial insights for wildlife assessment, particularly in complex environments such as forests, aiding species identification and ecological monitoring. Recent advances in deep learning have enabled automatic species classification from their vocalizations. However, classifying species unseen during training remains challenging. To address this limitation, we introduce AnimalCLAP, a taxonomy-aware language-audio framework comprising a new dataset and model that incorporate hierarchical biological information. Specifically, our vocalization dataset consists of 4,225 hours of recordings covering 6,823 species, annotated with 22 ecological traits. The AnimalCLAP model is trained on this dataset to align audio and textual representations using taxonomic structures, improving the recognition of unseen species. We demonstrate that our proposed model effectively infers ecological and biological attributes of species directly from their vocalizations, achieving superior performance compared to CLAP. Our dataset, code, and models will be publicly available at this https URL.
Abstract (translated)
动物发声为野生动物评估提供了关键见解，尤其在森林等复杂环境中，有助于物种识别和生态监测。近年来深度学习的进展已实现基于发声的自动物种分类。然而，对训练中未出现过的物种进行分类仍具挑战性。针对这一局限，我们提出了AnimalCLAP——一个融合分层生物信息的分类学感知音文框架，包含新数据集与模型。具体而言，我们的发声数据集包含4,225小时录音，覆盖6,823个物种，并标注了22项生态特征。AnimalCLAP模型基于该数据集训练，通过分类学结构对齐音频与文本表征，提升了未见过物种的识别能力。研究表明，我们提出的模型能直接从动物发声推断物种的生态与生物属性，性能优于CLAP。数据集、代码及模型将在此网址公开。
URL
https://arxiv.org/abs/2603.22053
