Abstract
Person recognition systems often rely on audio, visual, or behavioral cues, but real-world conditions frequently result in missing or degraded modalities. To address this challenge, we propose a Trimodal person identification framework that integrates voice, face, and gesture modalities, while remaining robust to modality loss. Our approach leverages multi-task learning to process each modality independently, followed by a cross-attention and gated fusion mechanisms to facilitate interaction across modalities. Moreover, a confidence-weighted fusion strategy dynamically adapts to missing and low-quality data, ensuring optimal classification even in Unimodal or Bimodal scenarios. We evaluate our method on CANDOR, a newly introduced interview-based multimodal dataset, which we benchmark for the first time. Our results demonstrate that the proposed Trimodal system achieves 99.18% Top-1 accuracy on person identification tasks, outperforming conventional Unimodal and late-fusion approaches. In addition, we evaluate our model on the VoxCeleb1 dataset as a benchmark and reach 99.92% accuracy in Bimodal mode. Moreover, we show that our system maintains high accuracy even when one or two modalities are unavailable, making it a robust solution for real-world person recognition applications. The code and data for this work are publicly available.
Abstract (translated)
人员识别系统通常依赖于音频、视觉或行为线索，但在实际环境中，这些模态往往会出现缺失或退化的情况。为了解决这一挑战，我们提出了一种三模态（语音、面部和手势）的人体识别框架，该框架能够在模态丢失的情况下保持稳健性。我们的方法利用多任务学习来独立处理每个模态，并通过跨模态交互的交叉注意力机制和门控融合机制加以增强。此外，采用基于置信度加权的融合策略可以动态适应缺失或低质量的数据，在单模态或多模态场景下确保最佳分类效果。 我们在CANDOR数据集上评估了这种方法，该数据集是新近推出的基于访谈的多模态数据集，并首次对它进行了基准测试。实验结果显示，我们的三模态系统在人员识别任务中达到了99.18%的第一名准确率，优于传统的单模态和晚期融合方法。此外，在VoxCeleb1数据集中，我们通过双模态模式达到了99.92%的精度。 更重要的是，即使当一个或两个模态不可用时，我们的系统仍能保持高准确性，使其成为现实世界中人员识别应用的一种稳健解决方案。本研究的相关代码和数据已公开发布。
URL
https://arxiv.org/abs/2512.14961
