Abstract
This technical report describes our QuAVF@NTU-NVIDIA submission to the Ego4D Talking to Me (TTM) Challenge 2023. Based on the observation from the TTM task and the provided dataset, we propose to use two separate models to process the input videos and audio. By doing so, we can utilize all the labeled training data, including those without bounding box labels. Furthermore, we leverage the face quality score from a facial landmark prediction model for filtering noisy face input data. The face quality score is also employed in our proposed quality-aware fusion for integrating the results from two branches. With the simple architecture design, our model achieves 67.4% mean average precision (mAP) on the test set, which ranks first on the leaderboard and outperforms the baseline method by a large margin. Code is available at: this https URL
Abstract (translated)
本技术报告描述了我们的 QuAVF@NTU-NVIDIA 提交了 Ego4D 与我对话 (TTM) 挑战 2023。基于 TTM 任务和提供的数据集观察,我们建议使用两个独立的模型来处理输入视频和音频。通过这样做,我们可以利用所有标记的训练数据,包括没有界框标签的数据。此外,我们利用面部地标预测模型的面部质量得分来过滤噪声的面部输入数据。面部质量得分也被用于我们提出的质量 aware 融合方法,以整合两个分支的结果。通过简单的架构设计,我们的模型在测试集上取得了 67.4% 的平均绝对精度 (mAP),在排行榜上排名第一,比基准方法高出很多。代码可在 this https URL 中找到。
URL
https://arxiv.org/abs/2306.17404
