Abstract
Captioning has attracted much attention in image and video understanding while little work examines audio captioning. This paper contributes a manually-annotated dataset on car scene, in extension to a previously published hospital audio captioning dataset. An encoder-decoder model with pretrained word embeddings and additional sentence loss is proposed. This current model can accelerate the training process and generate semantically correct but unseen unique sentences. We test the model on the current car dataset, previous Hospital Dataset and the Joint Dataset, indicating its generalization capability across different scenes. Further, we make an effort to provide a better objective evaluation metric, namely the BERT similarity score. It compares the semantic-level similarity and compensates for drawbacks of N-gram based metrics like BLEU, namely high scores for word-similar sentences. This new metric demonstrates higher correlation with human evaluation. However, though detailed audio captions can now be automatically generated, human annotations still outperform model captions in many aspects.
Abstract (translated)
字幕在图像和视频理解中引起了广泛的关注，而很少有人研究音频字幕。本文提供了一个汽车场景的手动注释数据集，扩展到以前发布的医院音频字幕数据集。提出了一种预训练嵌入词和附加语句丢失的编码器-解码器模型。该模型可以加快训练过程，生成语义正确但看不见的独特句子。我们在当前的汽车数据集、以前的医院数据集和联合数据集上对模型进行了测试，表明其在不同场景下的泛化能力。此外，我们还努力提供一个更好的客观评价指标，即伯特相似性得分。它比较了语义水平的相似性，弥补了BLeu等基于n-gram的度量的缺点，即单词相似句的高分。这一新的指标显示出与人类评价的更高相关性。然而，尽管现在可以自动生成详细的音频标题，但人工注释在许多方面仍然优于模型标题。
URL
https://arxiv.org/abs/1905.13448
