Abstract
We focus on the word-level visual lipreading, which requires recognizing the word being spoken, given only the video but not the audio. State-of-the-art methods explore the use of end-to-end neural networks, including a shallow (up to three layers) 3D convolutional neural network (CNN) + a deep 2D CNN (\emph{e.g.}, ResNet) as the front-end to extract visual features, and a recurrent neural network (\emph{e.g.}, bidirectional LSTM) as the back-end for classification. In this work, we propose to replace the shallow 3D CNNs + deep 2D CNNs front-end with recent successful deep 3D CNNs --- two-stream (\emph{i.e.}, grayscale video and optical flow streams) I3D. We evaluate different combinations of front-end and back-end modules with the grayscale video and optical flow inputs on the LRW dataset. The experiments show that, compared to the shallow 3D CNNs + deep 2D CNNs front-end, the deep 3D CNNs front-end with pre-training on the large-scale image and video datasets (\emph{e.g.}, ImageNet and Kinetics) can improve the classification accuracy. On the other hand, we demonstrate that using the optical flow input alone can achieve comparable performance as using the grayscale video as input. Moreover, the two-stream network using both the grayscale video and optical flow inputs can further improve the performance. Overall, our two-stream I3D front-end with a Bi-LSTM back-end results in an absolute improvement of 5.3\% over the previous art.
Abstract (translated)
我们将重点放在单词级的视觉唇读上，这需要识别所说的单词，只给出视频而不是音频。最先进的方法探索使用端到端的神经网络，包括一个浅层（最多三层）的三维卷积神经网络（CNN）+一个深二维CNN（如，resnet）作为前端提取视觉特征，以及一个循环神经网络（如，双向lstm）作为后端进行分类。在这项工作中，我们提议用最近成功的深三维CNN——两个流（即灰度视频和光流流流）i3d取代浅三维CNN+深二维CNN前端。我们评估了不同的前端和后端模块与LRW数据集上的灰度视频和光流输入的组合。实验表明，与浅三维CNN+深二维CNN前端相比，对大尺度图像和视频数据集（如、图像网和动力学）进行预训练的深三维CNN前端可以提高分类精度。另一方面，我们证明，单独使用光流输入可以获得与使用灰度视频输入相当的性能。此外，采用灰度视频和光流输入的双流网络可以进一步提高性能。总的来说，我们的两流i3d前端和一个bi lstm后端比以前的技术提高了5.3%。
URL
https://arxiv.org/abs/1905.02540
