Abstract
We focus on the word-level visual lipreading, which requires to decode the word from the speaker's video. Recently, many state-of-the-art visual lipreading methods explore the end-to-end trainable deep models, involving the use of 2D convolutional networks (e.g., ResNet) as the front-end visual feature extractor and the sequential model (e.g., Bi-LSTM or Bi-GRU) as the back-end. Although a deep 2D convolution neural network can provide informative image-based features, it ignores the temporal motion existing between the adjacent frames. In this work, we investigate the spatial-temporal capacity power of I3D (Inflated 3D ConvNet) for visual lipreading. We demonstrate that, after pre-trained on the large-scale video action recognition dataset (e.g., Kinetics), our models show a considerable improvement of performance on the task of lipreading. A comparison between a set of video model architectures and input data representation is also reported. Our extensive experiments on LRW shows that a two-stream I3D model with RGB video and optical flow as the inputs achieves the state-of-the-art performance.
Abstract (translated)
我们专注于单词级的视觉唇读,它需要从说话人的视频中解码单词。最近,许多最先进的视觉唇读方法探索端到端可培训的深度模型,包括使用二维卷积网络(如resnet)作为前端视觉特征提取程序,使用顺序模型(如bi lstm或bi gru)作为后端。虽然深二维卷积神经网络可以提供基于图像的信息特征,但它忽略了相邻帧之间存在的时间运动。在这项工作中,我们研究了视觉唇读的i3d(膨胀的3d convnet)的时空容量能力。我们证明,在对大规模视频动作识别数据集(例如,动力学)进行了预先培训后,我们的模型在唇读任务上表现出了相当大的性能改进。本文还比较了一组视频模型体系结构和输入数据表示。我们在LRW上的大量实验表明,以RGB视频和光流为输入的两流i3D模型达到了最先进的性能。
URL
https://arxiv.org/abs/1903.09616