Abstract
Video Captioning and Summarization have become very popular in the recent years due to advancements in Sequence Modelling, with the resurgence of Long-Short Term Memory networks (LSTMs) and introduction of Gated Recurrent Units (GRUs). Existing architectures extract spatio-temporal features using CNNs and utilize either GRUs or LSTMs to model dependencies with soft attention layers. These attention layers do help in attending to the most prominent features and improve upon the recurrent units, however, these models suffer from the inherent drawbacks of the recurrent units themselves. The introduction of the Transformer model has driven the Sequence Modelling field into a new direction. In this project, we implement a Transformer-based model for Video captioning, utilizing 3D CNN architectures like C3D and Two-stream I3D for video extraction. We also apply certain dimensionality reduction techniques so as to keep the overall size of the model within limits. We finally present our results on the MSVD and ActivityNet datasets for Single and Dense video captioning tasks respectively.
Abstract (translated)
近年来，随着长短期记忆网络（LSTMS）的兴起和门控循环单元（GRU）的引入，视频字幕和摘要技术在序列建模方面取得了很大进展。现有的体系结构使用CNN提取时空特征，并利用GRUS或LSTM来建模具有软注意层的依赖关系。这些关注层确实有助于关注最显著的特征并改进循环单元，但是这些模型本身存在循环单元固有的缺陷。变压器模型的引入，将时序建模领域推向了一个新的方向。在这个项目中，我们实现了一个基于变压器的视频字幕模型，利用3D CNN架构（如c3d）和双流i3d进行视频提取。为了使模型的总体尺寸保持在一定的范围内，我们还采用了一定的降维技术。最后，我们分别在msvd和activitynet数据集上展示了针对单个和密集视频字幕任务的结果。
URL
https://arxiv.org/abs/1906.02792
