Abstract
This paper aims at learning representations for long sequences of continuous signals. Recently, the BERT model has demonstrated the effectiveness of stacked transformers for representing sequences of discrete signals (i.e. word tokens). Inspired by its success, we adopt the stacked transformer architecture, but generalize its training objective to maximize the mutual information between the masked signals, and the bidirectional context, via contrastive loss. This enables the model to handle continuous signals, such as visual features. We further consider the case when there are multiple sequences that are semantically aligned at the sequence-level but not at the element-level (e.g. video and ASR), where we propose to use a Transformer to estimate the mutual information between the two sequences, which is again maximized via contrastive loss. We demonstrate the effectiveness of the learned representations on modeling long video sequences for action anticipation and video captioning. The results show that our method, referred to by Contrastive Bidirectional Transformer ({\bf CBT}), outperforms various baselines significantly. Furthermore, we improve over the state of the art.
Abstract (translated)
本文旨在学习连续信号长序列的表示方法。最近，伯特模型已经证明了堆叠变压器用于表示离散信号序列（即字标记）的有效性。受其成功的启发，我们采用了堆叠式变压器结构，但通过对比损耗，概括了其训练目标，使屏蔽信号与双向上下文之间的相互信息最大化。这使模型能够处理连续的信号，如视觉特征。我们进一步考虑的情况是，当有多个序列在序列级别语义上对齐，但在元素级别（如视频和ASR）不对齐时，我们建议使用变压器来估计两个序列之间的相互信息，通过对比损失再次最大化。我们证明了所学的表示法对动作预测和视频字幕的长视频序列建模的有效性。结果表明，对比双向变换器（f-cbt）所引用的方法明显优于各种基线。此外，我们还提高了技术水平。
URL
https://arxiv.org/abs/1906.05743
