Abstract
Neural attention (NA) is an effective mechanism for inferring complex structural data dependencies that span long temporal horizons. As a consequence, it has become a key component of sequence-to-sequence models that yield state-of-the-art performance in as hard tasks as abstractive document summarization (ADS), machine translation (MT), and video captioning (VC). NA mechanisms perform inference of context vectors; these constitute weighted sums of deterministic input sequence encodings, adaptively sourced over long temporal horizons. However, recent work in the field of amortized variational inference (AVI) has shown that it is often useful to treat the representations generated by deep networks as latent random variables. This allows for the models to better explore the space of possible representations. Based on this motivation, in this work we introduce a novel regard towards a popular NA mechanism, namely soft-attention (SA). Our approach treats the context vectors generated by SA models as latent variables, the posteriors of which are inferred by employing AVI. Both the means and the covariance matrices of the inferred posteriors are parameterized via deep network mechanisms similar to those employed in the context of standard SA. To illustrate our method, we implement it in the context of popular sequence-to-sequence model variants with SA. We conduct an extensive experimental evaluation using challenging ADS, VC, and MT benchmarks, and show how our approach compares to the baselines.
Abstract (translated)
神经关注（NA）是推断复杂结构数据依赖性的一种有效机制，其跨越时间跨度很长。因此，它已经成为序列到序列模型的关键组成部分，这些模型在抽象文档汇总（ADS），机器翻译（MT）和视频字幕（VC）等艰巨任务中产生最先进的性能。 ）。 NA机制执行上下文向量的推理;这些构成了确定性输入序列编码的加权和，其自适应地源自长时间视界。然而，近期在摊销变分推断（AVI）领域的工作表明，将深网络产生的表示看作潜在随机变量通常是有用的。这允许模型更好地探索可能表示的空间。基于这一动机，在这项工作中，我们介绍了一种新的关于流行的NA机制，即软注意（SA）。我们的方法将由SA模型生成的上下文向量视为潜在变量，后者通过使用AVI来推断。推断后验概率的均值和协方差矩阵都是通过类似于标准SA背景下使用的深层网络机制参数化的。为了说明我们的方法，我们使用SA在流行的序列到序列模型变体的上下文中实现它。我们使用具有挑战性的ADS，VC和MT基准进行广泛的实验评估，并展示我们的方法如何与基线进行比较。
URL
https://arxiv.org/abs/1805.09039
