Abstract
Vision-and-language navigation (VLN) is a crucial but challenging cross-modal navigation task. One powerful technique to enhance the generalization performance in VLN is the use of an independent speaker model to provide pseudo instructions for data augmentation. However, current speaker models based on Long-Short Term Memory (LSTM) lack the ability to attend to features relevant at different locations and time steps. To address this, we propose a novel progress-aware spatio-temporal transformer speaker (PASTS) model that uses the transformer as the core of the network. PASTS uses a spatio-temporal encoder to fuse panoramic representations and encode intermediate connections through steps. Besides, to avoid the misalignment problem that could result in incorrect supervision, a speaker progress monitor (SPM) is proposed to enable the model to estimate the progress of instruction generation and facilitate more fine-grained caption results. Additionally, a multifeature dropout (MFD) strategy is introduced to alleviate overfitting. The proposed PASTS is flexible to be combined with existing VLN models. The experimental results demonstrate that PASTS outperforms all existing speaker models and successfully improves the performance of previous VLN models, achieving state-of-the-art performance on the standard Room-to-Room (R2R) dataset.
Abstract (translated)
视觉和语言导航(VLN)是一个关键但具有挑战性的跨modal导航任务。一种增强VLN泛化性能的强大方法是使用独立的说话者模型提供数据增强的伪指令。然而，当前基于长短期记忆(LSTM)的说话者模型缺乏在不同位置和时间步骤中关注相关特征的能力。为了解决这一问题，我们提出了一种新的进度感知空间transformer说话者(PAsts)模型，该模型使用Transformer作为网络的核心。PAsts使用空间编码器来融合全景表示，并通过步骤编码中间连接。此外，为了避免可能导致错误监督的不匹配问题，我们提出了一个说话者进度监控器(SPM)，使模型能够估计指令生成的进展，并促进更精细的标题结果。此外，我们引入了一种多特征 dropout(MFD)策略，以减轻过拟合。提出的PAsts非常灵活，可以与现有的VLN模型相结合。实验结果显示，PAsts比所有现有的说话者模型表现更好，并成功地改善了先前的VLN模型的性能，在标准房间到房间(R2R)数据集上取得了最先进的表现。
URL
https://arxiv.org/abs/2305.11918
