Abstract
Most existing multimodal machine translation (MMT) datasets are predominantly composed of static images or short video clips, lacking extensive video data across diverse domains and topics. As a result, they fail to meet the demands of real-world MMT tasks, such as documentary translation. In this study, we developed TopicVD, a topic-based dataset for video-supported multimodal machine translation of documentaries, aiming to advance research in this field. We collected video-subtitle pairs from documentaries and categorized them into eight topics, such as economy and nature, to facilitate research on domain adaptation in video-guided MMT. Additionally, we preserved their contextual information to support research on leveraging the global context of documentaries in video-guided MMT. To better capture the shared semantics between text and video, we propose an MMT model based on a cross-modal bidirectional attention module. Extensive experiments on the TopicVD dataset demonstrate that visual information consistently improves the performance of the NMT model in documentary translation. However, the MMT model's performance significantly declines in out-of-domain scenarios, highlighting the need for effective domain adaptation methods. Additionally, experiments demonstrate that global context can effectively improve translation performance. % Dataset and our implementations are available at this https URL
Abstract (translated)
现有的多模态机器翻译（MMT）数据集主要由静态图像或短视频片段组成，缺乏跨不同领域和主题的大量视频数据。因此，这些数据集无法满足诸如纪录片翻译等实际场景中MMT任务的需求。为此，在这项研究中我们开发了基于话题的视频支持多模态机器翻译的数据集TopicVD，旨在推动该领域的研究进展。我们在文档片中收集并整理了视频-字幕配对，并将它们归类为八个话题（例如经济和自然），以便于研究视频引导下的MMT领域适应性问题。同时我们保留了它们的上下文信息以支持在视频指导下利用纪录片全局背景的研究。 为了更好地捕捉文本与视频之间的共同语义，我们提出了一种基于跨模态双向注意力模块的MMT模型。在TopicVD数据集上的广泛实验表明，在纪录片翻译中视觉信息能够持续改善NMT（神经机器翻译）模型的表现力。然而，当遇到域外场景时，MMT模型表现显著下降，这凸显了有效领域适应方法的需求。此外，实验还显示全局背景能有效地提升翻译性能。 这项研究的数据集及相关实现可以在这个链接下获取：[请在此插入数据集和代码的实际访问网址]
URL
https://arxiv.org/abs/2505.05714
