Abstract
Multi-modal relation extraction (MMRE) is a challenging task that aims to identify relations between entities in text leveraging image information. Existing methods are limited by their neglect of the multiple entity pairs in one sentence sharing very similar contextual information (ie, the same text and image), resulting in increased difficulty in the MMRE task. To address this limitation, we propose the Variational Multi-Modal Hypergraph Attention Network (VM-HAN) for multi-modal relation extraction. Specifically, we first construct a multi-modal hypergraph for each sentence with the corresponding image, to establish different high-order intra-/inter-modal correlations for different entity pairs in each sentence. We further design the Variational Hypergraph Attention Networks (V-HAN) to obtain representational diversity among different entity pairs using Gaussian distribution and learn a better hypergraph structure via variational attention. VM-HAN achieves state-of-the-art performance on the multi-modal relation extraction task, outperforming existing methods in terms of accuracy and efficiency.
Abstract (translated)
多模态关系提取（MMRE）是一个具有挑战性的任务，旨在利用图像信息识别文本中实体之间的关系。现有方法的一个局限是它们忽略了共享非常相似上下文信息的多个实体对，导致在MMRE任务中难度增加。为了应对这个局限，我们提出了用于多模态关系提取的变分多模态超图注意力网络（VM-HAN）。具体来说，我们首先为每句话构建一个带有相应图像的多模态超图，以建立不同实体对之间的高层次内部/间相互作用关系。我们进一步设计变分超图注意力网络（V-HAN）来通过高斯分布获得表示多样性，并通过变分注意来学习更好的超图结构。VM-HAN在多模态关系提取任务上实现了最先进的性能，在准确性和效率方面均优于现有方法。
URL
https://arxiv.org/abs/2404.12006
