Abstract
Machine comprehension of visual information from images and videos by neural networks faces two primary challenges. Firstly, there exists a computational and inference gap in connecting vision and language, making it difficult to accurately determine which object a given agent acts on and represent it through language. Secondly, classifiers trained by a single, monolithic neural network often lack stability and generalization. To overcome these challenges, we introduce MoE-VRD, a novel approach to visual relationship detection utilizing a mixture of experts. MoE-VRD identifies language triplets in the form of < subject, predicate, object> tuples to extract relationships from visual processing. Leveraging recent advancements in visual relationship detection, MoE-VRD addresses the requirement for action recognition in establishing relationships between subjects (acting) and objects (being acted upon). In contrast to single monolithic networks, MoE-VRD employs multiple small models as experts, whose outputs are aggregated. Each expert in MoE-VRD specializes in visual relationship learning and object tagging. By utilizing a sparsely-gated mixture of experts, MoE-VRD enables conditional computation and significantly enhances neural network capacity without increasing computational complexity. Our experimental results demonstrate that the conditional computation capabilities and scalability of the mixture-of-experts approach lead to superior performance in visual relationship detection compared to state-of-the-art methods.
Abstract (translated)
机器从图像和视频中理解视觉信息的主要挑战有两个。首先，在连接视觉和语言之间存在计算和推理差距，这使得准确确定给定代理对哪个对象进行操作并将其通过语言表示为困难。其次，由单个单体神经网络训练的分类器通常缺乏稳定性和泛化能力。为了克服这些挑战，我们引入了MoE-VRD，一种利用专家混合的新视觉关系检测方法。MoE-VRD以<主体，谓词，对象>元组的形式识别视觉处理中的语言三元组以提取关系。利用最近在视觉关系检测方面的进展，MoE-VRD在建立主体（进行操作）与物体（被操作）之间的关系方面解决了动作识别的要求。与单体网络相比，MoE-VRD采用多个小模型作为专家，其输出进行聚合。每个专家在MoE-VRD专门研究视觉关系学习和对象标记。通过使用稀疏门控的专家混合，MoE-VRD实现了条件计算，显著增强了神经网络能力，而不会增加计算复杂度。我们的实验结果表明，条件计算能力和可扩展性是专家混合方法的优越性能在视觉关系检测方面比最先进的方法更显著。
URL
https://arxiv.org/abs/2403.03994
