Abstract
Visual Relationship Detection (VRD) has seen significant advancements with Transformer-based architectures recently. However, we identify two key limitations in a conventional label assignment for training Transformer-based VRD models, which is a process of mapping a ground-truth (GT) to a prediction. Under the conventional assignment, an unspecialized query is trained since a query is expected to detect every relation, which makes it difficult for a query to specialize in specific relations. Furthermore, a query is also insufficiently trained since a GT is assigned only to a single prediction, therefore near-correct or even correct predictions are suppressed by being assigned no relation as a GT. To address these issues, we propose Groupwise Query Specialization and Quality-Aware Multi-Assignment (SpeaQ). Groupwise Query Specialization trains a specialized query by dividing queries and relations into disjoint groups and directing a query in a specific query group solely toward relations in the corresponding relation group. Quality-Aware Multi-Assignment further facilitates the training by assigning a GT to multiple predictions that are significantly close to a GT in terms of a subject, an object, and the relation in between. Experimental results and analyses show that SpeaQ effectively trains specialized queries, which better utilize the capacity of a model, resulting in consistent performance gains with zero additional inference cost across multiple VRD models and benchmarks. Code is available at this https URL.
Abstract (translated)
视觉关系检测（VRD）在最近使用Transformer架构取得了显著的进步。然而，我们发现在传统的标签分配过程中存在两个关键限制，这是将真实（GT）映射到预测的过程。在传统分配中，由于预计查询需要检测每个关系，因此查询需要 specialized。此外，由于 GT 只分配给单个预测，因此即使预测接近正确或正确，也会因为分配无关系而被压制。为了解决这些问题，我们提出了组内查询专业化和质量感知多分配（SpeaQ）。组内查询专业化通过将查询和关系划分为独立组，并仅将查询定向向相应关系组中的关系来训练专用查询。质量感知多分配进一步通过将一个 GT 分配给多个预测，这些预测与 GT 在主体、对象和它们之间的关系上非常接近，来促进训练。实验结果和分析表明，SpeaQ有效地训练了专用查询，这更好地利用了模型的能力，从而在多个 VRD 模型和基准上实现了显著的性能提升。代码可在此处访问：https://www.aclweb.org/anthology/N22-11969
URL
https://arxiv.org/abs/2403.17709
