Abstract
The task of Visual Relationship Recognition (VRR) aims to identify relationships between two interacting objects in an image and is particularly challenging due to the widely-spread and highly imbalanced distribution of <subject, relation, object> triplets. To overcome the resultant performance bias in existing VRR approaches, we introduce DiffAugment -- a method which first augments the tail classes in the linguistic space by making use of WordNet and then utilizes the generative prowess of Diffusion Models to expand the visual space for minority classes. We propose a novel hardness-aware component in diffusion which is based upon the hardness of each <S,R,O> triplet and demonstrate the effectiveness of hardness-aware diffusion in generating visual embeddings for the tail classes. We also propose a novel subject and object based seeding strategy for diffusion sampling which improves the discriminative capability of the generated visual embeddings. Extensive experimentation on the GQA-LT dataset shows favorable gains in the subject/object and relation average per-class accuracy using Diffusion augmented samples.
Abstract (translated)
视觉关系识别（VRR）的任务是识别图像中两个相互作用的物体之间的关系，这尤其具有挑战性，因为<主题，关系，物体>三元组分布广泛且不平衡。为了克服现有VRR方法中导致结果性能偏差，我们引入了DiffAugment方法——一种利用WordNet对语言空间尾巴类进行增强的方法，然后利用扩散模型的生成能力来扩展少数类视觉空间。我们提出了一个新颖的复杂性感知组件，基于每个<S，R，O>三元组的不平衡性，并证明了复杂性感知扩散在生成尾类视觉嵌入方面的有效性。我们还提出了一个新的基于扩散的采样策略，可以改善生成的视觉嵌入的判别能力。在GQA-LT数据集上进行广泛的实验，使用扩散增强样本后，主题/物体和关系平均每类准确度都取得了良好的提高。
URL
https://arxiv.org/abs/2401.01387
