Abstract
Scene graphs capture complex relationships among objects, serving as strong priors for content generation and manipulation. Yet, reasonably manipulating scene graphs -- whether by adding nodes or modifying edges -- remains a challenging and untouched task. Tasks such as adding a node to the graph or reasoning about a node's relationships with all others are computationally intractable, as even a single edge modification can trigger conflicts due to the intricate interdependencies within the graph. To address these challenges, we introduce SG-Tailor, an autoregressive model that predicts the conflict-free relationship between any two nodes. SG-Tailor not only infers inter-object relationships, including generating commonsense edges for newly added nodes but also resolves conflicts arising from edge modifications to produce coherent, manipulated graphs for downstream tasks. For node addition, the model queries the target node and other nodes from the graph to predict the appropriate relationships. For edge modification, SG-Tailor employs a Cut-And-Stitch strategy to solve the conflicts and globally adjust the graph. Extensive experiments demonstrate that SG-Tailor outperforms competing methods by a large margin and can be seamlessly integrated as a plug-in module for scene generation and robotic manipulation tasks.
Abstract (translated)
场景图捕捉了物体之间的复杂关系，作为内容生成和操作的强大先验知识。然而，合理地操纵场景图（无论是添加节点还是修改边）仍然是一个具有挑战性的未被充分探索的任务。例如，向图中添加一个节点或推理某个节点与其他所有节点的关系在计算上是不可行的，因为即使是单个边的改动也可能由于图内部复杂的相互依赖关系而引发冲突。 为了解决这些挑战，我们提出了SG-Tailor，这是一种自回归模型，它能够预测任意两个节点之间的无冲突关系。SG-Tailor不仅推断了对象间的关系，包括生成新添加节点的常识性边缘连接，而且还解决了由于边修改产生的冲突以产生连贯且可以操作的图供下游任务使用。 对于节点添加，该模型查询目标节点和图中的其他节点来预测适当的关系。 在边修改时，SG-Tailor采用了一种剪切并缝合（Cut-And-Stitch）策略解决冲突，并全局调整整个图形结构。 广泛的实验表明，与竞争方法相比，SG-Tailor的性能显著更优，并且可以无缝集成作为场景生成和机器人操作任务中的插件模块。
URL
https://arxiv.org/abs/2503.18988
