Abstract
This paper proposes a novel scene understanding task called Visual Jenga. Drawing inspiration from the game Jenga, the proposed task involves progressively removing objects from a single image until only the background remains. Just as Jenga players must understand structural dependencies to maintain tower stability, our task reveals the intrinsic relationships between scene elements by systematically exploring which objects can be removed while preserving scene coherence in both physical and geometric sense. As a starting point for tackling the Visual Jenga task, we propose a simple, data-driven, training-free approach that is surprisingly effective on a range of real-world images. The principle behind our approach is to utilize the asymmetry in the pairwise relationships between objects within a scene and employ a large inpainting model to generate a set of counterfactuals to quantify the asymmetry.
Abstract (translated)
这篇论文提出了一项新的场景理解任务，称为视觉积木（Visual Jenga）。这项任务借鉴了游戏Jenga中的原理，即逐步从单张图片中移除对象直到只剩下背景。就像Jenga玩家必须理解结构依赖关系以保持塔的稳定性一样，我们的任务通过系统性地探索在保持场景物理和几何连贯性的前提下可以移除哪些物体，来揭示场景元素之间的内在联系。 为了开始解决视觉积木的任务，我们提出了一种简单、数据驱动且无需训练的方法，在各种现实世界图片上表现出惊人的效果。该方法背后的原理是利用场景中对象之间成对关系的不对称性，并使用一个大型图像修复模型生成一组反事实情况来量化这种不对称性。
URL
https://arxiv.org/abs/2503.21770
