Abstract
Successfully solving long-horizon manipulation tasks remains a fundamental challenge. These tasks involve extended action sequences and complex object interactions, presenting a critical gap between high-level symbolic planning and low-level continuous control. To bridge this gap, two essential capabilities are required: robust long-horizon task planning and effective goal-conditioned manipulation. Existing task planning methods, including traditional and LLM-based approaches, often exhibit limited generalization or sparse semantic reasoning. Meanwhile, image-conditioned control methods struggle to adapt to unseen tasks. To tackle these problems, we propose SAGE, a novel framework for Scene Graph-Aware Guidance and Execution in Long-Horizon Manipulation Tasks. SAGE utilizes semantic scene graphs as a structural representation for scene states. A structural scene graph enables bridging task-level semantic reasoning and pixel-level visuo-motor control. This also facilitates the controllable synthesis of accurate, novel sub-goal images. SAGE consists of two key components: (1) a scene graph-based task planner that uses VLMs and LLMs to parse the environment and reason about physically-grounded scene state transition sequences, and (2) a decoupled structural image editing pipeline that controllably converts each target sub-goal graph into a corresponding image through image inpainting and composition. Extensive experiments have demonstrated that SAGE achieves state-of-the-art performance on distinct long-horizon tasks.
Abstract (translated)
成功解决长期操作任务仍然是一个基本挑战。这些任务涉及扩展的动作序列和复杂的对象交互，这在高层次的符号规划与低层次的连续控制之间形成了关键差距。为了弥合这一差距，需要两种核心能力：稳健的长时序任务规划和有效的目标导向操控。现有的任务规划方法（包括传统方法及基于LLM的方法）通常表现出有限的泛化能力和稀疏的语义推理。同时，以图像为基础的控制方法难以适应未见过的任务。为了应对这些问题，我们提出了SAGE框架——面向长期操作任务中场景图感知引导与执行的新颖架构。SAGE利用语义场景图作为场景状态的结构性表示。这种结构化的场景图有助于在任务级别的语义推理和像素级视觉-运动控制之间架桥，并促进精确且新颖子目标图像的可控合成。SAGE包含两个关键组件：（1）一个基于场景图的任务规划器，它使用视觉语言模型(VLM) 和大型语言模型(LLM) 来解析环境并思考物理基础的场景状态转变序列；（2）一条分离的结构化图像编辑流水线，该流水线通过图像修复与组合将每个目标子图可控地转换为对应图像。大量的实验表明，SAGE在不同的长期操作任务上达到了最先进的性能水平。
URL
https://arxiv.org/abs/2509.21928
