Abstract
The ability of robots to interpret human instructions and execute manipulation tasks necessitates the availability of task-relevant tabletop scenes for training. However, traditional methods for creating these scenes rely on time-consuming manual layout design or purely randomized layouts, which are limited in terms of plausibility or alignment with the tasks. In this paper, we formulate a novel task, namely task-oriented tabletop scene generation, which poses significant challenges due to the substantial gap between high-level task instructions and the tabletop scenes. To support research on such a challenging task, we introduce MesaTask-10K, a large-scale dataset comprising approximately 10,700 synthetic tabletop scenes with manually crafted layouts that ensure realistic layouts and intricate inter-object relations. To bridge the gap between tasks and scenes, we propose a Spatial Reasoning Chain that decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction for the final 3D layout. We present MesaTask, an LLM-based framework that utilizes this reasoning chain and is further enhanced with DPO algorithms to generate physically plausible tabletop scenes that align well with given task descriptions. Exhaustive experiments demonstrate the superior performance of MesaTask compared to baselines in generating task-conforming tabletop scenes with realistic layouts. Project page is at this https URL
Abstract (translated)
机器人解释人类指令并执行操作任务的能力需要具有与训练任务相关的桌面场景的可用性。然而，传统的创建这些场景的方法依赖于耗时的手动布局设计或完全随机化的布局设计，这两种方法在合理性和与任务的相关性方面都存在局限。 本文提出了一项新任务——目标导向型桌面场景生成（task-oriented tabletop scene generation），这项任务由于高层次的任务指令与实际的桌面场景之间的巨大差距而面临重大挑战。为了支持对此类具有挑战性的任务的研究，我们引入了MesaTask-10K数据集，这是一个大规模的数据集，包含大约10,700个合成桌面场景，这些场景的手动布局设计确保了现实主义和复杂的对象间关系。 为弥合任务与场景之间的差距，我们提出了一个空间推理链（Spatial Reasoning Chain），该链条将生成过程分解为物体推断、空间相互关联推理以及最终的3D布局场景图构建。我们提出了一种基于大型语言模型（LLM）的框架——MesaTask，该框架利用了上述推理链，并通过加入DPO算法进一步增强了其功能，以生成与给定任务描述相符且物理上合理的桌面场景。 详尽的实验表明，在生成符合任务要求、布局真实感强的桌面场景方面，MesaTask相较于基准方法具有显著的优势。项目页面位于[此处](https://该项目的网址在这里)。
URL
https://arxiv.org/abs/2509.22281
