Abstract
Generating high-fidelity, physically interactive 3D simulated tabletop scenes is essential for embodied AI--especially for robotic manipulation policy learning and data synthesis. However, current text- or image-driven 3D scene generation methods mainly focus on large-scale scenes, struggling to capture the high-density layouts and complex spatial relations that characterize tabletop scenes. To address these challenges, we propose TabletopGen, a training-free, fully automatic framework that generates diverse, instance-level interactive 3D tabletop scenes. TabletopGen accepts a reference image as input, which can be synthesized by a text-to-image model to enhance scene diversity. We then perform instance segmentation and completion on the reference to obtain per-instance images. Each instance is reconstructed into a 3D model followed by canonical coordinate alignment. The aligned 3D models then undergo pose and scale estimation before being assembled into a collision-free, simulation-ready tabletop scene. A key component of our framework is a novel pose and scale alignment approach that decouples the complex spatial reasoning into two stages: a Differentiable Rotation Optimizer for precise rotation recovery and a Top-view Spatial Alignment mechanism for robust translation and scale estimation, enabling accurate 3D reconstruction from 2D reference. Extensive experiments and user studies show that TabletopGen achieves state-of-the-art performance, markedly surpassing existing methods in visual fidelity, layout accuracy, and physical plausibility, capable of generating realistic tabletop scenes with rich stylistic and spatial diversity. Our code will be publicly available.
Abstract (translated)
生成高保真、物理互动的3D桌面仿真场景对于具身人工智能（embodied AI）至关重要，尤其是在机器人操作策略学习和数据合成方面。然而，当前基于文本或图像驱动的3D场景生成方法主要关注大规模场景的创建，难以捕捉到桌面场景中高密度布局和复杂空间关系的特点。为了应对这些挑战，我们提出了TabletopGen，这是一个无需训练、完全自动化的框架，可以生成多样化的实例级别的互动3D桌面场景。TabletopGen接受一张参考图像作为输入，该图像是通过文本到图像的模型合成而来，以增强场景多样性。然后，在参考图像上进行实例分割和补全操作，获取每个实例对应的图片。接着将每一个实例重构为一个3D模型，并进行标准坐标对齐。对齐后的3D模型会经历姿态与尺度估计过程，最后组装成一个无碰撞的、可以模拟的真实桌面场景。 我们框架的一个关键组成部分是新的姿态和尺度对齐方法，它将复杂的空间推理分解为两个阶段：第一阶段使用可微旋转优化器进行精确旋转恢复；第二阶段则采用俯视图空间对齐机制来稳健地估计位移与缩放比例。这种方法使得从2D参考图像到3D重建的准确度得到提升。 通过广泛的实验和用户研究，TabletopGen在视觉保真度、布局准确性以及物理真实性方面达到了最先进的水平，显著超越了现有方法，并能够生成具有丰富风格和空间多样性的现实桌面场景。我们的代码将公开发布。
URL
https://arxiv.org/abs/2512.01204
