Abstract
Most recent advances in 3D generative modeling rely on diffusion or flow-matching formulations. We instead explore a fully autoregressive alternative and introduce GaussianGPT, a transformer-based model that directly generates 3D Gaussians via next-token prediction, thus facilitating full 3D scene generation. We first compress Gaussian primitives into a discrete latent grid using a sparse 3D convolutional autoencoder with vector quantization. The resulting tokens are serialized and modeled using a causal transformer with 3D rotary positional embedding, enabling sequential generation of spatial structure and appearance. Unlike diffusion-based methods that refine scenes holistically, our formulation constructs scenes step-by-step, naturally supporting completion, outpainting, controllable sampling via temperature, and flexible generation horizons. This formulation leverages the compositional inductive biases and scalability of autoregressive modeling while operating on explicit representations compatible with modern neural rendering pipelines, positioning autoregressive transformers as a complementary paradigm for controllable and context-aware 3D generation.
Abstract (translated)
近期3D生成建模领域的前沿进展大多基于扩散或流匹配（flow-matching）框架。本文则探索了一种完全自回归的替代方案，并提出了GaussianGPT——一种基于Transformer的模型，通过下一个token预测直接生成3D高斯元，从而实现完整的3D场景生成。我们首先使用带有向量量化的稀疏3D卷积自编码器，将高斯基元压缩为离散的潜在网格。所得token被序列化，并采用具备三维旋转位置编码的因果Transformer进行建模，从而支持空间结构与外观的逐序生成。与需要整体优化场景的扩散方法不同，我们的方法以逐步构建的方式生成场景，自然支持场景补全、扩展绘制、通过温度参数实现可控采样，以及灵活的生成范围。该方法在利用自回归建模的组合归纳偏置与可扩展性的同时，操作于与现代神经渲染管线兼容的显式表示，使自回归Transformer成为可控且具备上下文感知能力的3D生成的一种有效补充范式。
URL
https://arxiv.org/abs/2603.26661
