Abstract
We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens. Existing approaches for 3D scene representation and generation commonly use 3D data structures or view-aligned fields. In contrast, we introduce the first method that encodes scene information into a small set of permutation-invariant tokens that is disentangled from the spatial grid. The scene tokens are predicted by a multi-view tokenizer given many context views and rendered into novel views by employing a light-weight rectified flow decoder. We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality. Further, our representation can be rendered from novel trajectories, including ones deviating from the input trajectory, and we show that the decoder gracefully handles uncertainty. Finally, the highly-compressed set of unstructured latent scene tokens enables simple and efficient scene generation in 5 seconds, achieving a much better quality-speed trade-off than previous paradigms.
Abstract (translated)
我们介绍了SceneTok，这是一种新颖的编码器，用于将场景视图集压缩并分解为一组无结构化的、可扩散的标记。现有的三维场景表示和生成方法通常使用3D数据结构或与视角对齐的字段。相比之下，我们首次提出了一种方法，可以将场景信息编码成一小组排列不变性标记，并且这些标记独立于空间网格。通过给定多个上下文视图，SceneTok由一个多视图编码器预测出场景标记，并利用轻量级校正流解码器将其渲染为新视角。 我们展示了这种压缩方法比其他表示方法强1-3个数量级的压缩能力，同时仍能达到业界领先的重建质量。此外，我们的表示可以从新的轨迹进行渲染，包括与输入轨迹偏离的情况，并且显示了解码器能够优雅地处理不确定性。最后，高度压缩的一组无结构潜在场景标记使得在5秒内简单高效地生成场景成为可能，实现了比以往范式更好的质量和速度权衡。
URL
https://arxiv.org/abs/2602.18882
