Abstract
We propose the first framework capable of computing a 4D spatio-temporal grid of video frames and 3D Gaussian particles for each time step using a feed-forward architecture. Our architecture has two main components, a 4D video model and a 4D reconstruction model. In the first part, we analyze current 4D video diffusion architectures that perform spatial and temporal attention either sequentially or in parallel within a two-stream design. We highlight the limitations of existing approaches and introduce a novel fused architecture that performs spatial and temporal attention within a single layer. The key to our method is a sparse attention pattern, where tokens attend to others in the same frame, at the same timestamp, or from the same viewpoint. In the second part, we extend existing 3D reconstruction algorithms by introducing a Gaussian head, a camera token replacement algorithm, and additional dynamic layers and training. Overall, we establish a new state of the art for 4D generation, improving both visual quality and reconstruction capability.
Abstract (translated)
我们提出了一种框架，该框架能够使用前馈架构计算视频帧的四维时空网格和每个时间步的三维高斯粒子。我们的架构主要包含两个部分：一个四维视频模型和一个四维重建模型。 在第一部分中，我们分析了当前的四维视频扩散架构，这些架构在其双流设计中要么顺序地、要么并行地执行空间和时间注意力操作。我们指出现有方法的局限性，并引入了一种新的融合架构，在单层内同时执行空间和时间注意力操作。我们的方法的关键在于一种稀疏注意模式，其中标记物（tokens）可以注意到同一帧内的其他标记物、相同时间戳上的或从相同视角来的标记物。 在第二部分中，我们通过引入高斯头（Gaussian head）、相机标记替换算法以及额外的动态层和训练扩展了现有的三维重建算法。总体而言，我们为四维生成建立了新的业界标准，在视觉质量和重建能力方面都有所提升。
URL
https://arxiv.org/abs/2506.18839
