Abstract
Instant reconstruction of dynamic 3D humans from uncalibrated sparse-view videos is critical for numerous downstream applications. Existing methods, however, are either limited by the slow reconstruction speeds or incapable of generating novel-time representations. To address these challenges, we propose Forge4D, a feed-forward 4D human reconstruction and interpolation model that efficiently reconstructs temporally aligned representations from uncalibrated sparse-view videos, enabling both novel view and novel time synthesis. Our model simplifies the 4D reconstruction and interpolation problem as a joint task of streaming 3D Gaussian reconstruction and dense motion prediction. For the task of streaming 3D Gaussian reconstruction, we first reconstruct static 3D Gaussians from uncalibrated sparse-view images and then introduce learnable state tokens to enforce temporal consistency in a memory-friendly manner by interactively updating shared information across different timestamps. For novel time synthesis, we design a novel motion prediction module to predict dense motions for each 3D Gaussian between two adjacent frames, coupled with an occlusion-aware Gaussian fusion process to interpolate 3D Gaussians at arbitrary timestamps. To overcome the lack of the ground truth for dense motion supervision, we formulate dense motion prediction as a dense point matching task and introduce a self-supervised retargeting loss to optimize this module. An additional occlusion-aware optical flow loss is introduced to ensure motion consistency with plausible human movement, providing stronger regularization. Extensive experiments demonstrate the effectiveness of our model on both in-domain and out-of-domain datasets. Project page and code at: this https URL.
Abstract (translated)
从未经校准的稀疏视角视频中即时重建动态3D人体对于许多下游应用至关重要。然而，现有方法要么受限于缓慢的重建速度，要么无法生成新时间表示。为了解决这些挑战，我们提出了Forge4D，这是一种前馈式4D人体重建和插值模型，能够高效地从未经校准的稀疏视角视频中重构出时间对齐的表现形式，从而支持新视角合成与新时间合成。 我们的模型将4D重建和插值问题简化为一个连续3D高斯重建和密集运动预测的任务。对于连续3D高斯重建任务，我们首先从未经校准的稀疏视图图像中重构静态3D高斯分布，并引入可学习的状态标记以通过互动更新不同时间戳之间的共享信息来强制执行一种内存友好的时间一致性方法。对于新时间合成，我们设计了一个新颖的运动预测模块，用于在两个相邻帧之间为每个3D高斯预测密集运动，并结合一个遮挡感知的高斯融合过程，在任意时间点插值3D高斯分布。 为了克服缺乏密集运动监督的真实标签的问题，我们将密集运动预测定义为密集点匹配任务，并引入自监督重定向损失来优化该模块。此外，我们还提出了一种遮挡感知的光流损失以确保与合理的身体动作相一致的运动一致性，提供更强的正则化。 大量的实验展示了我们的模型在域内和跨域数据集上的有效性。项目页面和代码可在[此处](https://this-url.com)访问（实际链接需要替换为正确的URL）。
URL
https://arxiv.org/abs/2509.24209
