Abstract
Volumetric video seeks to model dynamic scenes as temporally coherent 4D representations. While recent Gaussian-based approaches achieve impressive rendering fidelity, they primarily emphasize appearance but are largely agnostic to instance-level structure, limiting stable tracking and semantic reasoning in highly dynamic scenarios. In this paper, we present Director, a unified spatio-temporal Gaussian representation that jointly models human performance, high-fidelity rendering, and instance-level semantics. Our key insight is that embedding instance-consistent semantics naturally complements 4D modeling, enabling more accurate scene decomposition while supporting robust dynamic scene understanding. To this end, we leverage temporally aligned instance masks and sentence embeddings derived from Multimodal Large Language Models to supervise the learnable semantic features of each Gaussian via two MLP decoders, enabling language-aligned 4D representations and enforcing identity consistency over time. To enhance temporal stability, we bridge 2D optical flow with 4D Gaussians and finetune their motions, yielding reliable initialization and reducing drift. For the training, we further introduce a geometry-aware SDF constraints, along with regularization terms that enforces surface continuity, enhancing temporal coherence in dynamic foreground modeling. Experiments demonstrate that Director achieves temporally coherent 4D reconstructions while simultaneously enabling instance segmentation and open-vocabulary querying.
Abstract (translated)
体积视频旨在将动态场景建模为时间连贯的4D表示。尽管近期基于高斯的方法实现了令人印象深刻的渲染保真度，但它们主要侧重于外观表现，而在很大程度上忽略了实例级结构，这限制了在高度动态场景中的稳定追踪与语义推理。本文提出了一种名为Director的统一时空高斯表示，可联合建模人体动作、高保真渲染及实例级语义。我们的核心见解是：嵌入实例一致的语义能自然补充4D建模，从而在实现更精确场景分解的同时，支持稳健的动态场景理解。为此，我们利用多模态大语言模型生成的时间对齐实例掩码与句子嵌入，通过两个多层感知机解码器监督每个高斯点的可学习语义特征，实现语言对齐的4D表示并确保时间上身份一致性。为增强时间稳定性，我们将2D光流与4D高斯相连接并微调其运动，从而获得可靠的初始化并减少漂移。训练中，我们进一步引入几何感知的符号距离场约束，以及强制表面连续性的正则项，以提升动态前景建模的时间连贯性。实验表明，Director在实现时间连贯4D重建的同时，还能支持实例分割与开放词汇查询。
URL
https://arxiv.org/abs/2604.01678
