Abstract
Ego-centric driving videos available online provide an abundant source of visual data for autonomous driving, yet their lack of annotations makes it difficult to learn representations that capture both semantic structure and 3D geometry. Recent advances in large feedforward spatial models demonstrate that point maps and ego-motion can be inferred in a single forward pass, suggesting a promising direction for scalable driving perception. We therefore propose a label-free, teacher-guided framework for learning autonomous driving representations directly from unposed videos. Unlike prior self-supervised approaches that focus primarily on frame-to-frame consistency, we posit that safe and reactive driving depends critically on temporal context. To this end, we leverage a feedforward architecture equipped with a lightweight autoregressive module, trained using multi-modal supervisory signals that guide the model to jointly predict current and future point maps, camera poses, semantic segmentation, and motion masks. Multi-modal teachers provide sequence-level pseudo-supervision, enabling LFG to learn a unified pseudo-4D representation from raw YouTube videos without poses, labels, or LiDAR. The resulting encoder not only transfers effectively to downstream autonomous driving planning on the NAVSIM benchmark, surpassing multi-camera and LiDAR baselines with only a single monocular camera, but also yields strong performance when evaluated on a range of semantic, geometric, and qualitative motion prediction tasks. These geometry and motion-aware features position LFG as a compelling video-centric foundation model for autonomous driving.
Abstract (translated)
在线可用的以自我为中心的驾驶视频为自主驾驶提供了丰富的视觉数据来源，但缺乏标注使得学习捕捉语义结构和3D几何的表示变得困难。近期，在大型前馈空间模型方面的进展表明，点地图和自运动可以在一次前向传递中被推断出来，这预示着可扩展驾驶感知的一个有前景的方向。因此，我们提出了一种无标签、由教师指导的框架，直接从未标注视频中学习自主驾驶表示。与之前的自我监督方法主要关注帧到帧的一致性不同，我们认为安全和反应性的驾驶关键在于时间上下文。为此，我们利用了一个配备轻量级自回归模块的前馈架构，并通过多模态监管信号进行训练，这些信号引导模型同时预测当前和未来的点地图、相机姿态、语义分割以及运动掩码。 多模态教师提供序列级别的伪监督，使LFG能够从没有任何姿势、标签或激光雷达数据的原始YouTube视频中学习统一的伪4D表示。生成的编码器不仅在下游自动驾驶规划任务（如NAVSIM基准测试）上有效转移，并且仅使用单目相机就超过了多摄像头和激光雷达基线方法，而且还在一系列语义、几何以及定性和运动预测任务上表现出色。 这种具有几何与运动意识的功能使LFG成为自主驾驶视频中心基础模型的有力候选者。
URL
https://arxiv.org/abs/2602.22091
