Abstract
Ensuring safety in autonomous driving is a complex challenge requiring handling unknown objects and unforeseen driving scenarios. We develop multiscale video transformers capable of detecting unknown objects using only motion cues. Video semantic and panoptic segmentation often relies on known classes seen during training, overlooking novel categories. Recent visual grounding with large language models is computationally expensive, especially for pixel-level output. We propose an efficient video transformer trained end-to-end for class-agnostic segmentation without optical flow. Our method uses multi-stage multiscale query-memory decoding and a scale-specific random drop-token to ensure efficiency and accuracy, maintaining detailed spatiotemporal features with a shared, learnable memory module. Unlike conventional decoders that compress features, our memory-centric design preserves high-resolution information at multiple scales. We evaluate on DAVIS'16, KITTI, and Cityscapes. Our method consistently outperforms multiscale baselines while being efficient in GPU memory and run-time, demonstrating a promising direction for real-time, robust dense prediction in safety-critical robotics.
Abstract (translated)
确保自动驾驶的安全是一项复杂的挑战，需要处理未知物体和不可预见的驾驶场景。我们开发了一种多尺度视频变压器，仅通过运动线索就能检测到未知对象。视频语义分割和全景分割通常依赖于训练期间见到的已知类别，忽略了新出现的类别。最近使用大型语言模型进行视觉定位在计算上成本高昂，尤其是在生成像素级输出时。为此，我们提出了一种高效端到端训练的视频变压器，用于无类别的分割，并且不需要光学流信息。我们的方法采用了多阶段多尺度查询-内存解码器和特定于比例尺的随机丢弃令牌策略，确保效率与准确性的同时还能保持详细的空间时间特征。通过一个共享、可学习的记忆模块，我们设计保留了不同分辨率下的高分辨率信息，这不同于传统的压缩特征的解码器。 我们在DAVIS'16、KITTI以及Cityscapes数据集上进行了评估，我们的方法在性能上始终优于多尺度基线模型，并且在GPU内存和运行时间方面表现高效。这一研究方向展示了实时、鲁棒密集预测在未来安全关键型机器人技术中的巨大潜力。
URL
https://arxiv.org/abs/2508.14729
