Abstract
In domestic environments, robots require a comprehensive understanding of their surroundings to interact effectively and intuitively with untrained humans. In this paper, we propose DVEFormer - an efficient RGB-D Transformer-based approach that predicts dense text-aligned visual embeddings (DVE) via knowledge distillation. Instead of directly performing classical semantic segmentation with fixed predefined classes, our method uses teacher embeddings from Alpha-CLIP to guide our efficient student model DVEFormer in learning fine-grained pixel-wise embeddings. While this approach still enables classical semantic segmentation, e.g., via linear probing, it further enables flexible text-based querying and other applications, such as creating comprehensive 3D maps. Evaluations on common indoor datasets demonstrate that our approach achieves competitive performance while meeting real-time requirements, operating at 26.3 FPS for the full model and 77.0 FPS for a smaller variant on an NVIDIA Jetson AGX Orin. Additionally, we show qualitative results that highlight the effectiveness and possible use cases in real-world applications. Overall, our method serves as a drop-in replacement for traditional segmentation approaches while enabling flexible natural-language querying and seamless integration into 3D mapping pipelines for mobile robotics.
Abstract (translated)
在家庭环境中，机器人需要全面了解周围环境，以便与未经培训的人类用户有效且直观地互动。本文提出了DVEFormer——一种基于RGB-D的Transformer方法，通过知识蒸馏预测密集的文字对齐视觉嵌入（DVE）。不同于直接执行具有固定预定义类别划分的传统语义分割任务，我们的方法使用Alpha-CLIP教师模型生成的嵌入来指导学生模型DVEFormer学习精细粒度的像素级嵌入。尽管这种方法仍然支持传统的语义分割任务（例如通过线性探针进行），它还进一步支持基于文本查询和创建全面的3D地图等其他应用。 在常见的室内数据集上的评估表明，我们的方法实现了竞争性的性能并满足了实时需求，在NVIDIA Jetson AGX Orin平台上运行完整模型时帧率为26.3 FPS，较小变体版本为77.0 FPS。此外，我们展示了定性结果以突出该方法的有效性和可能的应用场景。 总体而言，我们的方法可以作为传统分割方法的直接替代品，并且支持灵活的自然语言查询和无缝集成到移动机器人的3D地图构建管道中。
URL
https://arxiv.org/abs/2601.00359
