Abstract
Efficient trajectory planning in off-road terrains presents a formidable challenge for autonomous vehicles, often necessitating complex multi-step pipelines. However, traditional approaches exhibit limited adaptability in dynamic environments. To address these limitations, this paper proposes OFF-EMMA, a novel end-to-end multimodal framework designed to overcome the deficiencies of insufficient spatial perception and unstable reasoning in visual-language-action (VLA) models for off-road autonomous driving scenarios. The framework explicitly annotates input images through the design of a visual prompt block and introduces a chain-of-thought with self-consistency (COT-SC) reasoning strategy to enhance the accuracy and robustness of trajectory planning. The visual prompt block utilizes semantic segmentation masks as visual prompts, enhancing the spatial understanding ability of pre-trained visual-language models for complex terrains. The COT- SC strategy effectively mitigates the error impact of outliers on planning performance through a multi-path reasoning mechanism. Experimental results on the RELLIS-3D off-road dataset demonstrate that OFF-EMMA significantly outperforms existing methods, reducing the average L2 error of the Qwen backbone model by 13.3% and decreasing the failure rate from 16.52% to 6.56%.
Abstract (translated)
在越野地形中为自主车辆进行高效的路径规划是一项艰巨的挑战，通常需要复杂的多步骤流程。然而，传统的解决方案在动态环境中表现出较低的适应性。为了应对这些限制，本文提出了一种新型端到端多模态框架OFF-EMMA，旨在克服视觉-语言-行动（VLA）模型在越野自主驾驶场景中空间感知不足和推理不稳定的缺点。该框架通过设计视觉提示模块来显式标注输入图像，并引入了链式思维与自我一致性(COT-SC)推理策略以提高路径规划的准确性和鲁棒性。视觉提示模块利用语义分割掩码作为视觉提示，增强了预训练模型在复杂地形中的空间理解能力。COT-SC策略通过多路径推理机制有效地减轻了异常值对规划性能的影响。 在RELLIS-3D越野数据集上的实验结果表明，OFF-EMMA显著优于现有方法，在降低Qwen骨干网络的平均L2误差方面表现出13.3%的改进，并将失败率从16.52%降至6.56%。
URL
https://arxiv.org/abs/2601.03519
