Abstract
Traditional autonomous driving pipelines decouple camera design from downstream perception, relying on fixed optics and handcrafted ISPs that prioritize human viewable imagery rather than machine semantics. This separation discards information during demosaicing, denoising, or quantization, while forcing models to adapt to sensor artifacts. We present a task-driven co-design framework that unifies optics, sensor modeling, and lightweight semantic segmentation networks into a single end-to-end RAW-to-task pipeline. Building on DeepLens[19], our system integrates realistic cellphone-scale lens models, learnable color filter arrays, Poisson-Gaussian noise processes, and quantization, all optimized directly for segmentation objectives. Evaluations on KITTI-360 show consistent mIoU improvements over fixed pipelines, with optics modeling and CFA learning providing the largest gains, especially for thin or low-light-sensitive classes. Importantly, these robustness gains are achieved with a compact ~1M-parameter model running at ~28 FPS, demonstrating edge deployability. Visual and quantitative analyses further highlight how co-designed sensors adapt acquisition to semantic structure, sharpening boundaries and maintaining accuracy under blur, noise, and low bit-depth. Together, these findings establish full-stack co-optimization of optics, sensors, and networks as a principled path toward efficient, reliable, and deployable perception in autonomous systems.
Abstract (translated)
传统的自动驾驶管道将相机设计与下游感知模块分离，依赖于固定的光学系统和优先考虑人类可视性而非机器语义的手动ISP。这种隔离在去马赛克化、降噪或量化过程中会丢失信息，并且迫使模型适应传感器的特性。我们提出了一种任务驱动的协同设计框架，将光学、传感器建模与轻量级语义分割网络统一为一个从原始数据（RAW）到特定任务的整体端到端管道。基于DeepLens[19]系统，我们的方法集成了真实的手机级别镜头模型、可学习的颜色滤波阵列、泊松-高斯噪声过程以及量化技术，并且所有这些都直接针对语义分割的目标进行优化。 在KITTI-360数据集上的评估表明，与固定管道相比，我们的模型在平均交并比（mIoU）方面持续取得改进。光学建模和颜色滤波阵列学习对这种改善贡献最大，尤其是在处理薄或低光敏感类别的场景时效果更加明显。 尤为重要的是，这些性能的提升是通过一个大约100万参数的小型模型实现的，该模型能够以约28帧每秒的速度运行，展示了其在边缘设备部署中的可行性。从视觉和量化分析中进一步可以看出，协同设计传感器是如何根据语义结构调整采集过程的，从而提高边界清晰度，并且即使在模糊、噪声或低位深度的情况下也能保持准确性。 这些研究结果共同证明了光学系统、传感器与网络之间的全栈协同优化是一种有效、可靠并可部署于自动驾驶系统的合理路径。
URL
https://arxiv.org/abs/2512.20815
