Abstract
Traditional Automatic License Plate Recognition (ALPR) systems employ multi-stage pipelines consisting of object detection networks followed by separate Optical Character Recognition (OCR) modules, introducing compounding errors, increased latency, and architectural complexity. This research presents Neural Sentinel, a novel unified approach that leverages Vision Language Models (VLMs) to perform license plate recognition, state classification, and vehicle attribute extraction through a single forward pass. Our primary contribution lies in demonstrating that a fine-tuned PaliGemma 3B model, adapted via Low-Rank Adaptation (LoRA), can simultaneously answer multiple visual questions about vehicle images, achieving 92.3% plate recognition accuracy, which is a 14.1% improvement over EasyOCR and 9.9% improvement over PaddleOCR baselines. We introduce a Human-in-the-Loop (HITL) continual learning framework that incorporates user corrections while preventing catastrophic forgetting through experience replay, maintaining a 70:30 ratio of original training data to correction samples. The system achieves a mean inference latency of 152ms with an Expected Calibration Error (ECE) of 0.048, indicating well calibrated confidence estimates. Additionally, the VLM first architecture enables zero-shot generalization to auxiliary tasks including vehicle color detection (89%), seatbelt detection (82%), and occupancy counting (78%) without task specific training. Through extensive experimentation on real world toll plaza imagery, we demonstrate that unified vision language approaches represent a paradigm shift in ALPR systems, offering superior accuracy, reduced architectural complexity, and emergent multi-task capabilities that traditional pipeline approaches cannot achieve.
Abstract (translated)
传统的自动车牌识别（ALPR）系统通常采用多阶段管道，包括对象检测网络和随后的单独光学字符识别（OCR）模块。这种设计不仅引入了累积误差、增加了延迟时间，还导致架构复杂性增加。这项研究提出了一种名为“Neural Sentinel”的新方法，它利用视觉语言模型（VLMs），通过单一前向传递就能完成车牌识别、状态分类和车辆属性提取等工作。 我们的主要贡献在于证明了一个经过低秩适应（LoRA）调整并微调后的PaliGemma 3B模型，在处理多个关于车辆图像的视觉问题时能够同时回答，实现了92.3%的车牌识别准确率。这比EasyOCR高出了14.1%，相比PaddleOCR基线也提高了9.9%。 此外，我们还引入了一个“人在环路”（HITL）持续学习框架，在这个框架中，用户反馈被整合进系统来纠正错误，同时通过经验重播防止灾难性遗忘的发生。该系统维持着训练数据与修正样本之间70:30的比例关系，并且实现了平均推理延迟为152毫秒以及预期校准误差（ECE）仅为0.048，这表明其具有良好的置信度估计。 此外，VLM架构还允许在没有特定任务训练的情况下进行零样本泛化处理辅助任务，例如车辆颜色检测（准确率为89%）、安全带识别（准确率为82%）和占用计数（准确率为78%）。通过大量实验研究现实世界中的收费广场图像，我们展示了统一的视觉语言方法在ALPR系统中代表了一种范式转变，提供了更精确、减少架构复杂性，并且传统的流水线方法无法实现新兴多任务能力。
URL
https://arxiv.org/abs/2602.07051
