Abstract
Current smart glasses equipped with RGB cameras struggle to perceive the environment in low-light and high-speed motion scenarios due to motion blur and the limited dynamic range of frame cameras. Additionally, capturing dense images with a frame camera requires large bandwidth and power consumption, consequently draining the battery faster. These challenges are especially relevant for developing algorithms that can read text from images. In this work, we propose a novel event-based Optical Character Recognition (OCR) approach for smart glasses. By using the eye gaze of the user, we foveate the event stream to significantly reduce bandwidth by around 98% while exploiting the benefits of event cameras in high-dynamic and fast scenes. Our proposed method performs deep binary reconstruction trained on synthetic data and leverages multimodal LLMs for OCR, outperforming traditional OCR solutions. Our results demonstrate the ability to read text in low light environments where RGB cameras struggle while using up to 2400 times less bandwidth than a wearable RGB camera.
Abstract (translated)
当前配备RGB摄像头的智能眼镜在低光和高速运动场景中感知环境的能力受限，因为这些情况下会出现运动模糊，并且帧相机的动态范围有限。此外，在使用帧相机捕获密集图像时，需要较大的带宽和能耗，从而导致电池消耗更快。这些问题尤其影响了开发能够从图像中读取文本的算法的工作。 在这项工作中，我们提出了一种基于事件的光学字符识别（OCR）方法来解决上述问题，该方法特别适用于智能眼镜。通过利用用户的视线，我们将事件流聚焦到用户注视的地方，这可以将带宽显著减少约98%，同时还能在高动态范围和快速变化场景中利用事件相机的优势。 我们提出的方法采用基于合成数据训练的深度二值重建技术，并利用多模态大语言模型进行OCR识别。这种方法不仅超过了传统的OCR解决方案，在低光环境中读取文本的能力上也表现出色，而且使用的带宽比可穿戴RGB相机少多达2400倍。
URL
https://arxiv.org/abs/2506.06918
