Abstract
Event-guided imaging has received significant attention due to its potential to revolutionize instant imaging systems. However, the prior methods primarily focus on enhancing RGB images in a post-processing manner, neglecting the challenges of image signal processor (ISP) dealing with event sensor and the benefits events provide for reforming the ISP process. To achieve this, we conduct the first research on event-guided ISP. First, we present a new event-RAW paired dataset, collected with a novel but still confidential sensor that records pixel-level aligned events and RAW images. This dataset includes 3373 RAW images with 2248 x 3264 resolution and their corresponding events, spanning 24 scenes with 3 exposure modes and 3 lenses. Second, we propose a conventional ISP pipeline to generate good RGB frames as reference. This conventional ISP pipleline performs basic ISP operations, this http URL, white balancing, denoising and color space transforming, with a ColorChecker as reference. Third, we classify the existing learnable ISP methods into 3 classes, and select multiple methods to train and evaluate on our new dataset. Lastly, since there is no prior work for reference, we propose a simple event-guided ISP method and test it on our dataset. We further put forward key technical challenges and future directions in RGB-Event ISP. In summary, to the best of our knowledge, this is the very first research focusing on event-guided ISP, and we hope it will inspire the community. The code and dataset are available at: this https URL.
Abstract (translated)
事件引导成像由于其有可能革新即时成像系统而受到了广泛关注。然而，此前的方法主要集中在通过后处理方式提升RGB图像质量上，忽略了图像信号处理器（ISP）在处理事件传感器时面临的挑战以及事件所提供的改进步ISP过程的好处。为了实现这一目标，我们开展了关于事件引导ISP的首次研究。 首先，我们介绍了一套新的事件-RAW配对数据集，该数据集通过一种新颖但依然保密的传感器收集，记录了像素级别的对齐事件和RAW图像。此数据集中包含3373张2248 x 3264分辨率的RAW图片及其对应的事件数据，涵盖24个场景、三种曝光模式以及三个镜头。 其次，我们提出了一条传统的ISP流水线以生成高质量的RGB帧作为参考标准。这条传统ISP流程执行基本的ISP操作，包括白平衡调整、降噪处理和色彩空间转换，并以ColorChecker为参照基准进行校正。 第三，我们将现有的可学习ISP方法分为三大类，并选择多种方法在我们的新数据集上进行训练与评估。 最后，在没有先前工作可以参考的情况下，我们提出了一种简单的事件引导ISP方法，并在此数据集上进行了测试。此外，我们还指出了RGB-Event ISP领域中关键技术挑战和未来研究方向。 总的来说，据我们所知，这是首次专注于事件引导ISP的研究工作，希望它可以激发更多的社区关注与讨论。代码及数据集可在以下网址获取：[此链接](请将"this https URL"替换为实际的URL)。
URL
https://arxiv.org/abs/2501.19129
