Abstract
Quanta image sensors, such as SPAD arrays, are an emerging sensor technology, producing 1-bit arrays representing photon detection events over exposures as short as a few nanoseconds. In practice, raw data are post-processed using heavy spatiotemporal binning to create more useful and interpretable images at the cost of degrading spatiotemporal resolution. In this work, we propose bit2bit, a new method for reconstructing high-quality image stacks at the original spatiotemporal resolution from sparse binary quanta image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data by predicting the photon arrival location probability distribution. However, due to the binary nature of the data, we show that the assumption of a Poisson distribution is inadequate. Instead, we model the process with a Bernoulli lattice process from the truncated Poisson. This leads to the proposal of a novel self-supervised solution based on a masked loss function. We evaluate our method using both simulated and real data. On simulated data from a conventional video, we achieve 34.35 mean PSNR with extremely photon-sparse binary input (<0.06 photons per pixel per frame). We also present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions. The scenes cover strong/weak ambient light, strong motion, ultra-fast events, etc., which will be made available to the community, on which we demonstrate the promise of our approach. Both reconstruction quality and throughput substantially surpass the state-of-the-art methods (e.g., Quanta Burst Photography (QBP)). Our approach significantly enhances the visualization and usability of the data, enabling the application of existing analysis techniques.
Abstract (translated)
量子图像传感器，如SPAD阵列，是一项新兴的传感技术，可以生成表示几纳秒曝光时间内光子检测事件的1位数组。在实践中，原始数据会通过大量的时空分箱后处理来创建更有用和可解释的图像，但代价是降低时空分辨率。在这项工作中，我们提出了一种新的方法bit2bit，可以从稀疏二进制量子图像数据中重建具有原始时空分辨率的高质量图像堆栈。受到最近关于泊松降噪工作的启发，我们开发了一种算法，通过预测光子到达位置的概率分布，从稀疏的二进制光子数据创建密集的图像序列。然而，由于数据的二进制性质，我们认为泊松分布假设是不充分的。相反，我们将过程建模为截断泊松中的伯努利格点过程。这导致提出了一个基于掩码损失函数的新自监督解决方案。我们使用模拟和真实数据评估了我们的方法。在传统的视频生成的模拟数据上，即使输入极度光子稀疏（每像素每帧少于0.06个光子），我们也达到了34.35的平均PSNR值。我们还提出了一种新的数据集，包含各种具有挑战性成像条件下拍摄的真实SPAD高速视频。场景涵盖了强/弱环境光、强烈运动、超快速事件等，并将向社区开放使用，展示我们的方法潜力。在重建质量和吞吐量方面，我们的方法显著超越了现有技术（如量子爆发摄影（QBP））。我们的方法显著提高了数据的可视化和可用性，使现有的分析技术得以应用。
URL
https://arxiv.org/abs/2410.23247
