Abstract
Event cameras like the Hybrid Event-based Vision Sensor (HybridEVS) camera capture brightness changes as asynchronous "events" instead of frames, offering advanced application on mobile photography. However, challenges arise from combining a Quad Bayer Color Filter Array (CFA) sensor with event pixels lacking color information, resulting in aliasing and artifacts on the demosaicing process before downstream application. Current methods struggle to address these issues, especially on resource-limited mobile devices. In response, we introduce \textbf{TSANet}, a lightweight \textbf{T}wo-stage network via \textbf{S}tate space augmented cross-\textbf{A}ttention, which can handle event pixels inpainting and demosaicing separately, leveraging the benefits of dividing complex tasks into manageable subtasks. Furthermore, we introduce a lightweight Cross-Swin State Block that uniquely utilizes positional prior for demosaicing and enhances global dependencies through the state space model with linear complexity. In summary, TSANet demonstrates excellent demosaicing performance on both simulated and real data of HybridEVS while maintaining a lightweight model, averaging better results than the previous state-of-the-art method DemosaicFormer across seven diverse datasets in both PSNR and SSIM, while respectively reducing parameter and computation costs by $1.86\times$ and $3.29\times$. Our approach presents new possibilities for efficient image demosaicing on mobile devices. Code is available in the supplementary materials.
Abstract (translated)
事件相机，如混合事件基视觉传感器（HybridEVS）相机，通过捕捉亮度变化作为异步的“事件”，而不是帧的方式工作，为移动摄影提供了高级应用。然而，将四重拜耳色彩滤镜阵列（CFA）传感器与缺乏颜色信息的事件像素结合使用时，会导致去马赛克处理前出现混叠和伪影的问题。现有的方法在资源受限的移动设备上难以解决这些问题。 为此，我们提出了一种轻量级的方法——\textbf{TSANet}，即通过状态空间增强跨注意力机制的两阶段网络（\textbf{T}wo-stage network via \textbf{S}tate space augmented cross-\textbf{A}ttention）。此方法能够分别处理事件像素的填充和去马赛克化，将复杂任务分解为可管理的小任务。此外，我们还引入了一种轻量级的交叉-斯温状态块，该块独特地利用了位置先验进行去马赛克，并通过具有线性复杂度的状态空间模型增强了全局依赖关系。 总之，TSANet 在 HybridEVS 的模拟和真实数据集上展示了卓越的去马赛克性能，并保持了一个轻量级的模型。在七个不同的数据集中，它比之前的最先进方法 DemosaicFormer 平均获得了更高的 PSNR 和 SSIM 结果，同时分别减少了 $1.86\times$ 参数成本和 $3.29\times$ 计算成本。我们的方法为移动设备上的高效图像去马赛克处理开辟了新的可能性。 代码可在补充材料中获取。
URL
https://arxiv.org/abs/2508.06058
