Abstract
Event cameras capture motion dynamics, offering a unique modality with great potential in various computer vision tasks. However, RGB-Event fusion faces three intrinsic misalignments: (i) temporal, (ii) spatial, and (iii) modal misalignment. Existing voxel grid representations neglect temporal correlations between consecutive event windows, and their formulation with simple accumulation of asynchronous and sparse events is incompatible with the synchronous and dense nature of RGB modality. To tackle these challenges, we propose a novel event representation, Motion-enhanced Event Tensor (MET), which transforms sparse event voxels into a dense and temporally coherent form by leveraging dense optical flows and event temporal features. In addition, we introduce a Frequency-aware Bidirectional Flow Aggregation Module (BFAM) and a Temporal Fusion Module (TFM). BFAM leverages the frequency domain and MET to mitigate modal misalignment, while bidirectional flow aggregation and temporal fusion mechanisms resolve spatiotemporal misalignment. Experimental results on two large-scale datasets demonstrate that our framework significantly outperforms state-of-the-art RGB-Event semantic segmentation approaches. Our code is available at: this https URL.
Abstract (translated)
事件相机能够捕捉运动动态，为各种计算机视觉任务提供了一种独特且潜力巨大的模式。然而，RGB-事件融合面临三种内在的对齐问题：（i）时间上的不对齐，（ii）空间上的不对齐以及（iii）模态上的不对齐。现有的体素网格表示方法忽视了连续事件窗口间的时间相关性，并且其简单地累积异步且稀疏事件的做法与同步和密集的RGB模式本质不兼容。为了解决这些挑战，我们提出了一种新颖的事件表示方法——增强运动事件张量（Motion-enhanced Event Tensor，MET），该方法通过利用稠密光流和事件时间特征将稀疏事件体素转换成稠密且时间上连贯的形式。此外，我们还引入了频率感知双向流量聚合模块（Frequency-aware Bidirectional Flow Aggregation Module，BFAM）和时序融合模块（Temporal Fusion Module，TFM）。BFAM利用频域和MET来减轻模态不对齐问题，而双向流聚合与时序融合机制则解决了时空不对齐的问题。在两个大规模数据集上的实验结果表明，我们的框架显著优于当前最先进的RGB-事件语义分割方法。我们的代码可在以下网址获取：this https URL。
URL
https://arxiv.org/abs/2505.01548
