Paper Reading AI Learner

E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning

2023-11-30 10:33:49
Xiuhong Lin, Changjie Qiu, Zhipeng Cai, Siqi Shen, Yu Zang, Weiquan Liu, Xuesheng Bian, Matthias Müller, Cheng Wang

Abstract

Event cameras have emerged as a promising vision sensor in recent years due to their unparalleled temporal resolution and dynamic range. While registration of 2D RGB images to 3D point clouds is a long-standing problem in computer vision, no prior work studies 2D-3D registration for event cameras. To this end, we propose E2PNet, the first learning-based method for event-to-point cloud registration. The core of E2PNet is a novel feature representation network called Event-Points-to-Tensor (EP2T), which encodes event data into a 2D grid-shaped feature tensor. This grid-shaped feature enables matured RGB-based frameworks to be easily used for event-to-point cloud registration, without changing hyper-parameters and the training procedure. EP2T treats the event input as spatio-temporal point clouds. Unlike standard 3D learning architectures that treat all dimensions of point clouds equally, the novel sampling and information aggregation modules in EP2T are designed to handle the inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC and VECtor datasets demonstrate the superiority of E2PNet over hand-crafted and other learning-based methods. Compared to RGB-based registration, E2PNet is more robust to extreme illumination or fast motion due to the use of event data. Beyond 2D-3D registration, we also show the potential of EP2T for other vision tasks such as flow estimation, event-to-image reconstruction and object recognition. The source code can be found at: this https URL.

Abstract (translated)

事件相机近年来因其无与伦比的时序分辨率和动态范围而成为一项有前景的视觉传感器。虽然将2D RGB图像与3D点云进行注册是一个长期困扰计算机视觉的问题,但之前的工作没有研究过事件相机的2D-3D注册。因此,我们提出了E2PNet,第一个基于学习的事件到点云的注册方法。E2PNet的核心是一个名为Event-Points-to-Tensor(EP2T)的新特征表示网络,它将事件数据编码成一个2D网格形状的特征张量。这个网格形状的特征使得成熟基于RGB的框架可以很容易地用于事件到点云的注册,而无需更改超参数和训练过程。EP2T将事件输入视为空间和时间点云。与标准的3D学习架构不同,EP2T中的新颖采样和信息聚合模块旨在处理空间和时间维度的非均匀性。在MVSEC和VECtor数据集上的实验证明,E2PNet相对于手工craft和其他学习方法具有优越性。与基于RGB的注册相比,E2PNet对极端光照或快速运动更加鲁棒。除了2D-3D注册之外,我们还展示了EP2T在流估计、事件到图像重建和物体识别等其他视觉任务上的潜在能力。源代码可以从该链接找到:https:// this URL。

URL

https://arxiv.org/abs/2311.18433

PDF

https://arxiv.org/pdf/2311.18433.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot