Optical_Flow
-
eStonefish-scenes: A synthetically generated dataset for underwater event-based optical flow prediction tasks
2025-05-19 16:26:18Jad Mansour, Sebastian Realpe, Hayat Rajani, Michele Grimaldi, Rafael Garcia, Nuno GraciasAbstract
The combined use of event-based vision and Spiking Neural Networks (SNNs) is expected to significantly impact robotics, particularly in tasks like visual odometry and obstacle avoidance. While existing real-world event-based datasets for optical flow prediction, typically captured with Unmanned Aerial Vehicles (UAVs), offer valuable insights, they are limited in diversity, scalability, and are challenging to collect. Moreover, there is a notable lack of labelled datasets for underwater applications, which hinders the integration of event-based vision with Autonomous Underwater Vehicles (AUVs). To address this, synthetic datasets could provide a scalable solution while bridging the gap between simulation and reality. In this work, we introduce eStonefish-scenes, a synthetic event-based optical flow dataset based on the Stonefish simulator. Along with the dataset, we present a data generation pipeline that enables the creation of customizable underwater environments. This pipeline allows for simulating dynamic scenarios, such as biologically inspired schools of fish exhibiting realistic motion patterns, including obstacle avoidance and reactive navigation around corals. Additionally, we introduce a scene generator that can build realistic reef seabeds by randomly distributing coral across the terrain. To streamline data accessibility, we present eWiz, a comprehensive library designed for processing event-based data, offering tools for data loading, augmentation, visualization, encoding, and training data generation, along with loss functions and performance metrics.
Abstract (translated)
基于事件的视觉与脉冲神经网络（SNNs）的结合预计将显著影响机器人技术，特别是在如视觉测距和障碍物规避等任务中。尽管现有的用于光流预测的真实世界事件数据集（通常由无人驾驶飞机系统[UAV]捕获），提供了宝贵的见解，但这些数据集在多样性、可扩展性和收集难度上存在局限性。此外，缺乏水下应用的标记化数据集限制了基于事件的视觉与自主水下航行器(AUVs)的集成。为解决这些问题，合成数据集可以提供一种可扩展的解决方案，并弥合仿真和现实之间的差距。在本工作中，我们引入了eStonefish-scenes，这是一个基于石鱼模拟器的合成事件驱动光流数据集。除了该数据集外，我们还介绍了一个数据生成管道，能够创建定制化的水下环境。这一管道支持动态场景的模拟，例如受生物启发的鱼类群体表现出包括障碍物规避和珊瑚周围反应导航在内的现实运动模式。此外，我们引入了一种场景生成器，它可以构建真实的礁石海底地形，通过在地面上随机分布珊瑚来实现。 为了简化数据访问性，我们推出了eWiz，这是一个全面处理基于事件的数据的库，提供了用于数据加载、增强、可视化、编码和训练数据生成的工具，以及损失函数和性能指标。
URL
https://arxiv.org/abs/2505.13309
PDF
-
FlowCut: Unsupervised Video Instance Segmentation via Temporal Mask Matching
2025-05-19 14:30:33Alp Eren Sari, Paolo FavaroAbstract
We propose FlowCut, a simple and capable method for unsupervised video instance segmentation consisting of a three-stage framework to construct a high-quality video dataset with pseudo labels. To our knowledge, our work is the first attempt to curate a video dataset with pseudo-labels for unsupervised video instance segmentation. In the first stage, we generate pseudo-instance masks by exploiting the affinities of features from both images and optical flows. In the second stage, we construct short video segments containing high-quality, consistent pseudo-instance masks by temporally matching them across the frames. In the third stage, we use the YouTubeVIS-2021 video dataset to extract our training instance segmentation set, and then train a video segmentation model. FlowCut achieves state-of-the-art performance on the YouTubeVIS-2019, YouTubeVIS-2021, DAVIS-2017, and DAVIS-2017 Motion benchmarks.
Abstract (translated)
我们提出了FlowCut，这是一种简单而强大的无监督视频实例分割方法，通过一个三阶段框架构建带有伪标签的高质量视频数据集。据我们所知，我们的工作是首次尝试为无监督视频实例分割整理带伪标签的数据集。 在第一阶段，我们利用来自图像和光流的特征之间的亲和性生成伪实例掩码。 第二阶段中，我们通过跨帧匹配构造包含高质、一致伪实例掩码的短视频片段。 第三阶段中，我们将使用YouTubeVIS-2021视频数据集提取训练实例分割集合，并训练一个视频分割模型。 FlowCut在YouTubeVIS-2019、YouTubeVIS-2021、DAVIS-2017和DAVIS-2017 Motion基准测试上达到了当前最佳的性能。
URL
https://arxiv.org/abs/2505.13174
PDF
-
Just Dance with $pi$! A Poly-modal Inductor for Weakly-supervised Video Anomaly Detection
2025-05-19 13:51:57Snehashis Majhi, Giacomo D'Amicantonio, Antitza Dantcheva, Quan Kong, Lorenzo Garattoni, Gianpiero Francesca, Egor Bondarev, Francois BremondAbstract
Weakly-supervised methods for video anomaly detection (VAD) are conventionally based merely on RGB spatio-temporal features, which continues to limit their reliability in real-world scenarios. This is due to the fact that RGB-features are not sufficiently distinctive in setting apart categories such as shoplifting from visually similar events. Therefore, towards robust complex real-world VAD, it is essential to augment RGB spatio-temporal features by additional modalities. Motivated by this, we introduce the Poly-modal Induced framework for VAD: "PI-VAD", a novel approach that augments RGB representations by five additional modalities. Specifically, the modalities include sensitivity to fine-grained motion (Pose), three dimensional scene and entity representation (Depth), surrounding objects (Panoptic masks), global motion (optical flow), as well as language cues (VLM). Each modality represents an axis of a polygon, streamlined to add salient cues to RGB. PI-VAD includes two plug-in modules, namely Pseudo-modality Generation module and Cross Modal Induction module, which generate modality-specific prototypical representation and, thereby, induce multi-modal information into RGB cues. These modules operate by performing anomaly-aware auxiliary tasks and necessitate five modality backbones -- only during training. Notably, PI-VAD achieves state-of-the-art accuracy on three prominent VAD datasets encompassing real-world scenarios, without requiring the computational overhead of five modality backbones at inference.
Abstract (translated)
弱监督视频异常检测（VAD）方法通常仅基于RGB时空特征，这在现实场景中继续限制了其可靠性。这是因为RGB特征不足以区分如偷窃等事件与视觉相似的其他事件。因此，在针对复杂真实世界的VAD问题时，增强RGB时空特征以包含额外模态是必要的。受到这一观点的启发，我们引入了一种新的方法——多模态诱导框架“PI-VAD”，它通过五种附加模态来增强RGB表示。具体来说，这些模态包括对细粒度运动（姿态）敏感性、三维场景和实体表示（深度）、周围物体（全景掩码）、全局运动（光流），以及语言线索（视觉语言模型）。每种模态在PI-VAD框架中代表一个多边形的一个轴，通过这种方式向RGB添加显著的提示信息。 PI-VAD包括两个插件模块：伪模态生成模块和跨模态诱导模块。这些模块分别生成特定于每个模态的原型表示，并将多模态信息引入到RGB线索中。这些模块操作时执行异常感知辅助任务，仅在训练期间需要五个模态骨干网络的支持。 值得注意的是，PI-VAD在涵盖真实世界场景的三个重要的VAD数据集上达到了最先进的准确率水平，且无需在推理阶段使用五种模态骨干网络来避免额外计算开销。
URL
https://arxiv.org/abs/2505.13123
PDF
-
MonoMobility: Zero-Shot 3D Mobility Analysis from Monocular Videos
2025-05-17 06:21:05Hongyi Zhou, Xiaogang Wang, Yulan Guo, Kai XuAbstract
Accurately analyzing the motion parts and their motion attributes in dynamic environments is crucial for advancing key areas such as embodied intelligence. Addressing the limitations of existing methods that rely on dense multi-view images or detailed part-level annotations, we propose an innovative framework that can analyze 3D mobility from monocular videos in a zero-shot manner. This framework can precisely parse motion parts and motion attributes only using a monocular video, completely eliminating the need for annotated training data. Specifically, our method first constructs the scene geometry and roughly analyzes the motion parts and their initial motion attributes combining depth estimation, optical flow analysis and point cloud registration method, then employs 2D Gaussian splatting for scene representation. Building on this, we introduce an end-to-end dynamic scene optimization algorithm specifically designed for articulated objects, refining the initial analysis results to ensure the system can handle 'rotation', 'translation', and even complex movements ('rotation+translation'), demonstrating high flexibility and versatility. To validate the robustness and wide applicability of our method, we created a comprehensive dataset comprising both simulated and real-world scenarios. Experimental results show that our framework can effectively analyze articulated object motions in an annotation-free manner, showcasing its significant potential in future embodied intelligence applications.
Abstract (translated)
在动态环境中精确分析运动部件及其运动属性对于推进具身智能等关键领域至关重要。为了解决现有方法依赖密集多视角图像或详细部件级标注的限制，我们提出了一种创新框架，可以从单目视频中零样本地分析3D运动性。该框架仅使用单目视频就能精准解析运动部件和运动属性，并且完全不需要注释训练数据。 具体来说，我们的方法首先通过深度估计、光流分析和点云配准技术构建场景几何结构并粗略地分析运动部件及其初始运动属性，然后采用二维高斯喷涂法进行场景表示。在此基础上，我们引入了一种专为刚体对象设计的端到端动态场景优化算法，对初步分析结果进行细化处理，以确保系统能够应对“旋转”、“平移”，甚至复杂的组合动作（如“旋转+平移”），展示了极高的灵活性和实用性。 为了验证方法的鲁棒性和广泛适用性，我们创建了一个包含仿真和现实世界场景的综合数据集。实验结果显示，我们的框架能够在无需标注的情况下有效分析刚体对象的动作，展现出在未来具身智能应用中的巨大潜力。
URL
https://arxiv.org/abs/2505.11868
PDF
-
Planar Velocity Estimation for Fast-Moving Mobile Robots Using Event-Based Optical Flow
2025-05-16 11:00:33Liam Boyle, Jonas K\"uhne, Nicolas Baumann, Niklas Bastuck, Michele MagnoAbstract
Accurate velocity estimation is critical in mobile robotics, particularly for driver assistance systems and autonomous driving. Wheel odometry fused with Inertial Measurement Unit (IMU) data is a widely used method for velocity estimation; however, it typically requires strong assumptions, such as non-slip steering, or complex vehicle dynamics models that do not hold under varying environmental conditions like slippery surfaces. We introduce an approach to velocity estimation that is decoupled from wheel-to-surface traction assumptions by leveraging planar kinematics in combination with optical flow from event cameras pointed perpendicularly at the ground. The asynchronous micro-second latency and high dynamic range of event cameras make them highly robust to motion blur, a common challenge in vision-based perception techniques for autonomous driving. The proposed method is evaluated through in-field experiments on a 1:10 scale autonomous racing platform and compared to precise motion capture data, demonstrating not only performance on par with the state-of-the-art Event-VIO method but also a 38.3 % improvement in lateral error. Qualitative experiments at highway speeds of up to 32 m/s further confirm the effectiveness of our approach, indicating significant potential for real-world deployment.
Abstract (translated)
准确的速度估计对于移动机器人，尤其是驾驶员辅助系统和自动驾驶至关重要。轮式里程计与惯性测量单元（IMU）数据融合是一种广泛使用的速度估算方法；然而，这种方法通常需要较强的假设条件，例如非打滑转向，或者在不同环境条件下不成立的复杂车辆动力学模型。本文提出了一种独立于轮胎与地面摩擦力假设的速度估算方法，该方法结合了平面运动学和由垂直对准地面的事件相机提供的光学流数据。由于事件相机具有亚毫秒级异步延迟及高动态范围特性，因此它们在自动驾驶视觉感知技术中面对图像模糊时表现出高度鲁棒性。 所提出的这种方法通过在1:10比例自主赛车平台上进行实地实验，并与精确的动作捕捉数据对比进行了评估，不仅展示了与当前最先进的事件-VIO方法相当的性能，而且在横向误差方面提高了38.3%。此外，在高达32米/秒（约115公里/小时）的速度下进行的定性实验进一步证实了该方法的有效性，表明其具有实际应用的巨大潜力。
URL
https://arxiv.org/abs/2505.11116
PDF
-
TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation
2025-05-15 20:35:06Manthan Patel, Fan Yang, Yuheng Qiu, Cesar Cadena, Sebastian Scherer, Marco Hutter, Wenshan WangAbstract
We present TartanGround, a large-scale, multi-modal dataset to advance the perception and autonomy of ground robots operating in diverse environments. This dataset, collected in various photorealistic simulation environments includes multiple RGB stereo cameras for 360-degree coverage, along with depth, optical flow, stereo disparity, LiDAR point clouds, ground truth poses, semantic segmented images, and occupancy maps with semantic labels. Data is collected using an integrated automatic pipeline, which generates trajectories mimicking the motion patterns of various ground robot platforms, including wheeled and legged robots. We collect 910 trajectories across 70 environments, resulting in 1.5 million samples. Evaluations on occupancy prediction and SLAM tasks reveal that state-of-the-art methods trained on existing datasets struggle to generalize across diverse scenes. TartanGround can serve as a testbed for training and evaluation of a broad range of learning-based tasks, including occupancy prediction, SLAM, neural scene representation, perception-based navigation, and more, enabling advancements in robotic perception and autonomy towards achieving robust models generalizable to more diverse scenarios. The dataset and codebase for data collection will be made publicly available upon acceptance. Webpage: this https URL
Abstract (translated)
我们介绍了TartanGround，这是一个大规模、多模态的数据集，旨在推动在各种环境中运行的地面机器人感知和自主性的进步。该数据集是在多个逼真的模拟环境中收集而成，包括了多台RGB立体相机以实现360度覆盖范围，以及深度信息、光流、立体视差、激光雷达点云、真实姿态、语义分割图像和带有语义标签的占用地图等信息。数据采集采用了一条集成自动管道，该管道生成模仿各种地面机器人平台（包括轮式和腿足式机器人）运动模式的轨迹。我们在70个环境中收集了910条轨迹，总计产生了150万个样本。在占据预测和SLAM任务上的评估结果表明，基于现有数据集训练出的方法难以泛化到不同的场景中。TartanGround可以作为广泛学习任务（包括占据预测、SLAM、神经场景表示、感知导航等）的训练与评测平台，有助于推动机器人感知和自主性的发展，使模型能够适用于更加多样的场景。该数据集及其收集代码将在接受后公开发布。官网：[此URL](此URL)
URL
https://arxiv.org/abs/2505.10696
PDF
-
FreeDriveRF: Monocular RGB Dynamic NeRF without Poses for Autonomous Driving via Point-Level Dynamic-Static Decoupling
2025-05-14 14:02:49Yue Wen, Liang Song, Yijia Liu, Siting Zhu, Yanzi Miao, Lijun Han, Hesheng WangAbstract
Dynamic scene reconstruction for autonomous driving enables vehicles to perceive and interpret complex scene changes more precisely. Dynamic Neural Radiance Fields (NeRFs) have recently shown promising capability in scene modeling. However, many existing methods rely heavily on accurate poses inputs and multi-sensor data, leading to increased system complexity. To address this, we propose FreeDriveRF, which reconstructs dynamic driving scenes using only sequential RGB images without requiring poses inputs. We innovatively decouple dynamic and static parts at the early sampling level using semantic supervision, mitigating image blurring and artifacts. To overcome the challenges posed by object motion and occlusion in monocular camera, we introduce a warped ray-guided dynamic object rendering consistency loss, utilizing optical flow to better constrain the dynamic modeling process. Additionally, we incorporate estimated dynamic flow to constrain the pose optimization process, improving the stability and accuracy of unbounded scene reconstruction. Extensive experiments conducted on the KITTI and Waymo datasets demonstrate the superior performance of our method in dynamic scene modeling for autonomous driving.
Abstract (translated)
用于自动驾驶的动态场景重建使车辆能够更精确地感知和解释复杂的场景变化。最近，动态神经辐射场（NeRF）在场景建模方面展现出了很有前景的能力。然而，许多现有的方法严重依赖于准确的姿态输入和多传感器数据，从而增加了系统的复杂性。为了解决这个问题，我们提出了FreeDriveRF，它仅使用连续的RGB图像来重建动态驾驶场景，并且不需要姿态输入。我们在早期采样阶段创新地将动态部分与静态部分解耦，利用语义监督来减少图像模糊和伪影。为了克服单目相机在面对物体运动和遮挡时所带来的挑战，我们引入了一种扭曲光线引导的动态对象渲染一致性损失，利用光流更好地约束动态建模过程。此外，我们还采用了估计的动态流来限制姿态优化过程，从而提高了无界场景重建的稳定性和准确性。我们在KITTI和Waymo数据集上进行了一系列实验，结果表明我们的方法在自动驾驶的动态场景建模方面具有优越的表现。
URL
https://arxiv.org/abs/2505.09406
PDF
-
RobustSpring: Benchmarking Robustness to Image Corruptions for Optical Flow, Scene Flow and Stereo
2025-05-14 13:21:34Jenny Schmalfuss, Victor Oei, Lukas Mehl, Madlen Bartsch, Shashank Agnihotri, Margret Keuper, Andr\'es BruhnAbstract
Standard benchmarks for optical flow, scene flow, and stereo vision algorithms generally focus on model accuracy rather than robustness to image corruptions like noise or rain. Hence, the resilience of models to such real-world perturbations is largely unquantified. To address this, we present RobustSpring, a comprehensive dataset and benchmark for evaluating robustness to image corruptions for optical flow, scene flow, and stereo models. RobustSpring applies 20 different image corruptions, including noise, blur, color changes, quality degradations, and weather distortions, in a time-, stereo-, and depth-consistent manner to the high-resolution Spring dataset, creating a suite of 20,000 corrupted images that reflect challenging conditions. RobustSpring enables comparisons of model robustness via a new corruption robustness metric. Integration with the Spring benchmark enables public two-axis evaluations of both accuracy and robustness. We benchmark a curated selection of initial models, observing that accurate models are not necessarily robust and that robustness varies widely by corruption type. RobustSpring is a new computer vision benchmark that treats robustness as a first-class citizen to foster models that combine accuracy with resilience. It will be available at this https URL.
Abstract (translated)
光学流、场景流和立体视觉算法的标准基准通常侧重于模型的准确性，而不是其对诸如噪声或雨等图像退化的鲁棒性。因此，这些模型应对现实世界扰动的能力没有得到充分量化。为了解决这个问题，我们提出了RobustSpring，这是一个全面的数据集和基准测试平台，用于评估光学流、场景流和立体视觉模型在图像退化情况下的鲁棒性。RobustSpring将20种不同的图像退化（包括噪声、模糊、颜色变化、质量下降以及天气扭曲）以时间上、立体视图上及深度上的一致方式应用于高分辨率的Spring数据集，从而生成一套反映挑战条件的20,000张受损图片。RobustSpring通过一个新的鲁棒性度量指标来比较不同模型的鲁棒性，并且与Spring基准测试平台集成可以进行公开的二维评估（准确性及鲁棒性）。我们对初始的一系列精心挑选的模型进行了基准测试，发现准确性的模型不一定具有较强的鲁棒性，并且不同的退化类型表现出不同程度的鲁棒性。RobustSpring是一个新的计算机视觉基准，将鲁棒性视为第一公民，旨在促进既能保持精度又具有抗干扰能力的模型的发展。更多信息将在 [此链接](https://www.example.com) 上发布。
URL
https://arxiv.org/abs/2505.09368
PDF
-
Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection
2025-05-13 13:35:41Ayush K. Rai, Kyle Min, Tarun Krishna, Feiyan Hu, Alan F. Smeaton, Noel E. O'ConnorAbstract
Masked video modeling~(MVM) has emerged as a highly effective pre-training strategy for visual foundation models, whereby the model reconstructs masked spatiotemporal tokens using information from visible tokens. However, a key challenge in such approaches lies in selecting an appropriate masking strategy. Previous studies have explored predefined masking techniques, including random and tube-based masking, as well as approaches that leverage key motion priors, optical flow and semantic cues from externally pre-trained models. In this work, we introduce a novel and generalizable Trajectory-Aware Adaptive Token Sampler (TATS), which models the motion dynamics of tokens and can be seamlessly integrated into the masked autoencoder (MAE) framework to select motion-centric tokens in videos. Additionally, we propose a unified training strategy that enables joint optimization of both MAE and TATS from scratch using Proximal Policy Optimization (PPO). We show that our model allows for aggressive masking without compromising performance on the downstream task of action recognition while also ensuring that the pre-training remains memory efficient. Extensive experiments of the proposed approach across four benchmarks, including Something-Something v2, Kinetics-400, UCF101, and HMDB51, demonstrate the effectiveness, transferability, generalization, and efficiency of our work compared to other state-of-the-art methods.
Abstract (translated)
遮罩视频建模（Masked Video Modeling，MVM）已成为视觉基础模型预训练的高效策略之一。在这种方法中，通过可见标记的信息来重构被掩盖的空间-时间标记。然而，在这种技术中的关键挑战是如何选择合适的掩码策略。先前的研究探索了多种预先定义的掩码技术，包括随机和基于管状的方法，并且也考虑了从外部预训练模型中提取的关键运动先验、光流以及语义线索。 在此项工作中，我们引入了一种新颖且具普适性的轨迹感知自适应标记采样器（Trajectory-Aware Adaptive Token Sampler, TATS），该方法能够建模标记的动态性，并可无缝集成到遮罩自动编码器（MAE）框架中以选择视频中的运动核心标记。此外，我们提出了一种统一的训练策略，利用近端策略优化（Proximal Policy Optimization，PPO）从零开始联合优化 MAE 和 TATS。我们的模型允许在不损害动作识别下游任务性能的情况下进行激进的掩码操作，并且还能确保预训练过程保持内存效率。 我们在四个基准数据集上的广泛实验——包括Something-Something v2、Kinetics-400、UCF101和HMDB51，表明我们提出的方法在有效性、可迁移性、泛化能力和效率方面优于其他最先进的方法。
URL
https://arxiv.org/abs/2505.08561
PDF
-
TT-DF: A Large-Scale Diffusion-Based Dataset and Benchmark for Human Body Forgery Detection
2025-05-13 11:01:25Wenkui Yang, Zhida Zhang, Xiaoqiang Zhou, Junxian Duan, Jie CaoAbstract
The emergence and popularity of facial deepfake methods spur the vigorous development of deepfake datasets and facial forgery detection, which to some extent alleviates the security concerns about facial-related artificial intelligence technologies. However, when it comes to human body forgery, there has been a persistent lack of datasets and detection methods, due to the later inception and complexity of human body generation methods. To mitigate this issue, we introduce TikTok-DeepFake (TT-DF), a novel large-scale diffusion-based dataset containing 6,120 forged videos with 1,378,857 synthetic frames, specifically tailored for body forgery detection. TT-DF offers a wide variety of forgery methods, involving multiple advanced human image animation models utilized for manipulation, two generative configurations based on the disentanglement of identity and pose information, as well as different compressed versions. The aim is to simulate any potential unseen forged data in the wild as comprehensively as possible, and we also furnish a benchmark on TT-DF. Additionally, we propose an adapted body forgery detection model, Temporal Optical Flow Network (TOF-Net), which exploits the spatiotemporal inconsistencies and optical flow distribution differences between natural data and forged data. Our experiments demonstrate that TOF-Net achieves favorable performance on TT-DF, outperforming current state-of-the-art extendable facial forgery detection models. For our TT-DF dataset, please refer to this https URL.
Abstract (translated)
面部深度伪造技术的出现和流行推动了深度伪造数据集的发展以及面部伪造检测方法的进步，这在一定程度上缓解了与面部相关的人工智能技术的安全担忧。然而，在涉及人体伪造的情况下，由于生成方法起步较晚且复杂性高，一直缺乏相应的数据集和检测方法。为解决这一问题，我们引入了TikTok-DeepFake（TT-DF），这是一个基于扩散的新型大规模数据集，包含6,120个伪造视频和1,378,857个合成帧，专门用于人体伪造检测。TT-DF提供了多种伪造方法，包括利用多个先进的图像动画模型进行操作、基于身份与姿态信息解耦的不同生成配置以及不同压缩版本。我们的目标是尽可能全面地模拟野外可能出现的任何未见过的伪造数据，并且我们在TT-DF上提供了一个基准测试。此外，我们还提出了一种适应性的人体伪造检测模型——时序光流网络（TOF-Net），该模型利用自然数据和伪造数据之间的时空不一致性和光流分布差异来进行检测。实验结果表明，TOF-Net在TT-DF上的表现优于当前最先进的可扩展面部伪造检测模型。 有关我们TT-DF数据集的更多信息，请参阅此[链接](https://this-is-the-url-for-tt-df-data-set)（请将方括号中的内容替换为实际URL）。
URL
https://arxiv.org/abs/2505.08437
PDF
-
EventDiff: A Unified and Efficient Diffusion Model Framework for Event-based Video Frame Interpolation
2025-05-13 05:25:58Hanle Zheng, Xujie Han, Zegang Peng, Shangbin Zhang, Guangxun Du, Zhuo Zou, Xilin Wang, Jibin Wu, Hao Guo, Lei DengAbstract
Video Frame Interpolation (VFI) is a fundamental yet challenging task in computer vision, particularly under conditions involving large motion, occlusion, and lighting variation. Recent advancements in event cameras have opened up new opportunities for addressing these challenges. While existing event-based VFI methods have succeeded in recovering large and complex motions by leveraging handcrafted intermediate representations such as optical flow, these designs often compromise high-fidelity image reconstruction under subtle motion scenarios due to their reliance on explicit motion modeling. Meanwhile, diffusion models provide a promising alternative for VFI by reconstructing frames through a denoising process, eliminating the need for explicit motion estimation or warping operations. In this work, we propose EventDiff, a unified and efficient event-based diffusion model framework for VFI. EventDiff features a novel Event-Frame Hybrid AutoEncoder (HAE) equipped with a lightweight Spatial-Temporal Cross Attention (STCA) module that effectively fuses dynamic event streams with static frames. Unlike previous event-based VFI methods, EventDiff performs interpolation directly in the latent space via a denoising diffusion process, making it more robust across diverse and challenging VFI scenarios. Through a two-stage training strategy that first pretrains the HAE and then jointly optimizes it with the diffusion model, our method achieves state-of-the-art performance across multiple synthetic and real-world event VFI datasets. The proposed method outperforms existing state-of-the-art event-based VFI methods by up to 1.98dB in PSNR on Vimeo90K-Triplet and shows superior performance in SNU-FILM tasks with multiple difficulty levels. Compared to the emerging diffusion-based VFI approach, our method achieves up to 5.72dB PSNR gain on Vimeo90K-Triplet and 4.24X faster inference.
Abstract (translated)
视频帧插值（VFI）是计算机视觉中一个基础但具有挑战性的任务，尤其是在处理大运动、遮挡和光照变化的情况下。最近在事件相机领域的进展为解决这些挑战提供了新的机会。现有的基于事件的VFI方法通过利用诸如光流等手工设计的中间表示来成功恢复大量复杂动作，但在细微运动场景下的高保真图像重建方面往往因依赖显式的运动建模而有所妥协。同时，扩散模型通过去噪过程重构帧提供了一种有前景的替代方案，消除了对显式运动估计或扭曲操作的需求。在这项工作中，我们提出了EventDiff，这是一种统一且高效的基于事件的扩散模型框架用于VFI。EventDiff具备一种新颖的事件-帧混合自编码器（HAE），配有一个轻量级的空间-时间交叉注意（STCA）模块，该模块能够有效地融合动态事件流与静态图像帧。不同于以往的基于事件的VFI方法，EventDiff通过在潜在空间中进行去噪扩散过程直接执行插值操作，在各种复杂和挑战性的VFI场景下表现出更强的鲁棒性。 我们采用两阶段训练策略：首先预训练HAE，然后将其与扩散模型共同优化。本方法在多个合成及真实世界事件VFI数据集上实现了最先进的性能。相较于现有的基于事件的VFI方法，我们的方法在Vimeo90K-Triplet上的PSNR值提高了高达1.98dB，并且在SNU-FILM任务中不同难度级别的表现更为优越。与新兴的基于扩散的VFI方法相比，在Vimeo90K-Triplet上本方法实现了高达5.72dB的PSNR增益，同时推理速度提升了4.24倍。
URL
https://arxiv.org/abs/2505.08235
PDF
-
Monocular Depth Guided Occlusion-Aware Disparity Refinement via Semi-supervised Learning in Laparoscopic Images
2025-05-13 02:29:56Ziteng Liu, Dongdong He, Chenghong Zhang, Wenpeng Gao, Yili FuAbstract
Occlusion and the scarcity of labeled surgical data are significant challenges in disparity estimation for stereo laparoscopic images. To address these issues, this study proposes a Depth Guided Occlusion-Aware Disparity Refinement Network (DGORNet), which refines disparity maps by leveraging monocular depth information unaffected by occlusion. A Position Embedding (PE) module is introduced to provide explicit spatial context, enhancing the network's ability to localize and refine features. Furthermore, we introduce an Optical Flow Difference Loss (OFDLoss) for unlabeled data, leveraging temporal continuity across video frames to improve robustness in dynamic surgical scenes. Experiments on the SCARED dataset demonstrate that DGORNet outperforms state-of-the-art methods in terms of End-Point Error (EPE) and Root Mean Squared Error (RMSE), particularly in occlusion and texture-less regions. Ablation studies confirm the contributions of the Position Embedding and Optical Flow Difference Loss, highlighting their roles in improving spatial and temporal consistency. These results underscore DGORNet's effectiveness in enhancing disparity estimation for laparoscopic surgery, offering a practical solution to challenges in disparity estimation and data limitations.
Abstract (translated)
遮挡和标记的手术数据稀缺是立体腹腔镜图像视差估计中的主要挑战。为解决这些问题，本研究提出了一种深度引导的遮挡感知视差精炼网络（DGORNet），该网络通过利用不受遮挡影响的单目深度信息来精炼视差图。引入了位置嵌入（PE）模块以提供显式的空间上下文，从而增强网络定位和精炼特征的能力。此外，我们提出了一种用于未标记数据的光流差异损失（OFDLoss），通过利用视频帧之间的时序连续性，在动态手术场景中提高鲁棒性。 在SCARED数据集上的实验表明，DGORNet在视差估计的端点误差（EPE）和均方根误差（RMSE）方面优于现有方法，特别是在遮挡区域和纹理缺乏的区域。消融研究证实了位置嵌入和光流差异损失对提高空间一致性和时间一致性的作用。 这些结果强调了DGORNet在提升腹腔镜手术视差估计效果方面的有效性，并为解决视差估计中的挑战及数据限制问题提供了一种实用解决方案。
URL
https://arxiv.org/abs/2505.08178
PDF
-
Asynchronous Multi-Object Tracking with an Event Camera
2025-05-12 23:53:08Angus Apps, Ziwei Wang, Vladimir Perejogin, Timothy Molloy, Robert MahonyAbstract
Events cameras are ideal sensors for enabling robots to detect and track objects in highly dynamic environments due to their low latency output, high temporal resolution, and high dynamic range. In this paper, we present the Asynchronous Event Multi-Object Tracking (AEMOT) algorithm for detecting and tracking multiple objects by processing individual raw events asynchronously. AEMOT detects salient event blob features by identifying regions of consistent optical flow using a novel Field of Active Flow Directions built from the Surface of Active Events. Detected features are tracked as candidate objects using the recently proposed Asynchronous Event Blob (AEB) tracker in order to construct small intensity patches of each candidate object. A novel learnt validation stage promotes or discards candidate objects based on classification of their intensity patches, with promoted objects having their position, velocity, size, and orientation estimated at their event rate. We evaluate AEMOT on a new Bee Swarm Dataset, where it tracks dozens of small bees with precision and recall performance exceeding that of alternative event-based detection and tracking algorithms by over 37%. Source code and the labelled event Bee Swarm Dataset will be open sourced
Abstract (translated)
事件相机由于其低延迟输出、高时间分辨率和宽动态范围，是使机器人在高度动态环境中检测和跟踪物体的理想传感器。本文介绍了异步事件多目标跟踪（AEMOT）算法，该算法通过异步处理单个原始事件来实现对多个对象的检测与追踪。 AEMOT通过识别一致的光学流区域，利用新颖的活跃流方向场从活动事件表面构建出来，以检测显著的事件团块特征。然后使用最近提出的异步事件团块（AEB）跟踪器将这些被检测到的特征作为候选物体进行追踪，并为每个候选对象构造小强度补丁。一种创新的学习验证阶段根据对它们强度补丁的分类来推广或排除候选物体，被推广的对象将在其事件速率下估计位置、速度、大小和方向。 我们使用新的蜜蜂群集数据集评估了AEMOT，该数据集中它能够精确地追踪数十只小蜜蜂，并且在精度和召回率性能上超越其他基于事件的检测与跟踪算法超过37%。源代码以及标记的事件蜜蜂群集数据集将开放源码提供。 这段翻译总结了论文的关键贡献和技术细节，强调了AEMOT算法如何利用新颖的技术来提升机器人在复杂动态环境下的感知能力，并且通过实际测试展示了该方法的有效性和优越性。
URL
https://arxiv.org/abs/2505.08126
PDF
-
MELLM: Exploring LLM-Powered Micro-Expression Understanding Enhanced by Subtle Motion Perception
2025-05-11 15:08:23Zhengye Zhang, Sirui Zhao, Shifeng Liu, Shukang Yin, Xinglong Mao, Tong Xu, Enhong ChenAbstract
Micro-expressions (MEs) are crucial psychological responses with significant potential for affective computing. However, current automatic micro-expression recognition (MER) research primarily focuses on discrete emotion classification, neglecting a convincing analysis of the subtle dynamic movements and inherent emotional cues. The rapid progress in multimodal large language models (MLLMs), known for their strong multimodal comprehension and language generation abilities, offers new possibilities. MLLMs have shown success in various vision-language tasks, indicating their potential to understand MEs comprehensively, including both fine-grained motion patterns and underlying emotional semantics. Nevertheless, challenges remain due to the subtle intensity and short duration of MEs, as existing MLLMs are not designed to capture such delicate frame-level facial dynamics. In this paper, we propose a novel Micro-Expression Large Language Model (MELLM), which incorporates a subtle facial motion perception strategy with the strong inference capabilities of MLLMs, representing the first exploration of MLLMs in the domain of ME analysis. Specifically, to explicitly guide the MLLM toward motion-sensitive regions, we construct an interpretable motion-enhanced color map by fusing onset-apex optical flow dynamics with the corresponding grayscale onset frame as the model input. Additionally, specialized fine-tuning strategies are incorporated to further enhance the model's visual perception of MEs. Furthermore, we construct an instruction-description dataset based on Facial Action Coding System (FACS) annotations and emotion labels to train our MELLM. Comprehensive evaluations across multiple benchmark datasets demonstrate that our model exhibits superior robustness and generalization capabilities in ME understanding (MEU). Code is available at this https URL.
Abstract (translated)
微表情（MEs）是具有重要情感计算潜力的关键心理反应。然而，当前的自动微表情识别（MER）研究主要集中在离散情绪分类上，忽略了对细微动态运动和内在情绪线索进行深入分析的问题。多模态大型语言模型（MLLMs），以其强大的跨模态理解和语言生成能力而闻名，在许多视觉-语言任务中表现出色，显示出全面理解MEs的潜力，包括精细的动作模式和潜在的情感语义。然而，由于微表情的细微强度和短暂持续时间，现有的MLLMs尚未设计为捕捉这种微妙的画面级面部动态变化。因此，本文提出了一种新颖的微表情大型语言模型（MELLM），该模型结合了细致面部运动感知策略与MLLM强大的推理能力，首次探索了MLLM在ME分析领域的应用。 具体而言，为了明确指导MLLM关注敏感的动作区域，我们构建了一个可解释的增强色彩地图，通过融合起始-峰值光学流动态和相应的灰度起始帧作为模型输入。此外，还引入了专门的微调策略，以进一步提升模型对微表情视觉感知的能力。另外，基于面部动作编码系统（FACS）注释和情绪标签构建了一个指令描述数据集来训练我们的MELLM。 在多个基准测试数据集上的全面评估表明，我们的模型在微表情理解（MEU）方面表现出色的鲁棒性和泛化能力。代码可在提供的链接中获取。
URL
https://arxiv.org/abs/2505.07007
PDF
-
Nonlinear Motion-Guided and Spatio-Temporal Aware Network for Unsupervised Event-Based Optical Flow
2025-05-08 09:39:19Zuntao Liu, Hao Zhuang, Junjie Jiang, Yuhang Song, Zheng FangAbstract
Event cameras have the potential to capture continuous motion information over time and space, making them well-suited for optical flow estimation. However, most existing learning-based methods for event-based optical flow adopt frame-based techniques, ignoring the spatio-temporal characteristics of events. Additionally, these methods assume linear motion between consecutive events within the loss time window, which increases optical flow errors in long-time sequences. In this work, we observe that rich spatio-temporal information and accurate nonlinear motion between events are crucial for event-based optical flow estimation. Therefore, we propose E-NMSTFlow, a novel unsupervised event-based optical flow network focusing on long-time sequences. We propose a Spatio-Temporal Motion Feature Aware (STMFA) module and an Adaptive Motion Feature Enhancement (AMFE) module, both of which utilize rich spatio-temporal information to learn spatio-temporal data associations. Meanwhile, we propose a nonlinear motion compensation loss that utilizes the accurate nonlinear motion between events to improve the unsupervised learning of our network. Extensive experiments demonstrate the effectiveness and superiority of our method. Remarkably, our method ranks first among unsupervised learning methods on the MVSEC and DSEC-Flow datasets. Our project page is available at this https URL.
Abstract (translated)
事件相机有能力捕捉长时间和空间上的连续运动信息，因此非常适合用于光流估计。然而，大多数现有的基于学习的方法在处理以事件为基础的光流时采用的是帧基技术，忽视了事件的空间时间和特性特征。此外，这些方法假设在损失时间窗口内的连续事件之间存在线性运动，这会导致长时间序列中的光流误差增加。 在这项工作中，我们观察到丰富的空间和时间信息以及事件之间的准确非线性运动对于基于事件的光流估计至关重要。因此，我们提出了E-NMSTFlow，这是一种新的无监督基于事件的光流网络，专注于长时序数据。我们提出了一种时空运动特征感知（STMFA）模块和一种自适应运动特征增强（AMFE）模块，这两个模块都利用了丰富的时空信息来学习时空数据关联。同时，我们还提出了一种非线性运动补偿损失函数，该函数利用事件之间的准确非线性运动以改善我们网络的无监督学习过程。 大量的实验表明我们的方法有效且优越，在MVSEC和DSEC-Flow数据集上基于无监督学习的方法中，我们的方法排名第一。我们的项目页面可在此链接访问：[提供的URL]
URL
https://arxiv.org/abs/2505.05089
PDF
-
A Simple Detector with Frame Dynamics is a Strong Tracker
2025-05-08 03:16:03Chenxu Peng, Chenxu Wang, Minrui Zou, Danyang Li, Zhengpeng Yang, Yimian Dai, Ming-Ming Cheng, Xiang LiAbstract
Infrared object tracking plays a crucial role in Anti-Unmanned Aerial Vehicle (Anti-UAV) applications. Existing trackers often depend on cropped template regions and have limited motion modeling capabilities, which pose challenges when dealing with tiny targets. To address this, we propose a simple yet effective infrared tiny-object tracker that enhances tracking performance by integrating global detection and motion-aware learning with temporal priors. Our method is based on object detection and achieves significant improvements through two key innovations. First, we introduce frame dynamics, leveraging frame difference and optical flow to encode both prior target features and motion characteristics at the input level, enabling the model to better distinguish the target from background clutter. Second, we propose a trajectory constraint filtering strategy in the post-processing stage, utilizing spatio-temporal priors to suppress false positives and enhance tracking robustness. Extensive experiments show that our method consistently outperforms existing approaches across multiple metrics in challenging infrared UAV tracking scenarios. Notably, we achieve state-of-the-art performance in the 4th Anti-UAV Challenge, securing 1st place in Track 1 and 2nd place in Track 2.
Abstract (translated)
红外目标跟踪在反无人机（Anti-UAV）应用中扮演着至关重要的角色。现有的追踪器通常依赖于裁剪的模板区域，并且具有有限的动作建模能力，这在处理微小目标时会带来挑战。为了解决这些问题，我们提出了一种简单而有效的红外微小物体追踪器，通过整合全局检测和带有时间先验的知识驱动学习来增强跟踪性能。 我们的方法基于对象检测，并通过两个关键创新实现了显著的改进。首先，我们引入了帧动态性，利用帧差分和光流技术在输入层面编码先前的目标特征以及动作特性，从而使模型能够更好地将目标与背景噪声区分开来。其次，在后期处理阶段，我们提出了一种轨迹约束过滤策略，利用时空先验来抑制假阳性并增强跟踪鲁棒性。 广泛的实验表明，我们的方法在具有挑战性的红外无人机追踪场景中以多种度量标准上始终优于现有方法。特别值得注意的是，我们在第四届反无人机挑战赛中取得了最新技术水平的表现，在Track 1和Track 2分别获得了第一名和第二名的成绩。
URL
https://arxiv.org/abs/2505.04917
PDF
-
Read My Ears! Horse Ear Movement Detection for Equine Affective State Assessment
2025-05-06 14:05:49Jo\~ao Alves, Pia Haubro Andersen, Rikke GadeAbstract
The Equine Facial Action Coding System (EquiFACS) enables the systematic annotation of facial movements through distinct Action Units (AUs). It serves as a crucial tool for assessing affective states in horses by identifying subtle facial expressions associated with discomfort. However, the field of horse affective state assessment is constrained by the scarcity of annotated data, as manually labelling facial AUs is both time-consuming and costly. To address this challenge, automated annotation systems are essential for leveraging existing datasets and improving affective states detection tools. In this work, we study different methods for specific ear AU detection and localization from horse videos. We leverage past works on deep learning-based video feature extraction combined with recurrent neural networks for the video classification task, as well as a classic optical flow based approach. We achieve 87.5% classification accuracy of ear movement presence on a public horse video dataset, demonstrating the potential of our approach. We discuss future directions to develop these systems, with the aim of bridging the gap between automated AU detection and practical applications in equine welfare and veterinary diagnostics. Our code will be made publicly available at this https URL.
Abstract (translated)
《马面部动作编码系统（EquiFACS）》允许通过特定的动作单元（AUs，Action Units）对马的面部运动进行系统的标注。它作为评估马的情绪状态的重要工具，在识别与不适相关的小幅面部表情方面发挥着关键作用。然而，由于缺乏标注数据，马情绪状态评估领域的研究受到了限制，因为手动标记面部动作单位既耗时又昂贵。为了应对这一挑战，自动注释系统对于利用现有数据集和改进情绪状态检测工具至关重要。 在本项工作中，我们研究了不同方法，用于从马视频中识别和定位特定的耳朵AUs。我们结合深度学习基线视频特征提取与递归神经网络应用于视频分类任务，并且采用了一种经典的基于光学流的方法。我们在一个公共马视频数据集中实现了87.5%的耳部运动存在分类准确率，展示了我们方法的潜力。 我们将讨论未来的发展方向，旨在弥合自动AUs检测和在赛马福利及兽医诊断中的实际应用之间的差距。我们的代码将在以下链接公开提供：[此 URL]。
URL
https://arxiv.org/abs/2505.03554
PDF
-
TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion
2025-05-06 02:12:19Haoyue Liu, Jinghan Xu, Yi Chang, Hanyu Zhou, Haozhi Zhao, Lin Wang, Luxin YanAbstract
Video frame interpolation (VFI) that leverages the bio-inspired event cameras as guidance has recently shown better performance and memory efficiency than the frame-based methods, thanks to the event cameras' advantages, such as high temporal resolution. A hurdle for event-based VFI is how to effectively deal with non-linear motion, caused by the dynamic changes in motion direction and speed within the scene. Existing methods either use events to estimate sparse optical flow or fuse events with image features to estimate dense optical flow. Unfortunately, motion errors often degrade the VFI quality as the continuous motion cues from events do not align with the dense spatial information of images in the temporal dimension. In this paper, we find that object motion is continuous in space, tracking local regions over continuous time enables more accurate identification of spatiotemporal feature correlations. In light of this, we propose a novel continuous point tracking-based VFI framework, named TimeTracker. Specifically, we first design a Scene-Aware Region Segmentation (SARS) module to divide the scene into similar patches. Then, a Continuous Trajectory guided Motion Estimation (CTME) module is proposed to track the continuous motion trajectory of each patch through events. Finally, intermediate frames at any given time are generated through global motion optimization and frame refinement. Moreover, we collect a real-world dataset that features fast non-linear motion. Extensive experiments show that our method outperforms prior arts in both motion estimation and frame interpolation quality.
Abstract (translated)
视频帧插值（VFI）技术利用受生物启发的事件相机作为指导，近期表现出优于基于帧的方法的性能和内存效率，这得益于事件相机的优势，如高时间分辨率。然而，基于事件的VFI的一个挑战是如何有效处理非线性运动，这是由于场景中运动方向和速度的动态变化引起的。现有方法要么利用事件来估计稀疏光流，要么融合事件与图像特征以估计密集光流。不幸的是，这些方法中的运动误差往往降低了VFI的质量，因为连续的运动线索（来自事件）在时间维度上并不总是与图像的空间信息对齐。 在这篇论文中，我们发现物体的运动在空间上是连续的，并且通过持续的时间跟踪局部区域能够更准确地识别时空特征的相关性。基于这一发现，我们提出了一种新颖的基于连续点追踪的VFI框架，名为TimeTracker。具体来说，我们首先设计了一个场景感知区域分割（SARS）模块来将场景划分为相似的小块。然后，提出了一个连续轨迹引导运动估计（CTME）模块，通过事件跟踪每个小块的持续运动轨迹。最后，通过全局运动优化和帧细化生成任意时间点上的中间帧。 此外，我们收集了一个具有快速非线性运动的真实世界数据集。广泛的实验表明，我们的方法在运动估计和插值帧质量方面都优于现有的技术。
URL
https://arxiv.org/abs/2505.03116
PDF
-
Unaligned RGB Guided Hyperspectral Image Super-Resolution with Spatial-Spectral Concordance
2025-05-04 13:29:31Yingkai Zhang, Zeqiang Lai, Tao Zhang, Ying Fu, Chenghu ZhouAbstract
Hyperspectral images super-resolution aims to improve the spatial resolution, yet its performance is often limited at high-resolution ratios. The recent adoption of high-resolution reference images for super-resolution is driven by the poor spatial detail found in low-resolution HSIs, presenting it as a favorable method. However, these approaches cannot effectively utilize information from the reference image, due to the inaccuracy of alignment and its inadequate interaction between alignment and fusion modules. In this paper, we introduce a Spatial-Spectral Concordance Hyperspectral Super-Resolution (SSC-HSR) framework for unaligned reference RGB guided HSI SR to address the issues of inaccurate alignment and poor interactivity of the previous approaches. Specifically, to ensure spatial concordance, i.e., align images more accurately across resolutions and refine textures, we construct a Two-Stage Image Alignment with a synthetic generation pipeline in the image alignment module, where the fine-tuned optical flow model can produce a more accurate optical flow in the first stage and warp model can refine damaged textures in the second stage. To enhance the interaction between alignment and fusion modules and ensure spectral concordance during reconstruction, we propose a Feature Aggregation module and an Attention Fusion module. In the feature aggregation module, we introduce an Iterative Deformable Feature Aggregation block to achieve significant feature matching and texture aggregation with the fusion multi-scale results guidance, iteratively generating learnable offset. Besides, we introduce two basic spectral-wise attention blocks in the attention fusion module to model the inter-spectra interactions. Extensive experiments on three natural or remote-sensing datasets show that our method outperforms state-of-the-art approaches on both quantitative and qualitative evaluations.
Abstract (translated)
超光谱图像的超分辨率处理旨在提高空间分辨率，但其性能在高比例尺时往往受限。近年来，采用高分辨率参考图像是为了克服低分辨率HSI（超光谱图像）中发现的空间细节不足的问题，这被认为是一种有利的方法。然而，这些方法无法有效地利用参考图的信息，原因在于对齐的不准确性以及对齐和融合模块之间交互不足。 在本文中，我们提出了一种新的空间-光谱一致性超光谱超分辨率（SSC-HSR）框架，该框架用于未对准的RGB图像引导HSI SR，并解决了以往方法中存在的对齐不准确及互动性差的问题。具体而言，为了确保空间一致性，即跨不同分辨率更精确地对齐图像并细化纹理，我们在图像对齐模块中构建了一个两阶段图像对齐与合成生成管道，其中经过微调的光流模型可以在第一阶段产生更为精确的光流，而变形模型则在第二阶段可以修复受损的纹理。为了增强对齐和融合模块之间的互动性，并确保重建过程中的光谱一致性，我们还提出了特征聚合模块和注意力融合模块。 在特征聚合模块中，我们引入了迭代可变形特征聚合块（Iterative Deformable Feature Aggregation block），该块可以在多尺度结果的指导下实现显著的特征匹配与纹理聚合，并且通过迭代生成学习到偏移量。此外，在注意力融合模块中，我们还介绍了两个基本的光谱注意块来建模不同光谱之间的相互作用。 在三个自然或遥感数据集上的大量实验表明，我们的方法在定量和定性评估方面均优于最先进的方法。
URL
https://arxiv.org/abs/2505.02109
PDF
-
Rethinking RGB-Event Semantic Segmentation with a Novel Bidirectional Motion-enhanced Event Representation
2025-05-02 19:19:58Zhen Yao, Xiaowen Ying, Mooi Choo ChuahAbstract
Event cameras capture motion dynamics, offering a unique modality with great potential in various computer vision tasks. However, RGB-Event fusion faces three intrinsic misalignments: (i) temporal, (ii) spatial, and (iii) modal misalignment. Existing voxel grid representations neglect temporal correlations between consecutive event windows, and their formulation with simple accumulation of asynchronous and sparse events is incompatible with the synchronous and dense nature of RGB modality. To tackle these challenges, we propose a novel event representation, Motion-enhanced Event Tensor (MET), which transforms sparse event voxels into a dense and temporally coherent form by leveraging dense optical flows and event temporal features. In addition, we introduce a Frequency-aware Bidirectional Flow Aggregation Module (BFAM) and a Temporal Fusion Module (TFM). BFAM leverages the frequency domain and MET to mitigate modal misalignment, while bidirectional flow aggregation and temporal fusion mechanisms resolve spatiotemporal misalignment. Experimental results on two large-scale datasets demonstrate that our framework significantly outperforms state-of-the-art RGB-Event semantic segmentation approaches. Our code is available at: this https URL.
Abstract (translated)
事件相机能够捕捉运动动态，为各种计算机视觉任务提供了一种独特且潜力巨大的模式。然而，RGB-事件融合面临三种内在的对齐问题：（i）时间上的不对齐，（ii）空间上的不对齐以及（iii）模态上的不对齐。现有的体素网格表示方法忽视了连续事件窗口间的时间相关性，并且其简单地累积异步且稀疏事件的做法与同步和密集的RGB模式本质不兼容。为了解决这些挑战，我们提出了一种新颖的事件表示方法——增强运动事件张量（Motion-enhanced Event Tensor，MET），该方法通过利用稠密光流和事件时间特征将稀疏事件体素转换成稠密且时间上连贯的形式。此外，我们还引入了频率感知双向流量聚合模块（Frequency-aware Bidirectional Flow Aggregation Module，BFAM）和时序融合模块（Temporal Fusion Module，TFM）。BFAM利用频域和MET来减轻模态不对齐问题，而双向流聚合与时序融合机制则解决了时空不对齐的问题。在两个大规模数据集上的实验结果表明，我们的框架显著优于当前最先进的RGB-事件语义分割方法。我们的代码可在以下网址获取：this https URL。
URL
https://arxiv.org/abs/2505.01548
PDF
