Optical_Flow
-
AnimalMotionCLIP: Embedding motion in CLIP for Animal Behavior Analysis
2025-04-30 12:26:37Enmin Zhong, Carlos R. del-Blanco, Daniel Berj\'on, Fernando Jaureguizar, Narciso Garc\'iaAbstract
Recently, there has been a surge of interest in applying deep learning techniques to animal behavior recognition, particularly leveraging pre-trained visual language models, such as CLIP, due to their remarkable generalization capacity across various downstream tasks. However, adapting these models to the specific domain of animal behavior recognition presents two significant challenges: integrating motion information and devising an effective temporal modeling scheme. In this paper, we propose AnimalMotionCLIP to address these challenges by interleaving video frames and optical flow information in the CLIP framework. Additionally, several temporal modeling schemes using an aggregation of classifiers are proposed and compared: dense, semi dense, and sparse. As a result, fine temporal actions can be correctly recognized, which is of vital importance in animal behavior analysis. Experiments on the Animal Kingdom dataset demonstrate that AnimalMotionCLIP achieves superior performance compared to state-of-the-art approaches.
Abstract (translated)
近期，人们对将深度学习技术应用于动物行为识别的兴趣大增，尤其是在利用像 CLIP 这样的预训练视觉语言模型方面。这些模型因其在各种下游任务中的卓越泛化能力而备受青睐。然而，要将这类模型适配到特定的动物行为识别领域，则面临着两个主要挑战：一是如何整合运动信息，二是如何设计有效的时序建模方案。 为此，在本文中我们提出了一种名为 AnimalMotionCLIP 的方法来解决这些挑战。该方法通过在 CLIP 模型框架内交错插入视频帧和光流信息来实现这一目标。此外，还提出了并比较了几种使用分类器聚合进行时序建模的方法：密集（dense）、半密集（semi dense）和稀疏（sparse）。结果表明，这些方案能够准确识别出精细的时序动作，在动物行为分析中具有重要意义。 通过在“动物王国”数据集上的实验验证，AnimalMotionCLIP 方法相较于现有最先进的方法取得了更优的表现。
URL
https://arxiv.org/abs/2505.00569
PDF
-
LPVIMO-SAM: Tightly-coupled LiDAR/Polarization Vision/Inertial/Magnetometer/Optical Flow Odometry via Smoothing and Mapping
2025-04-29 02:54:12Derui Shan, Peng Guo, Wenshuo Li, Du TaoAbstract
We propose a tightly-coupled LiDAR/Polarization Vision/Inertial/Magnetometer/Optical Flow Odometry via Smoothing and Mapping (LPVIMO-SAM) framework, which integrates LiDAR, polarization vision, inertial measurement unit, magnetometer, and optical flow in a tightly-coupled fusion. This framework enables high-precision and highly robust real-time state estimation and map construction in challenging environments, such as LiDAR-degraded, low-texture regions, and feature-scarce areas. The LPVIMO-SAM comprises two subsystems: a Polarized Vision-Inertial System and a LiDAR/Inertial/Magnetometer/Optical Flow System. The polarized vision enhances the robustness of the Visual/Inertial odometry in low-feature and low-texture scenarios by extracting the polarization information of the scene. The magnetometer acquires the heading angle, and the optical flow obtains the speed and height to reduce the accumulated error. A magnetometer heading prior factor, an optical flow speed observation factor, and a height observation factor are designed to eliminate the cumulative errors of the LiDAR/Inertial odometry through factor graph optimization. Meanwhile, the LPVIMO-SAM can maintain stable positioning even when one of the two subsystems fails, further expanding its applicability in LiDAR-degraded, low-texture, and low-feature environments. Code is available on this https URL.
Abstract (translated)
我们提出了一种基于平滑与映射（Smoothing and Mapping，SAM）的紧密耦合激光雷达/偏振视觉/惯性/磁力计/光流里程计框架（LPVIMO-SAM）。该框架集成了激光雷达、偏振视觉系统、惯性测量单元、磁力计和光流，并实现了紧密融合。此框架能够在复杂环境中实现高精度和高度鲁棒的实时状态估计与地图构建，包括在激光雷达性能下降、低纹理区域以及特征稀疏地区等挑战性条件下。 LPVIMO-SAM由两个子系统组成：偏振视觉/惯性系统（Polarized Vision-Inertial System）和激光雷达/惯性/磁力计/光流系统。其中，偏振视觉增强了在低特征与低纹理场景中视觉/惯性里程计的鲁棒性，通过提取场景中的偏振信息来实现这一点；磁力计获取航向角数据，而光流则用于获取速度和高度以减少累积误差。设计了磁力计航向先验因子、光流速度观测因子以及高度观测因子，通过图优化方法消除激光雷达/惯性里程计的累积误差。 此外，即使其中一个子系统失效时，LPVIMO-SAM仍能保持稳定定位性能，进一步扩展了其在激光雷达性能受限、低纹理和特征稀疏环境中的应用范围。相关代码可在此网址获取：[https URL]（原文中给出的链接未提供具体地址）。
URL
https://arxiv.org/abs/2504.20380
PDF
-
Motion-Enhanced Nonlocal Similarity Implicit Neural Representation for Infrared Dim and Small Target Detection
2025-04-22 07:42:00Pei Liu, Yisi Luo, Wenzhen Wang, Xiangyong CaoAbstract
Infrared dim and small target detection presents a significant challenge due to dynamic multi-frame scenarios and weak target signatures in the infrared modality. Traditional low-rank plus sparse models often fail to capture dynamic backgrounds and global spatial-temporal correlations, which results in background leakage or target loss. In this paper, we propose a novel motion-enhanced nonlocal similarity implicit neural representation (INR) framework to address these challenges. We first integrate motion estimation via optical flow to capture subtle target movements, and propose multi-frame fusion to enhance motion saliency. Second, we leverage nonlocal similarity to construct patch tensors with strong low-rank properties, and propose an innovative tensor decomposition-based INR model to represent the nonlocal patch tensor, effectively encoding both the nonlocal low-rankness and spatial-temporal correlations of background through continuous neural representations. An alternating direction method of multipliers is developed for the nonlocal INR model, which enjoys theoretical fixed-point convergence. Experimental results show that our approach robustly separates dim targets from complex infrared backgrounds, outperforming state-of-the-art methods in detection accuracy and robustness.
Abstract (translated)
红外低信噪比小目标检测由于动态多帧场景和弱目标特征，在红外模式下面临重大挑战。传统的低秩加稀疏模型常常无法捕捉动态背景以及全局时空相关性，从而导致背景泄露或目标丢失的问题。在本文中，我们提出了一种新的基于运动增强的非局部相似性的隐式神经网络表示（INR）框架来应对这些挑战。 首先，我们将通过光流估计的方式整合运动估算以捕获细微的目标移动，并提出了多帧融合技术来增强运动显著性。其次，我们利用非局部相似性构造具有强低秩性质的补丁张量，并提出了一种基于张量分解的创新INR模型来表示这些非局部补丁张量，通过连续神经网络表示有效地编码背景中的非局部低秩性和时空相关性。 为了对非局部INR模型进行优化，我们开发了一种交替方向乘子法（ADMM），该方法具有理论上的固定点收敛特性。实验结果表明，我们的方法能够稳健地将微弱目标从复杂的红外背景中分离出来，在检测准确率和鲁棒性方面优于现有的最先进技术。
URL
https://arxiv.org/abs/2504.15665
PDF
-
DiTPainter: Efficient Video Inpainting with Diffusion Transformers
2025-04-22 07:36:45Xian Wu, Chang LiuAbstract
Many existing video inpainting algorithms utilize optical flows to construct the corresponding maps and then propagate pixels from adjacent frames to missing areas by mapping. Despite the effectiveness of the propagation mechanism, they might encounter blurry and inconsistencies when dealing with inaccurate optical flows or large masks. Recently, Diffusion Transformer (DiT) has emerged as a revolutionary technique for video generation tasks. However, pretrained DiT models for video generation all contain a large amount of parameters, which makes it very time consuming to apply to video inpainting tasks. In this paper, we present DiTPainter, an end-to-end video inpainting model based on Diffusion Transformer (DiT). DiTPainter uses an efficient transformer network designed for video inpainting, which is trained from scratch instead of initializing from any large pretrained models. DiTPainter can address videos with arbitrary lengths and can be applied to video decaptioning and video completion tasks with an acceptable time cost. Experiments show that DiTPainter outperforms existing video inpainting algorithms with higher quality and better spatial-temporal consistency.
Abstract (translated)
许多现有的视频修复算法利用光流来构建相应的映射，然后通过映射将相邻帧的像素传播到缺失区域。虽然这种传播机制非常有效，但在处理不准确的光流或大面积遮罩时可能会导致模糊和不一致的问题。最近，扩散变换器（Diffusion Transformer, DiT）作为视频生成任务中的革命性技术崭露头角。然而，用于视频生成的预训练DiT模型参数量巨大，这使得它们在应用于视频修复任务时非常耗时。 在这篇论文中，我们介绍了DiTPainter，这是一个基于扩散变换器（Diffusion Transformer, DiT）的端到端视频修复模型。DiTPainter使用了一种专门设计用于视频修复的有效Transformer网络，并且是从头开始训练的，而不是从任何大型预训练模型初始化的。DiTPainter可以处理任意长度的视频，并能以可接受的时间成本应用于视频去遮挡和视频完成任务。实验表明，DiTPainter在质量和时空一致性方面优于现有的视频修复算法。
URL
https://arxiv.org/abs/2504.15661
PDF
-
PIV-FlowDiffuser:Transfer-learning-based denoising diffusion models for PIV
2025-04-21 08:22:58Qianyu Zhu, Junjie Wang, Jeremiah Hu, Jia Ai, Yong LeeAbstract
Deep learning algorithms have significantly reduced the computational time and improved the spatial resolution of particle image velocimetry~(PIV). However, the models trained on synthetic datasets might have a degraded performance on practical particle images due to domain gaps. As a result, special residual patterns are often observed for the vector fields of deep learning-based estimators. To reduce the special noise step-by-step, we employ a denoising diffusion model~(FlowDiffuser) for PIV analysis. And the data-hungry iterative denoising diffusion model is trained via a transfer learning strategy, resulting in our PIV-FlowDiffuser method. Specifically, (1) pre-training a FlowDiffuser model with multiple optical flow datasets of the computer vision community, such as Sintel, KITTI, etc; (2) fine-tuning the pre-trained model on synthetic PIV datasets. Note that the PIV images are upsampled by a factor of two to resolve the small-scale turbulent flow structures. The visualized results indicate that our PIV-FlowDiffuser effectively suppresses the noise patterns. Therefore, the denoising diffusion model reduces the average end-point error~($AEE$) by 59.4% over RAFT256-PIV baseline on the classic Cai's dataset. Besides, PIV-FlowDiffuser exhibits enhanced generalization performance on unseen particle images due to transfer learning. Overall, this study highlights the transfer-learning-based denoising diffusion models for PIV. And a detailed implementation is recommended for interested readers in the repository this https URL.
Abstract (translated)
深度学习算法已显著减少了粒子图像 velocimetry (PIV) 的计算时间，并提高了其空间分辨率。然而，基于合成数据集训练的模型在实际颗粒图像上的性能可能会因领域差距而下降。因此，在基于深度学习的估计器产生的向量场中经常会观察到特殊的噪声模式。为了逐步减少这种特殊噪声，我们使用了一种去噪扩散模型（FlowDiffuser）来进行 PIV 分析。通过采用迁移学习策略训练数据驱动型迭代去噪扩散模型，最终形成了我们的PIV-FlowDiffuser方法。 具体来说： 1. 使用计算机视觉社区中的多个光学流数据集（如Sintel、KITTI等）对 FlowDiffuser 模型进行预训练。 2. 在合成的 PIV 数据集上微调经过预训练的模型。需要注意的是，通过将PIV图像放大两倍来解析小规模湍流结构。 可视化结果表明，我们的PIV-FlowDiffuser方法能够有效抑制噪声模式。因此，在经典的Cai数据集上，与RAFT256-PIV基线相比，去噪扩散模型使平均终点误差（$AEE$）降低了59.4%。此外，由于采用了迁移学习，PIV-FlowDiffuser 在未见过的颗粒图像上的泛化性能得到了增强。 总的来说，这项研究强调了基于迁移学习的去噪扩散模型在 PIV 中的应用价值，并推荐有兴趣的读者参考该研究的详细实现（详见此链接：this https URL）。
URL
https://arxiv.org/abs/2504.14952
PDF
-
FlowLoss: Dynamic Flow-Conditioned Loss Strategy for Video Diffusion Models
2025-04-20 08:22:29Kuanting Wu, Kei Ota, Asako KanezakiAbstract
Video Diffusion Models (VDMs) can generate high-quality videos, but often struggle with producing temporally coherent motion. Optical flow supervision is a promising approach to address this, with prior works commonly employing warping-based strategies that avoid explicit flow matching. In this work, we explore an alternative formulation, FlowLoss, which directly compares flow fields extracted from generated and ground-truth videos. To account for the unreliability of flow estimation under high-noise conditions in diffusion, we propose a noise-aware weighting scheme that modulates the flow loss across denoising steps. Experiments on robotic video datasets suggest that FlowLoss improves motion stability and accelerates convergence in early training stages. Our findings offer practical insights for incorporating motion-based supervision into noise-conditioned generative models.
Abstract (translated)
视频扩散模型（VDMs）能够生成高质量的视频，但在产生连贯的时间运动方面常常遇到困难。光学流监督是一种有前景的方法来解决这一问题，之前的工作通常采用基于变形的策略，避免了显式的流匹配。在这项工作中，我们探索了一种替代方案FlowLoss，它直接比较从生成和真实视频中提取的流动场。为了应对扩散过程中的高噪声条件下流动估计不可靠的问题，我们提出了一种噪声感知加权方案，该方案调节去噪步骤中的流动损失。在机器人视频数据集上的实验表明，FlowLoss提高了运动稳定性，并且在早期训练阶段加速了收敛速度。我们的研究结果为将基于运动的监督纳入噪声条件下的生成模型提供了实用见解。
URL
https://arxiv.org/abs/2504.14535
PDF
-
Neural Ganglion Sensors: Learning Task-specific Event Cameras Inspired by the Neural Circuit of the Human Retina
Abstract
Inspired by the data-efficient spiking mechanism of neurons in the human eye, event cameras were created to achieve high temporal resolution with minimal power and bandwidth requirements by emitting asynchronous, per-pixel intensity changes rather than conventional fixed-frame rate images. Unlike retinal ganglion cells (RGCs) in the human eye, however, which integrate signals from multiple photoreceptors within a receptive field to extract spatio-temporal features, conventional event cameras do not leverage local spatial context when deciding which events to fire. Moreover, the eye contains around 20 different kinds of RGCs operating in parallel, each attuned to different features or conditions. Inspired by this biological design, we introduce Neural Ganglion Sensors, an extension of traditional event cameras that learns task-specific spatio-temporal retinal kernels (i.e., RGC "events"). We evaluate our design on two challenging tasks: video interpolation and optical flow. Our results demonstrate that our biologically inspired sensing improves performance relative to conventional event cameras while reducing overall event bandwidth. These findings highlight the promise of RGC-inspired event sensors for edge devices and other low-power, real-time applications requiring efficient, high-resolution visual streams.
Abstract (translated)
受人类眼睛中神经元高效数据处理机制的启发，事件相机被创造出来以实现高时间分辨率，并且在能耗和带宽需求极低的情况下工作。它们通过发射异步、像素级的强度变化来代替传统的固定帧率图像。然而，与人眼中的视网膜节细胞（RGCs）不同的是，后者整合多个感光细胞在其感受野内的信号以提取空间-时间特征，传统事件相机在决定哪些事件触发时并不利用局部的空间信息上下文。 此外，眼睛中大约有20种不同类型的RGCs并行工作，每一种都针对不同的特性和条件。受到这种生物设计的启发，我们引入了神经节细胞传感器（Neural Ganglion Sensors），这是传统事件相机的一种扩展形式，它学习特定任务的空间-时间视网膜核（即，“RGC事件”）。我们在两个具有挑战性的任务上评估了我们的设计方案：视频插值和光流计算。我们的结果显示，在与传统的事件相机相比时，生物启发式传感提高了性能，并且减少了整体的事件带宽需求。 这些发现强调了受RGC启发的事件传感器在边缘设备及其他需要低功耗、实时应用中的潜力，特别是在那些要求高效、高分辨率视觉数据流的应用场景中。
URL
https://arxiv.org/abs/2504.13457
PDF
-
MicroFlow: Domain-Specific Optical Flow for Ground Deformation Estimation in Seismic Events
2025-04-18 04:10:40Juliette Bertrand, Sophie Giffard-Roisin, James Hollingsworth, Julien MairalAbstract
Dense ground displacement measurements are crucial for geological studies but are impractical to collect directly. Traditionally, displacement fields are estimated using patch matching on optical satellite images from different acquisition times. While deep learning-based optical flow models are promising, their adoption in ground deformation analysis is hindered by challenges such as the absence of real ground truth, the need for sub-pixel precision, and temporal variations due to geological or anthropogenic changes. In particular, we identify that deep learning models relying on explicit correlation layers struggle at estimating small displacements in real-world conditions. Instead, we propose a model that employs iterative refinements with explicit warping layers and a correlation-independent backbone, enabling sub-pixel precision. Additionally, a non-convex variant of Total Variation regularization preserves fault-line sharpness while maintaining smoothness elsewhere. Our model significantly outperforms widely used geophysics methods on semi-synthetic benchmarks and generalizes well to challenging real-world scenarios captured by both medium- and high-resolution sensors. Project page: this https URL.
Abstract (translated)
密集地面位移测量对于地质研究至关重要，但直接收集这些数据是不切实际的。传统上，通过不同时间获取的光学卫星图像进行块匹配来估计位移场。虽然基于深度学习的光流模型前景广阔，但在地表变形分析中的应用却因缺乏真实的数据基准、需要亚像素精度以及由于地质或人为变化引起的时序波动等挑战而受阻。特别地，我们发现依赖于显式相关层的深度学习模型在实际条件下估计微小位移方面存在困难。 为此，我们提出了一种采用迭代细化和显式的变形层，并且与相关性无关的基础架构的方法，这使得亚像素精度成为可能。此外，非凸Total Variation正则化方法可以保持断层面的锐度，同时确保其他区域的平滑性。我们的模型在半合成基准测试中显著优于广泛使用的地球物理方法，并且能够很好地推广到由中高分辨率传感器捕捉的各种实际挑战场景中。 项目页面: [此链接](https://this https URL)
URL
https://arxiv.org/abs/2504.13452
PDF
-
Event-Enhanced Blurry Video Super-Resolution
2025-04-17 15:55:41Dachun Kai, Yueyi Zhang, Jin Wang, Zeyu Xiao, Zhiwei Xiong, Xiaoyan SunAbstract
In this paper, we tackle the task of blurry video super-resolution (BVSR), aiming to generate high-resolution (HR) videos from low-resolution (LR) and blurry inputs. Current BVSR methods often fail to restore sharp details at high resolutions, resulting in noticeable artifacts and jitter due to insufficient motion information for deconvolution and the lack of high-frequency details in LR frames. To address these challenges, we introduce event signals into BVSR and propose a novel event-enhanced network, Ev-DeblurVSR. To effectively fuse information from frames and events for feature deblurring, we introduce a reciprocal feature deblurring module that leverages motion information from intra-frame events to deblur frame features while reciprocally using global scene context from the frames to enhance event features. Furthermore, to enhance temporal consistency, we propose a hybrid deformable alignment module that fully exploits the complementary motion information from inter-frame events and optical flow to improve motion estimation in the deformable alignment process. Extensive evaluations demonstrate that Ev-DeblurVSR establishes a new state-of-the-art performance on both synthetic and real-world datasets. Notably, on real data, our method is +2.59 dB more accurate and 7.28$\times$ faster than the recent best BVSR baseline FMA-Net. Code: this https URL.
Abstract (translated)
在这篇论文中，我们处理的是模糊视频超分辨率（BVSR）任务，目标是从低分辨率（LR）和模糊的输入生成高分辨率（HR）视频。当前的BVSR方法通常无法在高分辨率下恢复出清晰细节，导致由于解卷积所需的运动信息不足以及低分辨率帧缺乏高频细节而产生明显的伪影和抖动问题。为了解决这些挑战，我们引入了事件信号到BVSR中，并提出了一种新的基于事件增强网络Ev-DeblurVSR。为了有效地融合来自帧和事件的信息以进行特征去模糊化处理，我们设计了一个互反特征去模糊模块，该模块利用帧内事件的运动信息来对帧特征进行去模糊化处理，同时又利用全局场景上下文从帧中提升事件特征的质量。此外，为增强时间一致性，我们提出了一种混合可变形对齐模块，它充分利用了来自帧间事件和光流的互补运动信息，从而改进了可变形对齐过程中的运动估计。 广泛的评估表明，Ev-DeblurVSR在合成数据集和真实世界数据集中都建立了新的最先进的性能。值得注意的是，在真实数据上，我们的方法比最近最好的BVSR基准FMA-Net更准确+2.59 dB且速度提高了7.28倍。代码：[请参考相关链接] (此链接处应填入实际的代码链接地址)
URL
https://arxiv.org/abs/2504.13042
PDF
-
SC3EF: A Joint Self-Correlation and Cross-Correspondence Estimation Framework for Visible and Thermal Image Registration
2025-04-17 11:54:12Xi Tong, Xing Luo, Jiangxin Yang, Yanpeng CaoAbstract
Multispectral imaging plays a critical role in a range of intelligent transportation applications, including advanced driver assistance systems (ADAS), traffic monitoring, and night vision. However, accurate visible and thermal (RGB-T) image registration poses a significant challenge due to the considerable modality differences. In this paper, we present a novel joint Self-Correlation and Cross-Correspondence Estimation Framework (SC3EF), leveraging both local representative features and global contextual cues to effectively generate RGB-T correspondences. For this purpose, we design a convolution-transformer-based pipeline to extract local representative features and encode global correlations of intra-modality for inter-modality correspondence estimation between unaligned visible and thermal images. After merging the local and global correspondence estimation results, we further employ a hierarchical optical flow estimation decoder to progressively refine the estimated dense correspondence maps. Extensive experiments demonstrate the effectiveness of our proposed method, outperforming the current state-of-the-art (SOTA) methods on representative RGB-T datasets. Furthermore, it also shows competitive generalization capabilities across challenging scenarios, including large parallax, severe occlusions, adverse weather, and other cross-modal datasets (e.g., RGB-N and RGB-D).
Abstract (translated)
多光谱成像在包括高级驾驶辅助系统（ADAS）、交通监控和夜视在内的智能运输应用中发挥着关键作用。然而，由于模态差异显著，准确地注册可见光和热图像（RGB-T）构成了重大挑战。本文提出了一种新颖的联合自相关与跨模式对应估计框架（SC3EF），该框架利用局部代表性特征和全局上下文线索有效生成RGB-T对应的配对关系。为此，我们设计了一个基于卷积-变压器的管道，用于提取局部代表性特征，并编码同一模态内的全球关联以进行不同模态之间的未对齐可见光与热图像间的跨模式对应估计。在合并了局部和全局对应性估算结果后，进一步利用层次化光学流估计算法解码器逐步细化所估计的密集对应的映射图。 广泛的实验表明，我们提出的方法优于当前最先进的（SOTA）方法，在代表性的RGB-T数据集上取得了卓越的效果。此外，它还展示了在具有挑战性场景中的强大泛化能力，包括大视差、严重遮挡、恶劣天气及其他跨模态数据集（如RGB-N和RGB-D）。
URL
https://arxiv.org/abs/2504.12869
PDF
-
SAM-Based Building Change Detection with Distribution-Aware Fourier Adaptation and Edge-Constrained Warping
2025-04-17 03:47:43Yun-Cheng Li, Sen Lei, Yi-Tao Zhao, Heng-Chao Li, Jun Li, Antonio PlazaAbstract
Building change detection remains challenging for urban development, disaster assessment, and military reconnaissance. While foundation models like Segment Anything Model (SAM) show strong segmentation capabilities, SAM is limited in the task of building change detection due to domain gap issues. Existing adapter-based fine-tuning approaches face challenges with imbalanced building distribution, resulting in poor detection of subtle changes and inaccurate edge extraction. Additionally, bi-temporal misalignment in change detection, typically addressed by optical flow, remains vulnerable to background noises. This affects the detection of building changes and compromises both detection accuracy and edge recognition. To tackle these challenges, we propose a new SAM-Based Network with Distribution-Aware Fourier Adaptation and Edge-Constrained Warping (FAEWNet) for building change detection. FAEWNet utilizes the SAM encoder to extract rich visual features from remote sensing images. To guide SAM in focusing on specific ground objects in remote sensing scenes, we propose a Distribution-Aware Fourier Aggregated Adapter to aggregate task-oriented changed information. This adapter not only effectively addresses the domain gap issue, but also pays attention to the distribution of changed buildings. Furthermore, to mitigate noise interference and misalignment in height offset estimation, we design a novel flow module that refines building edge extraction and enhances the perception of changed buildings. Our state-of-the-art results on the LEVIR-CD, S2Looking and WHU-CD datasets highlight the effectiveness of FAEWNet. The code is available at this https URL.
Abstract (translated)
构建变化检测在城市发展、灾害评估和军事侦察中仍面临挑战。尽管像Segment Anything Model (SAM)这样的基础模型显示出了强大的分割能力，但SAM在建筑变化检测任务上因领域差距问题而受到限制。现有的基于适配器的微调方法面对不平衡的建筑物分布时也遇到了挑战，导致细微变化难以被发现，并且边缘提取不够准确。此外，在使用光学流处理双时间序列错位的变化检测中，背景噪声对其构成了威胁，影响了建筑变化的检测并削弱了检测精度和边缘识别。 为了解决这些问题，我们提出了一种新的基于SAM的网络——带有分布感知傅里叶适应和边界约束扭曲（FAEWNet）的建筑物变化检测方法。该网络利用SAM编码器从遥感图像中提取丰富的视觉特征。为了引导SAM关注特定地面目标，我们提出了一个分布感知傅里叶聚合适配器来整合任务导向的变化信息。这个适配器不仅有效地解决了领域差距问题，还注意到了变化建筑的分布情况。此外，为了解决噪声干扰和高度偏移估计中的错位问题，我们设计了一个新的流模块以精化建筑物边缘提取，并增强了对变化建筑物的感知。 在LEVIR-CD、S2Looking以及WHU-CD数据集上的最先进的实验结果证明了FAEWNet的有效性。该代码可在以下链接获取：[这里提供原网址]。
URL
https://arxiv.org/abs/2504.12619
PDF
-
Perturbed State Space Feature Encoders for Optical Flow with Event Cameras
2025-04-14 19:45:07Gokul Raju Govinda Raju, Nikola Zubi\'c, Marco Cannici, Davide ScaramuzzaAbstract
With their motion-responsive nature, event-based cameras offer significant advantages over traditional cameras for optical flow estimation. While deep learning has improved upon traditional methods, current neural networks adopted for event-based optical flow still face temporal and spatial reasoning limitations. We propose Perturbed State Space Feature Encoders (P-SSE) for multi-frame optical flow with event cameras to address these challenges. P-SSE adaptively processes spatiotemporal features with a large receptive field akin to Transformer-based methods, while maintaining the linear computational complexity characteristic of SSMs. However, the key innovation that enables the state-of-the-art performance of our model lies in our perturbation technique applied to the state dynamics matrix governing the SSM system. This approach significantly improves the stability and performance of our model. We integrate P-SSE into a framework that leverages bi-directional flows and recurrent connections, expanding the temporal context of flow prediction. Evaluations on DSEC-Flow and MVSEC datasets showcase P-SSE's superiority, with 8.48% and 11.86% improvements in EPE performance, respectively.
Abstract (translated)
事件相机通过其运动响应特性，在光流估计方面相对于传统相机具有显著优势。尽管深度学习已经改进了传统的光流估算方法，但目前用于基于事件的光流的神经网络仍然面临时间和空间推理的局限性。为了解决这些问题，我们提出了扰动状态空间特征编码器（P-SSE），这是一种用于多帧光流处理的方法，适用于事件相机。P-SSE以类似于Transformer方法的方式自适应地处理具有大感受野的空间时间特征，同时保持SSM（状态空间模型）特有的线性计算复杂度特点。 然而，使我们的模型达到最先进的性能的关键创新在于我们应用于控制SSM系统的状态动力学矩阵的扰动技术。这种方法显著提高了模型的稳定性和性能。我们将P-SSE整合到一个框架中，该框架利用双向流和递归连接来扩展光流预测的时间上下文。 在DSEC-Flow和MVSEC数据集上的评估展示了P-SSE的优势，分别在EPE（端点误差）性能上取得了8.48%和11.86%的提升。
URL
https://arxiv.org/abs/2504.10669
PDF
-
WildLive: Near Real-time Visual Wildlife Tracking onboard UAVs
2025-04-14 12:21:16Nguyen Ngoc Dat, Tom Richardson, Matthew Watson, Kilian Meier, Jenna Kline, Sid Reid, Guy Maalouf, Duncan Hine, Majid Mirmehdi, Tilo BurghardtAbstract
Live tracking of wildlife via high-resolution video processing directly onboard drones is widely unexplored and most existing solutions rely on streaming video to ground stations to support navigation. Yet, both autonomous animal-reactive flight control beyond visual line of sight and/or mission-specific individual and behaviour recognition tasks rely to some degree on this capability. In response, we introduce WildLive -- a near real-time animal detection and tracking framework for high-resolution imagery running directly onboard uncrewed aerial vehicles (UAVs). The system performs multi-animal detection and tracking at 17fps+ for HD and 7fps+ on 4K video streams suitable for operation during higher altitude flights to minimise animal disturbance. Our system is optimised for Jetson Orin AGX onboard hardware. It integrates the efficiency of sparse optical flow tracking and mission-specific sampling with device-optimised and proven YOLO-driven object detection and segmentation techniques. Essentially, computational resource is focused onto spatio-temporal regions of high uncertainty to significantly improve UAV processing speeds without domain-specific loss of accuracy. Alongside, we introduce our WildLive dataset, which comprises 200k+ annotated animal instances across 19k+ frames from 4K UAV videos collected at the Ol Pejeta Conservancy in Kenya. All frames contain ground truth bounding boxes, segmentation masks, as well as individual tracklets and tracking point trajectories. We compare our system against current object tracking approaches including OC-SORT, ByteTrack, and SORT. Our multi-animal tracking experiments with onboard hardware confirm that near real-time high-resolution wildlife tracking is possible on UAVs whilst maintaining high accuracy levels as needed for future navigational and mission-specific animal-centric operational autonomy.
Abstract (translated)
通过高分辨率视频处理在无人机上实时追踪野生动物的方法目前还很少被探索，而大多数现有的解决方案依赖于将视频流传输到地面站来支持导航。然而，无论是自主的动物反应式飞行控制（超越视线范围）还是特定任务下的个体和行为识别任务，在某种程度上都依赖于此能力。为此，我们介绍了WildLive——这是一个近实时野生动物检测与追踪框架，适用于在无人驾驶航空器(UAV)上直接运行高分辨率图像处理。 该系统能够在高清视频流中实现多动物检测和追踪速度超过17帧/秒，并且在4K视频流中的速度超过7帧/秒。这些性能对于高空飞行操作而言非常合适，能够尽量减少对野生动物的干扰。我们的系统优化了Jetson Orin AGX车载硬件，并结合稀疏光流跟踪、任务特定采样与设备优化和经过验证的YOLO驱动对象检测及分割技术的有效性。简言之，计算资源集中在时空区域中的高不确定性部分上，从而显著提升无人机处理速度，同时保持对领域的准确度不降低。 此外，我们还推出了WildLive数据集，该数据集包括超过20万个标注动物实例，这些实例来自在肯尼亚奥尔佩杰塔保护区收集的4K无人机视频中的19,000多帧。所有帧都包含地面真值边界框、分割掩码以及个体跟踪轨迹和跟踪点路径。 我们还将WildLive系统与现有对象追踪方法（如OC-SORT，ByteTrack 和SORT）进行了比较。通过车载硬件进行的多动物追踪实验表明，在无人驾驶航空器上实现近实时高分辨率野生动物追踪是可能的，并且在所需的未来导航及特定任务下的动物中心操作自主性方面仍能保持高水平的准确度。
URL
https://arxiv.org/abs/2504.10165
PDF
-
Hardware, Algorithms, and Applications of the Neuromorphic Vision Sensor: a Review
2025-04-11 14:46:36Claudio Cimarelli, Jose Andres Millan-Romera, Holger Voos, Jose Luis Sanchez-LopezAbstract
Neuromorphic, or event, cameras represent a transformation in the classical approach to visual sensing encodes detected instantaneous per-pixel illumination changes into an asynchronous stream of event packets. Their novelty compared to standard cameras lies in the transition from capturing full picture frames at fixed time intervals to a sparse data format which, with its distinctive qualities, offers potential improvements in various applications. However, these advantages come at the cost of reinventing algorithmic procedures or adapting them to effectively process the new data format. In this survey, we systematically examine neuromorphic vision along three main dimensions. First, we highlight the technological evolution and distinctive hardware features of neuromorphic cameras from their inception to recent models. Second, we review image processing algorithms developed explicitly for event-based data, covering key works on feature detection, tracking, and optical flow -which form the basis for analyzing image elements and transformations -as well as depth and pose estimation or object recognition, which interpret more complex scene structures and components. These techniques, drawn from classical computer vision and modern data-driven approaches, are examined to illustrate the breadth of applications for event-based cameras. Third, we present practical application case studies demonstrating how event cameras have been successfully used across various industries and scenarios. Finally, we analyze the challenges limiting widespread adoption, identify significant research gaps compared to standard imaging techniques, and outline promising future directions and opportunities that neuromorphic vision offers.
Abstract (translated)
神经形态（或事件）相机代表了传统视觉传感方法的重大转变，将检测到的即时像素光照变化编码为异步事件数据包流。与标准摄像头相比，其创新之处在于从固定时间间隔捕获完整图像帧转变为一种稀疏的数据格式，这种新型格式凭借其独特性质，在各种应用中具有潜在改进空间。然而，这些优势需要重新发明或适应现有算法以有效处理新的数据格式。在本综述中，我们系统地从三个主要维度研究神经形态视觉技术。 首先，我们概述了神经形态相机自诞生以来的技术演变及其独特的硬件特性，并涵盖了最新模型的发展情况。 其次，我们回顾专门为事件数据开发的图像处理算法，包括关键作品中的特征检测、跟踪和光流（这些构成了分析图像元素及变化的基础），以及深度估计和姿态估计或对象识别（用于解释更复杂的场景结构和组件）。这些技术结合了传统的计算机视觉方法与现代数据驱动的方法，用以展示事件相机在广泛应用领域的广阔前景。 第三，我们介绍了一些实用的应用案例研究，展示了事件摄像机如何成功应用于各种行业和场景中。 最后，我们分析了阻碍广泛采用的挑战，识别出与标准成像技术相比的重要研究缺口，并概述神经形态视觉未来有前景的发展方向及机遇。
URL
https://arxiv.org/abs/2504.08588
PDF
-
Extending Visual Dynamics for Video-to-Music Generation
2025-04-10 09:47:26Xiaohao Liu, Teng Tu, Yunshan Ma, Tat-Seng ChuaAbstract
Music profoundly enhances video production by improving quality, engagement, and emotional resonance, sparking growing interest in video-to-music generation. Despite recent advances, existing approaches remain limited in specific scenarios or undervalue the visual dynamics. To address these limitations, we focus on tackling the complexity of dynamics and resolving temporal misalignment between video and music representations. To this end, we propose DyViM, a novel framework to enhance dynamics modeling for video-to-music generation. Specifically, we extract frame-wise dynamics features via a simplified motion encoder inherited from optical flow methods, followed by a self-attention module for aggregation within frames. These dynamic features are then incorporated to extend existing music tokens for temporal alignment. Additionally, high-level semantics are conveyed through a cross-attention mechanism, and an annealing tuning strategy benefits to fine-tune well-trained music decoders efficiently, therefore facilitating seamless adaptation. Extensive experiments demonstrate DyViM's superiority over state-of-the-art (SOTA) methods.
Abstract (translated)
音乐在视频制作中通过提升质量、增强互动性和情感共鸣，极大地丰富了视频内容，这激发了人们对视频转音乐生成技术的兴趣。尽管最近有了很多进展，现有的方法仍然在特定场景下受到限制或未能充分重视视觉动态的变化。为了解决这些问题，我们专注于处理动态复杂性，并解决视频与音乐表示之间的时序不对齐问题。 为此，我们提出了DyViM（Dynamic Video to Music），这是一个旨在增强视频到音乐生成过程中动态建模的新框架。具体来说，我们通过简化自光学流方法继承而来的运动编码器提取逐帧的动态特征，随后采用自我注意模块进行帧内的聚合操作。这些动态特性被用来扩展现有的音乐令牌以实现时间上的对齐。此外，高层次语义信息则是通过交叉注意力机制传达，并且一个退火调整策略有助于高效地微调预训练好的音乐解码器，从而促进无缝适应。 广泛的实验表明了DyViM相对于现有最先进的（SOTA）方法的优越性。
URL
https://arxiv.org/abs/2504.07594
PDF
-
ZS-VCOS: Zero-Shot Outperforms Supervised Video Camouflaged Object Segmentation
2025-04-10 06:24:54Wenqi Guo, Shan DuAbstract
Camouflaged object segmentation presents unique challenges compared to traditional segmentation tasks, primarily due to the high similarity in patterns and colors between camouflaged objects and their backgrounds. Effective solutions to this problem have significant implications in critical areas such as pest control, defect detection, and lesion segmentation in medical imaging. Prior research has predominantly emphasized supervised or unsupervised pre-training methods, leaving zero-shot approaches significantly underdeveloped. Existing zero-shot techniques commonly utilize the Segment Anything Model (SAM) in automatic mode or rely on vision-language models to generate cues for segmentation; however, their performances remain unsatisfactory, likely due to the similarity of the camouflaged object and the background. Optical flow, commonly utilized for detecting moving objects, has demonstrated effectiveness even with camouflaged entities. Our method integrates optical flow, a vision-language model, and SAM 2 into a sequential pipeline. Evaluated on the MoCA-Mask dataset, our approach achieves outstanding performance improvements, significantly outperforming existing zero-shot methods by raising the F-measure ($F_\beta^w$) from 0.296 to 0.628. Remarkably, our approach also surpasses supervised methods, increasing the F-measure from 0.476 to 0.628. Additionally, evaluation on the MoCA-Filter dataset demonstrates an increase in the success rate from 0.628 to 0.697 when compared with FlowSAM, a supervised transfer method. A thorough ablation study further validates the individual contributions of each component. More details can be found on this https URL.
Abstract (translated)
伪装物体分割相较于传统的分割任务提出了独特的挑战，主要是由于伪装对象与其背景在图案和颜色上的高度相似性。该问题的有效解决方案对害虫控制、缺陷检测以及医学影像中的病变分割等关键领域具有重要意义。先前的研究主要集中在监督学习或无监督预训练方法上，而零样本（zero-shot）方法的发展相对滞后。现有的零样本技术通常利用Segment Anything Model (SAM) 的自动模式或者依赖视觉-语言模型来生成用于分割的提示；然而，由于伪装对象与背景的高度相似性，这些方法的表现不尽如人意。 光学流（optical flow），常被用来检测移动物体，在处理伪装实体时也表现出有效性。我们提出的方法结合了光学流、视觉-语言模型和SAM 2，并将其整合为一个顺序管道。在MoCA-Mask数据集上进行的评估显示，我们的方法实现了显著的性能提升，将F-measure（$F_\beta^w$）从0.296提高到0.628，大幅超越了现有的零样本方法。值得注意的是，这种方法还超过了监督学习的方法，提高了F-measure从0.476至0.628。此外，在MoCA-Filter数据集上的评估表明，相较于FlowSAM（一种监督式迁移学习方法），成功概率从0.628提高到了0.697。 详细的消融实验进一步验证了每个组成部分的独立贡献。有关此工作的更多详细信息，请参阅[链接]提供的资料。(注：实际链接未提供，在描述中提到的位置放置“https URL”仅为指示性表述)
URL
https://arxiv.org/abs/2505.01431
PDF
-
Intrinsic Saliency Guided Trunk-Collateral Network for Unsupervised Video Object Segmentation
2025-04-08 11:02:14Xiangyu Zheng, Wanyun Li, Songcheng He, Xiaoqiang Li, We ZhangAbstract
Recent unsupervised video object segmentation (UVOS) methods predominantly adopt the motion-appearance paradigm. Mainstream motion-appearance approaches use either the two-encoder structure to separately encode motion and appearance features, or the single-encoder structure for joint encoding. However, these methods fail to properly balance the motion-appearance relationship. Consequently, even with complex fusion modules for motion-appearance integration, the extracted suboptimal features degrade the models' overall performance. Moreover, the quality of optical flow varies across scenarios, making it insufficient to rely solely on optical flow to achieve high-quality segmentation results. To address these challenges, we propose the Intrinsic Saliency guided Trunk-Collateral Net}work (ISTC-Net), which better balances the motion-appearance relationship and incorporates model's intrinsic saliency information to enhance segmentation performance. Specifically, considering that optical flow maps are derived from RGB images, they share both commonalities and differences. We propose a novel Trunk-Collateral structure. The shared trunk backbone captures the motion-appearance commonality, while the collateral branch learns the uniqueness of motion features. Furthermore, an Intrinsic Saliency guided Refinement Module (ISRM) is devised to efficiently leverage the model's intrinsic saliency information to refine high-level features, and provide pixel-level guidance for motion-appearance fusion, thereby enhancing performance without additional input. Experimental results show that ISTC-Net achieved state-of-the-art performance on three UVOS datasets (89.2% J&F on DAVIS-16, 76% J on YouTube-Objects, 86.4% J on FBMS) and four standard video salient object detection (VSOD) benchmarks with the notable increase, demonstrating its effectiveness and superiority over previous methods.
Abstract (translated)
最近的无监督视频对象分割（UVOS）方法主要采用运动外观范式。主流的运动外观方法要么使用双编码器结构来分别编码运动和外观特征，要么使用单个编码器进行联合编码。然而，这些方法未能很好地平衡运动与外观之间的关系。因此，即使配备了复杂的融合模块用于整合运动与外观，提取出的次优特征仍然会降低模型的整体性能。此外，光流的质量在不同的场景中变化很大，这使得仅依赖于光流来实现高质量分割结果变得不够充分。 为了解决这些挑战，我们提出了基于内在显著性引导的主干分支网络（ISTC-Net）。该方法能够更好地平衡运动与外观之间的关系，并通过利用模型的内在显著性信息来提升分割性能。具体而言，考虑到光学流图是从RGB图像中推导出来的，它们既有共同之处也有差异点。为此，我们设计了一种新颖的主干分支结构：共享的主干背骨捕捉了运动和外观的共通性，而分支部分则学习到运动特征的独特性。 此外，还开发了一个基于内在显著性的精炼模块（ISRM），旨在有效地利用模型的内在显著信息来细化高层次特征，并为运动与外观融合提供像素级别的指导，从而在不增加额外输入的情况下提高性能。 实验结果显示，ISTC-Net 在三个 UVOS 数据集（DAVIS-16 的 J&F 评分达到 89.2%，YouTube-Objects 的 J 评分为 76%，FBMS 的 J 评分为 86.4%）以及四个标准视频显著对象检测（VSOD）基准测试中取得了最先进的性能，并且与之前的模型相比有明显提升，这证明了其有效性及优越性。
URL
https://arxiv.org/abs/2504.05904
PDF
-
Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modeling
2025-04-07 22:21:54Tasmiah Haque, Md. Asif Bin Syed, Byungheon Jeong, Xue Bai, Sumit Mohan, Somdyuti Paul, Imtiaz Ahmed, Srinjoy DasAbstract
We propose a deep learning framework designed to significantly optimize bandwidth for motion-transfer-enabled video applications, including video conferencing, virtual reality interactions, health monitoring systems, and vision-based real-time anomaly detection. To capture complex motion effectively, we utilize the First Order Motion Model (FOMM), which encodes dynamic objects by detecting keypoints and their associated local affine transformations. These keypoints are identified using a self-supervised keypoint detector and arranged into a time series corresponding to the successive frames. Forecasting is performed on these keypoints by integrating two advanced generative time series models into the motion transfer pipeline, namely the Variational Recurrent Neural Network (VRNN) and the Gated Recurrent Unit with Normalizing Flow (GRU-NF). The predicted keypoints are subsequently synthesized into realistic video frames using an optical flow estimator paired with a generator network, thereby facilitating accurate video forecasting and enabling efficient, low-frame-rate video transmission. We validate our results across three datasets for video animation and reconstruction using the following metrics: Mean Absolute Error, Joint Embedding Predictive Architecture Embedding Distance, Structural Similarity Index, and Average Pair-wise Displacement. Our results confirm that by utilizing the superior reconstruction property of the Variational Autoencoder, the VRNN integrated FOMM excels in applications involving multi-step ahead forecasts such as video conferencing. On the other hand, by leveraging the Normalizing Flow architecture for exact likelihood estimation, and enabling efficient latent space sampling, the GRU-NF based FOMM exhibits superior capabilities for producing diverse future samples while maintaining high visual quality for tasks like real-time video-based anomaly detection.
Abstract (translated)
我们提出了一种深度学习框架，旨在显著优化运动传输增强型视频应用的带宽使用，包括视频会议、虚拟现实交互、健康监测系统以及基于视觉的实时异常检测。为了有效捕捉复杂动作，我们采用了第一顺序运动模型（FOMM），该模型通过检测关键点及其相关的局部仿射变换来编码动态对象。 这些关键点通过自监督的关键点检测器识别，并根据连续帧排列成时间序列。接下来，在运动传输管道中结合了两种先进的生成式时间序列模型——变分递归神经网络（VRNN）和门控循环单元与规范化流（GRU-NF），对这些关键点进行预测。 随后，利用光流估算器配合生成网络将预测的关键点合成出逼真的视频帧。这不仅促进了准确的视频预测，还使得低帧率视频传输成为可能，并且更加高效。我们在三个用于视频动画和重建的数据集上通过以下指标验证了我们的结果：平均绝对误差、关节嵌入预测架构嵌入距离、结构相似性指数以及平均成对位移。 实验结果显示，利用变分自编码器的优越重构特性，VRNN集成FOMM在多步预测应用（如视频会议）中表现出色。而通过采用规范化流架构进行精确似然估计，并实现高效的潜在空间采样，基于GRU-NF的FOMM则展现了更强大的能力，在实时基于视频的异常检测等任务中生成多样化的未来样本同时保持高质量视觉效果。
URL
https://arxiv.org/abs/2504.05537
PDF
-
FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistency
2025-04-06 09:44:30Shiyan Liu, Rui Qu, Yan JinAbstract
Generating consecutive images of lip movements that align with a given speech in audio-driven lip synthesis is a challenging task. While previous studies have made strides in synchronization and visual quality, lip intelligibility and video fluency remain persistent challenges. This work proposes FluentLip, a two-stage approach for audio-driven lip synthesis, incorporating three featured strategies. To improve lip synchronization and intelligibility, we integrate a phoneme extractor and encoder to generate a fusion of audio and phoneme information for multimodal learning. Additionally, we employ optical flow consistency loss to ensure natural transitions between image frames. Furthermore, we incorporate a diffusion chain during the training of Generative Adversarial Networks (GANs) to improve both stability and efficiency. We evaluate our proposed FluentLip through extensive experiments, comparing it with five state-of-the-art (SOTA) approaches across five metrics, including a proposed metric called Phoneme Error Rate (PER) that evaluates lip pose intelligibility and video fluency. The experimental results demonstrate that our FluentLip approach is highly competitive, achieving significant improvements in smoothness and naturalness. In particular, it outperforms these SOTA approaches by approximately $\textbf{16.3%}$ in Fréchet Inception Distance (FID) and $\textbf{35.2%}$ in PER.
Abstract (translated)
生成与给定音频中的演讲同步的连续唇部动作图像，以进行基于音频驱动的唇形合成是一项具有挑战性的任务。尽管先前的研究在同步和视觉质量方面取得了进展，但唇形清晰度和视频流畅性仍然是持续存在的难题。本文提出了一种名为FluentLip的两阶段方法用于音频驱动的唇形合成，并整合了三个特色策略。为了提高唇部动作的同步性和可理解性，我们集成了一个音素提取器和编码器来生成音频与音素信息的融合，以进行多模态学习。此外，我们采用了光学流一致性损失，确保图像帧之间的自然过渡。另外，在训练生成对抗网络（GAN）时，我们引入了扩散链，旨在提高稳定性和效率。 我们通过广泛的实验评估了所提出的FluentLip方法，并将其与五个最先进的方法在包括音素错误率（PER）在内的五项指标上进行了比较，该指标用于评估唇部姿态的可理解性及视频流畅度。实验结果表明，我们的FluentLip方法具有很强的竞争性，在平滑性和自然度方面取得了显著改进。特别地，在Fréchet inception距离（FID）和音素错误率（PER）这两项关键指标上，与最先进的方法相比，分别提升了约16.3% 和 35.2%。
URL
https://arxiv.org/abs/2504.04427
PDF
-
Simultaneous Motion And Noise Estimation with Event Cameras
2025-04-05 02:47:40Shintaro Shiba, Yoshimitsu Aoki, Guillermo GallegoAbstract
Event cameras are emerging vision sensors, whose noise is challenging to characterize. Existing denoising methods for event cameras consider other tasks such as motion estimation separately (i.e., sequentially after denoising). However, motion is an intrinsic part of event data, since scene edges cannot be sensed without motion. This work proposes, to the best of our knowledge, the first method that simultaneously estimates motion in its various forms (e.g., ego-motion, optical flow) and noise. The method is flexible, as it allows replacing the 1-step motion estimation of the widely-used Contrast Maximization framework with any other motion estimator, such as deep neural networks. The experiments show that the proposed method achieves state-of-the-art results on the E-MLB denoising benchmark and competitive results on the DND21 benchmark, while showing its efficacy on motion estimation and intensity reconstruction tasks. We believe that the proposed approach contributes to strengthening the theory of event-data denoising, as well as impacting practical denoising use-cases, as we release the code upon acceptance. Project page: this https URL
Abstract (translated)
事件相机是一种新兴的视觉传感器，其噪声特征难以表征。现有的事件相机去噪方法通常将运动估计等其他任务与去噪过程分开处理（即，在去噪后依次进行）。然而，运动是事件数据的一个内在组成部分，因为如果没有物体或摄像机自身的移动，场景边缘无法被感知到。本工作提出了一种据我们所知首次同时估计各种形式的运动（例如自运动、光流）和噪声的方法。该方法具有灵活性，可以在广泛应用的对比最大化框架中替换1步运动估计算法，并可采用包括深度神经网络在内的其他运动估算器。 实验结果表明，在E-MLB去噪基准测试上，所提方法达到了当前最佳性能；在DND21基准测试上也取得了有竞争力的结果。同时，它还在运动估计和强度重建任务中证明了其有效性。我们认为该提出的办法有助于加强事件数据去噪理论，并且通过发布代码，将对实际应用中的去噪案例产生影响。 项目页面：[此链接](https://this-url)（请根据实际情况填写URL）。
URL
https://arxiv.org/abs/2504.04029
PDF
