Optical_Flow
-
Midway Network: Learning Representations for Recognition and Motion from Latent Dynamics
2025-10-07 04:07:44Christopher Hoang, Mengye RenAbstract
Object recognition and motion understanding are key components of perception that complement each other. While self-supervised learning methods have shown promise in their ability to learn from unlabeled data, they have primarily focused on obtaining rich representations for either recognition or motion rather than both in tandem. On the other hand, latent dynamics modeling has been used in decision making to learn latent representations of observations and their transformations over time for control and planning tasks. In this work, we present Midway Network, a new self-supervised learning architecture that is the first to learn strong visual representations for both object recognition and motion understanding solely from natural videos, by extending latent dynamics modeling to this domain. Midway Network leverages a midway top-down path to infer motion latents between video frames, as well as a dense forward prediction objective and hierarchical structure to tackle the complex, multi-object scenes of natural videos. We demonstrate that after pretraining on two large-scale natural video datasets, Midway Network achieves strong performance on both semantic segmentation and optical flow tasks relative to prior self-supervised learning methods. We also show that Midway Network's learned dynamics can capture high-level correspondence via a novel analysis method based on forward feature perturbation.
Abstract (translated)
物体识别和运动理解是感知中的关键组成部分，二者相辅相成。虽然自监督学习方法在从未标记数据中学习方面显示出巨大潜力，但它们主要专注于获取用于对象识别或运动的丰富表示，而不是两者并重。另一方面，在决策制定中使用的潜在动态建模旨在通过时间推移学习观察及其转换的隐式表示，以供控制和规划任务使用。在这项工作中，我们介绍了Midway Network，这是一种新的自监督学习架构，它首次仅从自然视频中同时学习出强大的视觉表征来支持物体识别和运动理解，并将潜在动态建模扩展到此领域。Midway Network利用一条中间的自顶向下路径，在视频帧之间推断运动隐式表示，同时还采用了密集前向预测目标和分层结构，以解决自然视频中的复杂多对象场景问题。我们证明了经过两个大规模自然视频数据集的预训练之后，Midway Network在语义分割和光流任务上相比先前的自监督学习方法具有更强的表现力。此外，我们展示了通过基于前向特征扰动的新分析方法发现，Midway Network所学得的动力学能够捕捉高层次的一致性。
URL
https://arxiv.org/abs/2510.05558
PDF
-
Learning Efficient Meshflow and Optical Flow from Event Cameras
2025-10-05 09:30:59Xinglong Luo, Ao Luo, Kunming Luo, Zhengning Wang, Ping Tan, Bing Zeng, Shuaicheng LiuAbstract
In this paper, we explore the problem of event-based meshflow estimation, a novel task that involves predicting a spatially smooth sparse motion field from event cameras. To start, we review the state-of-the-art in event-based flow estimation, highlighting two key areas for further research: i) the lack of meshflow-specific event datasets and methods, and ii) the underexplored challenge of event data density. First, we generate a large-scale High-Resolution Event Meshflow (HREM) dataset, which showcases its superiority by encompassing the merits of high resolution at 1280x720, handling dynamic objects and complex motion patterns, and offering both optical flow and meshflow labels. These aspects have not been fully explored in previous works. Besides, we propose Efficient Event-based MeshFlow (EEMFlow) network, a lightweight model featuring a specially crafted encoder-decoder architecture to facilitate swift and accurate meshflow estimation. Furthermore, we upgrade EEMFlow network to support dense event optical flow, in which a Confidence-induced Detail Completion (CDC) module is proposed to preserve sharp motion boundaries. We conduct comprehensive experiments to show the exceptional performance and runtime efficiency (30x faster) of our EEMFlow model compared to the recent state-of-the-art flow method. As an extension, we expand HREM into HREM+, a multi-density event dataset contributing to a thorough study of the robustness of existing methods across data with varying densities, and propose an Adaptive Density Module (ADM) to adjust the density of input event data to a more optimal range, enhancing the model's generalization ability. We empirically demonstrate that ADM helps to significantly improve the performance of EEMFlow and EEMFlow+ by 8% and 10%, respectively. Code and dataset are released at this https URL.
Abstract (translated)
在这篇论文中，我们探讨了基于事件的网格流估计问题，这是一个新颖的任务，涉及从事件相机预测出一个空间上平滑且稀疏的运动场。首先，我们回顾了基于事件的流动估计领域的最新进展，并指出了两个关键的研究方向：i) 缺乏专门针对网格流的事件数据集和方法；ii) 未充分探索的事件数据密度挑战。为此，我们生成了一个大规模的高分辨率事件网格流（HREM）数据集，该数据集展示了其优越性，包括在1280x720分辨率下具有高质量图像、处理动态物体以及复杂的运动模式的能力，并提供光学流动和网格流标签。这些方面在之前的工作中并未得到充分探索。 此外，我们提出了一种轻量级的基于事件的网格流（EEMFlow）网络模型，其特征是一个特别设计的编码-解码架构，旨在加速并提高网格流估计的准确性。另外，我们将EEMFlow网络升级为支持密集事件光学流动，在此过程中提出了一个信心诱导细节完成（CDC）模块来保持尖锐的运动边界。 我们进行了全面实验，展示了我们的EEMFlow模型在性能和运行时间效率方面的卓越表现，与最近最先进的流方法相比快30倍。作为扩展，我们将HREM数据集升级为HREM+，这是一个多密度事件数据集，有助于对现有方法在不同密度的数据上鲁棒性的彻底研究，并提出了一个自适应密度模块（ADM），用于调整输入事件数据的密度范围，从而增强模型的泛化能力。 我们通过实证展示了ADM能够显著提升EEMFlow和EEMFlow+的表现，分别提高了8%和10%。代码和数据集可在以下网址获取：[此链接]（请替换为实际发布的URL）。
URL
https://arxiv.org/abs/2510.04111
PDF
-
Test-Time Defense Against Adversarial Attacks via Stochastic Resonance of Latent Ensembles
2025-10-03 17:57:25Dong Lao, Yuxiang Zhang, Haniyeh Ehsani Oskouie, Yangchao Wu, Alex Wong, Stefano SoattoAbstract
We propose a test-time defense mechanism against adversarial attacks: imperceptible image perturbations that significantly alter the predictions of a model. Unlike existing methods that rely on feature filtering or smoothing, which can lead to information loss, we propose to "combat noise with noise" by leveraging stochastic resonance to enhance robustness while minimizing information loss. Our approach introduces small translational perturbations to the input image, aligns the transformed feature embeddings, and aggregates them before mapping back to the original reference image. This can be expressed in a closed-form formula, which can be deployed on diverse existing network architectures without introducing additional network modules or fine-tuning for specific attack types. The resulting method is entirely training-free, architecture-agnostic, and attack-agnostic. Empirical results show state-of-the-art robustness on image classification and, for the first time, establish a generic test-time defense for dense prediction tasks, including stereo matching and optical flow, highlighting the method's versatility and practicality. Specifically, relative to clean (unperturbed) performance, our method recovers up to 68.1% of the accuracy loss on image classification, 71.9% on stereo matching, and 29.2% on optical flow under various types of adversarial attacks.
Abstract (translated)
我们提出了一种对抗攻击的测试时间防御机制：通过施加几乎不可察觉的图像扰动来显著改变模型预测结果。与现有的依赖特征过滤或平滑的方法不同，这些方法可能会导致信息丢失，我们提议采用“以噪音对抗噪音”的策略，利用随机共振增强鲁棒性的同时将信息损失最小化。我们的方法通过对输入图像进行小的平移扰动、对转换后的特征嵌入进行对齐并聚合，然后映射回原始参考图像来实现这一目标。这种方法可以用闭合形式公式表示，并且可以在各种现有的网络架构上部署，而无需引入额外的网络模块或针对特定攻击类型进行微调。该方法完全不需要训练，对于不同的网络架构和攻击方式都是通用的。 实验结果表明，在图像分类任务中，我们的方法达到了最先进的鲁棒性；此外，首次为密集预测任务（如立体匹配和光流）建立了一种通用测试时间防御机制，突显了该方法的灵活性和实用性。具体而言，相对于干净（未受干扰）的表现，我们的方法在各种类型的对抗攻击下，在图像分类中恢复了高达68.1%的精度损失，在立体匹配中恢复了71.9%，在光流中恢复了29.2%。
URL
https://arxiv.org/abs/2510.03224
PDF
-
How Far Do Time Series Foundation Models Paint the Landscape of Real-World Benchmarks ?
2025-09-30 14:53:05Lujun Li, Lama Sleem, Yiqun Wang, Yangjie Xu, Niccol\`o Gentile, Radu StateAbstract
Recent evaluations of time-series foundation models (TSFMs) have emphasized synthetic benchmarks, leaving real-world generalization less thoroughly examined. This work proposes a novel benchmarking approach that bridges synthetic and realistic data by extracting temporal signals from real-world video using optical flow and curating datasets reflecting everyday temporal dynamics. Building upon this pipeline, we introduce REAL-V-TSFM, a novel dataset designed to capture rich and diverse time series derived from real-world videos. Experimental results on three state-of-the-art of TSFMs under zero-shot forecasting shows that, despite strong performance on conventional benchmarks, these models predominantly exhibit performance degradation on the proposed dataset, indicating limited generalizability in these foundation models. These findings highlight the urgent need for data-centric benchmarking and diverse model structure to advance TSFMs toward genuine universality, while further validating the effectiveness of our video-based time series data extraction pipeline.
Abstract (translated)
最近对时间序列基础模型（TSFM）的评估主要集中在合成基准上，而现实世界中的泛化性能则较少被深入研究。本文提出了一种新的基准测试方法，该方法通过使用光学流从真实世界的视频中提取时间信号，并整理出反映日常动态变化的数据集来连接合成数据和实际数据之间的鸿沟。在此基础上，我们引入了REAL-V-TSFM，这是一个新型数据集，旨在捕捉来自真实世界视频的丰富多样的时间序列。 在三个最先进的TSFM模型上进行的零样本预测实验表明，尽管这些模型在传统的基准测试中表现出色，但在我们的提议数据集中却普遍表现下降，这表明现有的基础模型在泛化能力方面存在局限性。这一发现强调了数据为中心的基准测试方法以及多样化模型结构对于推动时间序列基础模型向真正普适性的发展的迫切需求，并进一步验证了我们基于视频的时间序列数据提取流水线的有效性。
URL
https://arxiv.org/abs/2509.26347
PDF
-
Fast Feature Field : A Predictive Representation of Events
2025-09-29 17:52:31Richeek Das, Kostas Daniilidis, Pratik ChaudhariAbstract
This paper develops a mathematical argument and algorithms for building representations of data from event-based cameras, that we call Fast Feature Field ($\text{F}^3$). We learn this representation by predicting future events from past events and show that it preserves scene structure and motion information. $\text{F}^3$ exploits the sparsity of event data and is robust to noise and variations in event rates. It can be computed efficiently using ideas from multi-resolution hash encoding and deep sets - achieving 120 Hz at HD and 440 Hz at VGA resolutions. $\text{F}^3$ represents events within a contiguous spatiotemporal volume as a multi-channel image, enabling a range of downstream tasks. We obtain state-of-the-art performance on optical flow estimation, semantic segmentation, and monocular metric depth estimation, on data from three robotic platforms (a car, a quadruped robot and a flying platform), across different lighting conditions (daytime, nighttime), environments (indoors, outdoors, urban, as well as off-road) and dynamic vision sensors (resolutions and event rates). Our implementations can predict these tasks at 25-75 Hz at HD resolution.
Abstract (translated)
本文开发了一个数学论证和算法，用于从事件相机生成数据表示，我们称之为快速特征场（Fast Feature Field, $\text{F}^3$）。通过预测过去事件的未来事件来学习这种表示，并展示了它能保留场景结构和运动信息。$\text{F}^3$ 利用了事件数据的稀疏性，在噪声和事件率变化下具有鲁棒性，可以通过多分辨率哈希编码和深度集的思想高效计算出来，在高清分辨率下实现120 Hz，在VGA分辨率下实现440 Hz。$\text{F}^3$ 将事件在连续时空体积内表示为一个多通道图像，从而支持一系列下游任务。我们在三个机器人平台（汽车、四足机器人和飞行平台）的数据上获得了光流估计、语义分割以及单目度量深度估计方面的最佳性能，在不同光照条件（白天、夜间）、环境（室内、室外、城市及越野）以及动态视觉传感器（分辨率和事件率）下的数据也取得了良好效果。我们的实现可以在高清分辨率下以25-75 Hz 的速率预测这些任务。
URL
https://arxiv.org/abs/2509.25146
PDF
-
Forge4D: Feed-Forward 4D Human Reconstruction and Interpolation from Uncalibrated Sparse-view Videos
2025-09-29 02:47:14Yingdong Hu, Yisheng He, Jinnan Chen, Weihao Yuan, Kejie Qiu, Zehong Lin, Siyu Zhu, Zilong Dong, Jun ZhangAbstract
Instant reconstruction of dynamic 3D humans from uncalibrated sparse-view videos is critical for numerous downstream applications. Existing methods, however, are either limited by the slow reconstruction speeds or incapable of generating novel-time representations. To address these challenges, we propose Forge4D, a feed-forward 4D human reconstruction and interpolation model that efficiently reconstructs temporally aligned representations from uncalibrated sparse-view videos, enabling both novel view and novel time synthesis. Our model simplifies the 4D reconstruction and interpolation problem as a joint task of streaming 3D Gaussian reconstruction and dense motion prediction. For the task of streaming 3D Gaussian reconstruction, we first reconstruct static 3D Gaussians from uncalibrated sparse-view images and then introduce learnable state tokens to enforce temporal consistency in a memory-friendly manner by interactively updating shared information across different timestamps. For novel time synthesis, we design a novel motion prediction module to predict dense motions for each 3D Gaussian between two adjacent frames, coupled with an occlusion-aware Gaussian fusion process to interpolate 3D Gaussians at arbitrary timestamps. To overcome the lack of the ground truth for dense motion supervision, we formulate dense motion prediction as a dense point matching task and introduce a self-supervised retargeting loss to optimize this module. An additional occlusion-aware optical flow loss is introduced to ensure motion consistency with plausible human movement, providing stronger regularization. Extensive experiments demonstrate the effectiveness of our model on both in-domain and out-of-domain datasets. Project page and code at: this https URL.
Abstract (translated)
从未经校准的稀疏视角视频中即时重建动态3D人体对于许多下游应用至关重要。然而，现有方法要么受限于缓慢的重建速度，要么无法生成新时间表示。为了解决这些挑战，我们提出了Forge4D，这是一种前馈式4D人体重建和插值模型，能够高效地从未经校准的稀疏视角视频中重构出时间对齐的表现形式，从而支持新视角合成与新时间合成。 我们的模型将4D重建和插值问题简化为一个连续3D高斯重建和密集运动预测的任务。对于连续3D高斯重建任务，我们首先从未经校准的稀疏视图图像中重构静态3D高斯分布，并引入可学习的状态标记以通过互动更新不同时间戳之间的共享信息来强制执行一种内存友好的时间一致性方法。对于新时间合成，我们设计了一个新颖的运动预测模块，用于在两个相邻帧之间为每个3D高斯预测密集运动，并结合一个遮挡感知的高斯融合过程，在任意时间点插值3D高斯分布。 为了克服缺乏密集运动监督的真实标签的问题，我们将密集运动预测定义为密集点匹配任务，并引入自监督重定向损失来优化该模块。此外，我们还提出了一种遮挡感知的光流损失以确保与合理的身体动作相一致的运动一致性，提供更强的正则化。 大量的实验展示了我们的模型在域内和跨域数据集上的有效性。项目页面和代码可在[此处](https://this-url.com)访问（实际链接需要替换为正确的URL）。
URL
https://arxiv.org/abs/2509.24209
PDF
-
Temporal-Aware Iterative Speech Model for Dementia Detection
2025-09-26 01:56:07Chukwuemeka Ugwu, Oluwafemi OyelekeAbstract
Deep learning systems often struggle with processing long sequences, where computational complexity can become a bottleneck. Current methods for automated dementia detection using speech frequently rely on static, time-agnostic features or aggregated linguistic content, lacking the flexibility to model the subtle, progressive deterioration inherent in speech production. These approaches often miss the dynamic temporal patterns that are critical early indicators of cognitive decline. In this paper, we introduce TAI-Speech, a Temporal Aware Iterative framework that dynamically models spontaneous speech for dementia detection. The flexibility of our method is demonstrated through two key innovations: 1) Optical Flow-inspired Iterative Refinement: By treating spectrograms as sequential frames, this component uses a convolutional GRU to capture the fine-grained, frame-to-frame evolution of acoustic features. 2) Cross-Attention Based Prosodic Alignment: This component dynamically aligns spectral features with prosodic patterns, such as pitch and pauses, to create a richer representation of speech production deficits linked to functional decline (IADL). TAI-Speech adaptively models the temporal evolution of each utterance, enhancing the detection of cognitive markers. Experimental results on the DementiaBank dataset show that TAI-Speech achieves a strong AUC of 0.839 and 80.6\% accuracy, outperforming text-based baselines without relying on ASR. Our work provides a more flexible and robust solution for automated cognitive assessment, operating directly on the dynamics of raw audio.
Abstract (translated)
深度学习系统在处理长序列时常常遇到挑战，计算复杂性往往成为瓶颈。目前用于自动检测痴呆症的语音方法通常依赖于静态、时间无关特征或聚合的语言内容，缺乏建模言语生产中细微且渐进式退化的灵活性。这些方法经常会错过动态的时间模式，而这些模式是认知衰退早期的重要指标。在本文中，我们介绍了一种新的框架TAI-Speech（Temporal Aware Iterative），它能够动态地对自发语音进行痴呆症检测。通过两个关键创新展示了我们的方法的灵活性： 1) **光学流启发的迭代优化**：该组件将光谱图视为连续帧，并使用卷积GRU来捕捉声学特征在帧到帧之间的细微演变。 2) **基于交叉注意力的韵律对齐**：此部分动态地将频谱特征与音高和停顿等韵律模式进行对齐，以创建更丰富的言语产出缺陷表示，这些缺陷与功能衰退（IADL）相关联。 TAI-Speech能够适应性地建模每个话语的时间演变，从而增强认知标志的检测能力。在DementiaBank数据集上的实验结果显示，TAI-Speech取得了AUC为0.839和准确率为80.6%的良好性能，在不依赖于ASR（自动语音识别）的情况下超越了基于文本的方法基准。我们的工作提供了一种更为灵活且稳健的自动化认知评估解决方案，直接在原始音频的动力学上进行操作。
URL
https://arxiv.org/abs/2510.00030
PDF
-
DeLiVR: Differential Spatiotemporal Lie Bias for Efficient Video Deraining
2025-09-26 00:29:36Shuning Sun, Jialang Lu, Xiang Chen, Jichao Wang, Dianjie Lu, Guijuan Zhang, Guangwei Gao, Zhuoran ZhengAbstract
Videos captured in the wild often suffer from rain streaks, blur, and noise. In addition, even slight changes in camera pose can amplify cross-frame mismatches and temporal artifacts. Existing methods rely on optical flow or heuristic alignment, which are computationally expensive and less robust. To address these challenges, Lie groups provide a principled way to represent continuous geometric transformations, making them well-suited for enforcing spatial and temporal consistency in video modeling. Building on this insight, we propose DeLiVR, an efficient video deraining method that injects spatiotemporal Lie-group differential biases directly into attention scores of the network. Specifically, the method introduces two complementary components. First, a rotation-bounded Lie relative bias predicts the in-plane angle of each frame using a compact prediction module, where normalized coordinates are rotated and compared with base coordinates to achieve geometry-consistent alignment before feature aggregation. Second, a differential group displacement computes angular differences between adjacent frames to estimate a velocity. This bias computation combines temporal decay and attention masks to focus on inter-frame relationships while precisely matching the direction of rain streaks. Extensive experimental results demonstrate the effectiveness of our method on publicly available benchmarks.
Abstract (translated)
野外拍摄的视频常常会受到雨痕、模糊和噪声的影响。此外，即使是相机姿态的轻微变化也会放大帧间不匹配和时间上的伪影。现有的方法依赖于光学流或启发式对齐方式，这些方法在计算上成本高昂且不够鲁棒。为了解决这些问题，李群（Lie groups）提供了一种原理性的方法来表示连续几何变换，这使得它们非常适合用于确保视频建模中的空间和时间一致性。 基于这一洞察，我们提出了DeLiVR，这是一种高效的去雨迹视频处理方法，它直接在注意力分数中注入了时空李群微分偏差。具体来说，该方法引入了两个互补的组件： 1. **旋转受限李相关偏置**：利用一个紧凑的预测模块预测每个帧的平面内角度，其中归一化坐标被旋转并与基准坐标进行比较，以实现几何一致性对齐前的功能聚合。 2. **微分群位移**：计算相邻帧之间的角差来估计速度。这种偏差计算结合了时间衰减和注意力掩码，专注于帧间关系的同时精确匹配雨迹的方向。 广泛的实验结果证明了我们的方法在公开可用的基准测试中的有效性。
URL
https://arxiv.org/abs/2509.21719
PDF
-
MedVSR: Medical Video Super-Resolution with Cross State-Space Propagation
2025-09-25 14:56:59Xinyu Liu, Guolei Sun, Cheng Wang, Yixuan Yuan, Ender KonukogluAbstract
High-resolution (HR) medical videos are vital for accurate diagnosis, yet are hard to acquire due to hardware limitations and physiological constraints. Clinically, the collected low-resolution (LR) medical videos present unique challenges for video super-resolution (VSR) models, including camera shake, noise, and abrupt frame transitions, which result in significant optical flow errors and alignment difficulties. Additionally, tissues and organs exhibit continuous and nuanced structures, but current VSR models are prone to introducing artifacts and distorted features that can mislead doctors. To this end, we propose MedVSR, a tailored framework for medical VSR. It first employs Cross State-Space Propagation (CSSP) to address the imprecise alignment by projecting distant frames as control matrices within state-space models, enabling the selective propagation of consistent and informative features to neighboring frames for effective alignment. Moreover, we design an Inner State-Space Reconstruction (ISSR) module that enhances tissue structures and reduces artifacts with joint long-range spatial feature learning and large-kernel short-range information aggregation. Experiments across four datasets in diverse medical scenarios, including endoscopy and cataract surgeries, show that MedVSR significantly outperforms existing VSR models in reconstruction performance and efficiency. Code released at this https URL.
Abstract (translated)
高分辨率（HR）医学视频对于准确诊断至关重要，但由于硬件限制和生理约束，很难获取。在临床上，收集到的低分辨率（LR）医学视频为视频超分辨率（VSR）模型带来了独特的挑战，包括摄像机抖动、噪声及画面间的突然过渡，这些问题会导致显著的光流误差和对齐困难。此外，组织和器官展示出持续且细微的结构特征，但目前的VSR模型容易引入伪影和扭曲特性，这些可能误导医生的判断。为此，我们提出了MedVSR，这是一个专门用于医学视频超分辨率（VSR）的框架。 MedVSR首先采用交叉状态空间传播（CSSP），通过在状态空间模型中将相隔较远的画面投影为控制矩阵来解决不精确对齐的问题。这使得可以有选择地向相邻帧传递一致且具有信息量的特征，以实现有效的画面对齐。此外，我们设计了一个内态空间重建模块（ISSR），它通过联合学习长程空间特征和大核短程信息聚合，增强组织结构并减少伪影。 在包括内窥镜检查和白内障手术在内的四个不同医学场景的数据集上进行的实验表明，MedVSR在重构性能和效率方面显著优于现有的VSR模型。代码可在上述链接中获取。
URL
https://arxiv.org/abs/2509.21265
PDF
-
Surgical Video Understanding with Label Interpolation
2025-09-23 08:49:07Garam Kim, Tae Kyeong Jeong, Juyoun ParkAbstract
Robot-assisted surgery (RAS) has become a critical paradigm in modern surgery, promoting patient recovery and reducing the burden on surgeons through minimally invasive approaches. To fully realize its potential, however, a precise understanding of the visual data generated during surgical procedures is essential. Previous studies have predominantly focused on single-task approaches, but real surgical scenes involve complex temporal dynamics and diverse instrument interactions that limit comprehensive understanding. Moreover, the effective application of multi-task learning (MTL) requires sufficient pixel-level segmentation data, which are difficult to obtain due to the high cost and expertise required for annotation. In particular, long-term annotations such as phases and steps are available for every frame, whereas short-term annotations such as surgical instrument segmentation and action detection are provided only for key frames, resulting in a significant temporal-spatial imbalance. To address these challenges, we propose a novel framework that combines optical flow-based segmentation label interpolation with multi-task learning. optical flow estimated from annotated key frames is used to propagate labels to adjacent unlabeled frames, thereby enriching sparse spatial supervision and balancing temporal and spatial information for training. This integration improves both the accuracy and efficiency of surgical scene understanding and, in turn, enhances the utility of RAS.
Abstract (translated)
机器人辅助手术（RAS）已成为现代外科手术中的关键范式，通过微创方法促进患者康复并减轻外科医生的负担。然而，为了充分发挥其潜力，准确理解手术过程中生成的视觉数据至关重要。以往的研究主要集中在单一任务的方法上，但实际情况中手术场景涉及复杂的时序动态和多样化的器械交互，这限制了对其全面的理解。此外，多任务学习（MTL）的有效应用需要充足的像素级分割数据，而由于标注所需的高昂成本和专业知识，这些数据难以获取。特别是，长时间的注释如阶段和步骤为每一帧提供，而短期注释如手术器械分割和动作检测仅针对关键帧给出，导致显著的时间-空间不平衡。 为了应对这些挑战，我们提出了一种新的框架，该框架结合了基于光学流的分段标签插值与多任务学习。通过从标注的关键帧中估计出的光学流，可以将标签传播到相邻未标记的帧上，从而丰富稀疏的空间监督并平衡训练中的时间和空间信息。这种集成不仅提高了手术场景理解的准确性和效率，而且增强了RAS的应用价值。
URL
https://arxiv.org/abs/2509.18802
PDF
-
Interpretable Modeling of Articulatory Temporal Dynamics from real-time MRI for Phoneme Recognition
2025-09-19 07:04:44Jay Park, Hong Nguyen, Sean Foley, Jihwan Lee, Yoonjeong Lee, Dani Byrd, Shrikanth NarayananAbstract
Real-time Magnetic Resonance Imaging (rtMRI) visualizes vocal tract action, offering a comprehensive window into speech articulation. However, its signals are high dimensional and noisy, hindering interpretation. We investigate compact representations of spatiotemporal articulatory dynamics for phoneme recognition from midsagittal vocal tract rtMRI videos. We compare three feature types: (1) raw video, (2) optical flow, and (3) six linguistically-relevant regions of interest (ROIs) for articulator movements. We evaluate models trained independently on each representation, as well as multi-feature combinations. Results show that multi-feature models consistently outperform single-feature baselines, with the lowest phoneme error rate (PER) of 0.34 obtained by combining ROI and raw video. Temporal fidelity experiments demonstrate a reliance on fine-grained articulatory dynamics, while ROI ablation studies reveal strong contributions from tongue and lips. Our findings highlight how rtMRI-derived features provide accuracy and interpretability, and establish strategies for leveraging articulatory data in speech processing.
Abstract (translated)
实时磁共振成像（rtMRI）能够可视化声带动作，为语音发音提供了一个全面的窗口。然而，其信号具有高维度和噪音的特点，这阻碍了对其的理解。我们研究了用于从中矢状面声带rtMRI视频中识别音素的时空构音动力学的紧凑表示形式。我们将三种类型的特征进行了比较：（1）原始视频、（2）光学流以及（3）六个与语言相关的感兴趣区域（ROI），这些区域代表发音器的动作。我们评估了在每种表示类型上单独训练的模型，以及多特征组合的效果。 实验结果表明，多特征模型始终优于单一特征基线，在将ROI和原始视频结合后实现了最低的音素错误率（PER）0.34。时间保真度实验展示了对精细构音动态的依赖性，而ROI消融研究揭示了舌头和嘴唇的贡献尤为显著。 我们的研究成果突显了从rtMRI衍生特征中获得准确性与可解释性的潜力，并为在语音处理中利用构音数据建立了策略。
URL
https://arxiv.org/abs/2509.15689
PDF
-
WorldForge: Unlocking Emergent 3D/4D Generation in Video Diffusion Model via Training-Free Guidance
2025-09-18 16:40:47Chenxi Song, Yanming Yang, Tong Zhao, Ruibo Li, Chi ZhangAbstract
Recent video diffusion models demonstrate strong potential in spatial intelligence tasks due to their rich latent world priors. However, this potential is hindered by their limited controllability and geometric inconsistency, creating a gap between their strong priors and their practical use in 3D/4D tasks. As a result, current approaches often rely on retraining or fine-tuning, which risks degrading pretrained knowledge and incurs high computational costs. To address this, we propose WorldForge, a training-free, inference-time framework composed of three tightly coupled modules. Intra-Step Recursive Refinement introduces a recursive refinement mechanism during inference, which repeatedly optimizes network predictions within each denoising step to enable precise trajectory injection. Flow-Gated Latent Fusion leverages optical flow similarity to decouple motion from appearance in the latent space and selectively inject trajectory guidance into motion-related channels. Dual-Path Self-Corrective Guidance compares guided and unguided denoising paths to adaptively correct trajectory drift caused by noisy or misaligned structural signals. Together, these components inject fine-grained, trajectory-aligned guidance without training, achieving both accurate motion control and photorealistic content generation. Extensive experiments across diverse benchmarks validate our method's superiority in realism, trajectory consistency, and visual fidelity. This work introduces a novel plug-and-play paradigm for controllable video synthesis, offering a new perspective on leveraging generative priors for spatial intelligence.
Abstract (translated)
最近的视频扩散模型在空间智能任务中展示了强大的潜力，这得益于它们丰富的潜在世界先验知识。然而，这种潜力受到其控制能力有限和几何不一致性的阻碍，导致了这些强大先验与三维或四维任务实际应用之间的差距。因此，目前的方法通常依赖于重新训练或微调，这会降低预训练的知识并产生高昂的计算成本。为了解决这个问题，我们提出了WorldForge，这是一个无需训练、仅在推理时使用的框架，由三个紧密耦合的模块组成。 1. **Intra-Step Recursive Refinement**（内部递归细化）：这种机制在推断过程中引入了重复优化网络预测的功能，在每个去噪步骤内启用精确轨迹注入。 2. **Flow-Gated Latent Fusion**（基于光流的潜在融合）：利用光流相似性来解耦潜在空间中的运动和外观，并选择性地将轨迹引导注入到与运动相关的通道中。 3. **Dual-Path Self-Corrective Guidance**（双路径自校正指导）：比较指导和未指导的去噪路径，以适应由噪声或对齐不良结构信号引起的轨迹漂移。 这些组件共同作用，无需训练即可实现细粒度、轨迹对齐的引导，从而同时实现了精确运动控制和逼真内容生成。在多种基准测试中的广泛实验验证了我们的方法在现实性、轨迹一致性和视觉保真度方面的优越性。这项工作引入了一种新的即插即用范式，用于可控视频合成，并为利用生成先验知识进行空间智能提供了全新的视角。
URL
https://arxiv.org/abs/2509.15130
PDF
-
BEV-ODOM2: Enhanced BEV-based Monocular Visual Odometry with PV-BEV Fusion and Dense Flow Supervision for Ground Robots
2025-09-18 05:29:32Yufei Wei, Wangtao Lu, Sha Lu, Chenxiao Hu, Fuzhang Han, Rong Xiong, Yue WangAbstract
Bird's-Eye-View (BEV) representation offers a metric-scaled planar workspace, facilitating the simplification of 6-DoF ego-motion to a more robust 3-DoF model for monocular visual odometry (MVO) in intelligent transportation systems. However, existing BEV methods suffer from sparse supervision signals and information loss during perspective-to-BEV projection. We present BEV-ODOM2, an enhanced framework addressing both limitations without additional annotations. Our approach introduces: (1) dense BEV optical flow supervision constructed from 3-DoF pose ground truth for pixel-level guidance; (2) PV-BEV fusion that computes correlation volumes before projection to preserve 6-DoF motion cues while maintaining scale consistency. The framework employs three supervision levels derived solely from pose data: dense BEV flow, 5-DoF for the PV branch, and final 3-DoF output. Enhanced rotation sampling further balances diverse motion patterns in training. Extensive evaluation on KITTI, NCLT, Oxford, and our newly collected ZJH-VO multi-scale dataset demonstrates state-of-the-art performance, achieving 40 improvement in RTE compared to previous BEV methods. The ZJH-VO dataset, covering diverse ground vehicle scenarios from underground parking to outdoor plazas, is publicly available to facilitate future research.
Abstract (translated)
鸟瞰视图（BEV）表示提供了一个度量比例的平面工作空间，有助于将6自由度的自我运动简化为一个更稳健的3自由度模型，从而用于智能交通系统中的单目视觉测距（MVO）。然而，现有的BEV方法由于稀疏的监督信号和透视到BEV投影过程中信息丢失的问题而受到限制。我们提出了BEV-ODOM2，这是一个改进的框架，它在没有额外标注的情况下解决了这些局限性。我们的方法引入了以下两项创新： 1. 从3自由度姿态真实值构建密集的BEV光流监督，为像素级别的指导提供支持。 2. PV-BEV融合计算投影前的相关体积，以保留6自由度运动线索的同时保持比例一致性。 该框架利用仅由姿态数据推导出的三种监督级别：密集的BEV流动、PV分支中的5自由度以及最终的3自由度输出。增强的旋转采样进一步平衡了训练过程中的各种运动模式。在KITTI、NCLT、Oxford和我们新收集的ZJH-VO多尺度数据集上的广泛评估显示，相比之前的BEV方法，在相对位移误差（RTE）上提高了40%的表现。 ZJH-VO数据集涵盖了从地下停车场到户外广场的各种地面车辆场景，并且公开可用以促进未来的研究。
URL
https://arxiv.org/abs/2509.14636
PDF
-
UM-Depth : Uncertainty Masked Self-Supervised Monocular Depth Estimation with Visual Odometry
2025-09-17 05:51:07Tae-Wook Um, Ki-Hyeon Kim, Hyun-Duck Choi, Hyo-Sung AhnAbstract
Monocular depth estimation has been increasingly adopted in robotics and autonomous driving for its ability to infer scene geometry from a single camera. In self-supervised monocular depth estimation frameworks, the network jointly generates and exploits depth and pose estimates during training, thereby eliminating the need for depth labels. However, these methods remain challenged by uncertainty in the input data, such as low-texture or dynamic regions, which can cause reduced depth accuracy. To address this, we introduce UM-Depth, a framework that combines motion- and uncertainty-aware refinement to enhance depth accuracy at dynamic object boundaries and in textureless regions. Specifically, we develop a teacherstudent training strategy that embeds uncertainty estimation into both the training pipeline and network architecture, thereby strengthening supervision where photometric signals are weak. Unlike prior motion-aware approaches that incur inference-time overhead and rely on additional labels or auxiliary networks for real-time generation, our method uses optical flow exclusively within the teacher network during training, which eliminating extra labeling demands and any runtime cost. Extensive experiments on the KITTI and Cityscapes datasets demonstrate the effectiveness of our uncertainty-aware refinement. Overall, UM-Depth achieves state-of-the-art results in both self-supervised depth and pose estimation on the KITTI datasets.
Abstract (translated)
单目深度估计因其从单一摄像头中推断场景几何结构的能力，已被机器人技术和自主驾驶领域广泛采用。在自监督的单目深度估计框架中，网络会同时生成和利用深度及姿态估算值来进行训练，从而无需依赖深度标签。然而，这些方法仍然受到输入数据不确定性的影响，如低纹理或动态区域会导致深度准确性降低的问题。为了解决这些问题，我们引入了UM-Depth框架，该框架结合了运动感知与不确定度感知的精炼技术，以提高在动态物体边界和无纹理区域中的深度准确性。 具体而言，我们开发了一种师生训练策略，将不确定性估算嵌入到训练流程及网络架构中，从而增强了光度信号较弱时的监督效果。不同于之前的基于运动的方法，在推理阶段会带来额外的计算开销，并依赖于辅助标签或网络进行实时生成，我们的方法仅在教师网络的训练过程中使用光学流技术，因此不需要额外标注和运行时成本。 我们在KITTI和Cityscapes数据集上进行了广泛实验，结果表明了我们不确定度感知精炼的有效性。总体而言，在无监督深度估计及姿态估计方面，UM-Depth在KITTI数据集中取得了最先进的性能。
URL
https://arxiv.org/abs/2509.13713
PDF
-
ROOM: A Physics-Based Continuum Robot Simulator for Photorealistic Medical Datasets Generation
2025-09-16 15:30:02Salvatore Esposito, Mat\'ias Mattamala, Daniel Rebain, Francis Xiatian Zhang, Kevin Dhaliwal, Mohsen Khadem, Subramanian RamamoorthyAbstract
Continuum robots are advancing bronchoscopy procedures by accessing complex lung airways and enabling targeted interventions. However, their development is limited by the lack of realistic training and test environments: Real data is difficult to collect due to ethical constraints and patient safety concerns, and developing autonomy algorithms requires realistic imaging and physical feedback. We present ROOM (Realistic Optical Observation in Medicine), a comprehensive simulation framework designed for generating photorealistic bronchoscopy training data. By leveraging patient CT scans, our pipeline renders multi-modal sensor data including RGB images with realistic noise and light specularities, metric depth maps, surface normals, optical flow and point clouds at medically relevant scales. We validate the data generated by ROOM in two canonical tasks for medical robotics -- multi-view pose estimation and monocular depth estimation, demonstrating diverse challenges that state-of-the-art methods must overcome to transfer to these medical settings. Furthermore, we show that the data produced by ROOM can be used to fine-tune existing depth estimation models to overcome these challenges, also enabling other downstream applications such as navigation. We expect that ROOM will enable large-scale data generation across diverse patient anatomies and procedural scenarios that are challenging to capture in clinical settings. Code and data: this https URL.
Abstract (translated)
连续机器人通过访问复杂的肺部气道并实现靶向干预，正在推进支气管镜检查程序。然而，其发展受限于缺乏现实的训练和测试环境：由于伦理限制和患者安全问题，收集真实数据较为困难；开发自主算法需要真实的成像及物理反馈。我们提出了ROOM（医学中的逼真光学观察），这是一个旨在生成逼真的支气管镜培训数据的全面模拟框架。通过利用患者的CT扫描，我们的流水线能够渲染包括带现实噪声和光斑点的RGB图像、度量深度图、表面法向量、光流及点云在内的多模态传感器数据，在医学相关的尺度下进行呈现。 我们用ROOM生成的数据在两个典型的医疗机器人任务——多视角姿态估计和单目深度估计中进行了验证，展示了最先进的方法必须克服的各种挑战才能转移到这些医疗环境中。此外，我们还表明，由ROOM产生的数据可以用于微调现有的深度估算模型以应对这些挑战，并促进其他下游应用如导航的实现。 我们预计，ROOM将能够生成大规模的数据，覆盖各种患者解剖结构和程序场景，在临床环境中难以捕捉。代码与数据：[此处提供具体链接]（请根据实际情况插入具体链接）。
URL
https://arxiv.org/abs/2509.13177
PDF
-
4DRadar-GS: Self-Supervised Dynamic Driving Scene Reconstruction with 4D Radar
2025-09-16 10:29:43Xiao Tang, Guirong Zhuo, Cong Wang, Boyuan Zheng, Minqing Huang, Lianqing Zheng, Long Chen, Shouyi LuAbstract
3D reconstruction and novel view synthesis are critical for validating autonomous driving systems and training advanced perception models. Recent self-supervised methods have gained significant attention due to their cost-effectiveness and enhanced generalization in scenarios where annotated bounding boxes are unavailable. However, existing approaches, which often rely on frequency-domain decoupling or optical flow, struggle to accurately reconstruct dynamic objects due to imprecise motion estimation and weak temporal consistency, resulting in incomplete or distorted representations of dynamic scene elements. To address these challenges, we propose 4DRadar-GS, a 4D Radar-augmented self-supervised 3D reconstruction framework tailored for dynamic driving scenes. Specifically, we first present a 4D Radar-assisted Gaussian initialization scheme that leverages 4D Radar's velocity and spatial information to segment dynamic objects and recover monocular depth scale, generating accurate Gaussian point representations. In addition, we propose a Velocity-guided PointTrack (VGPT) model, which is jointly trained with the reconstruction pipeline under scene flow supervision, to track fine-grained dynamic trajectories and construct temporally consistent representations. Evaluated on the OmniHD-Scenes dataset, 4DRadar-GS achieves state-of-the-art performance in dynamic driving scene 3D reconstruction.
Abstract (translated)
3D重建和新视角合成对于验证自动驾驶系统以及训练高级感知模型至关重要。最近的自监督方法由于其成本效益高且在缺乏标注边界框场景中的泛化能力增强，受到了广泛关注。然而，现有的大多数依赖频域解耦或光流的方法，在动态对象重建时往往因运动估计不精确和时间一致性弱而导致动态场景元素的表示不完整或扭曲。 为解决这些挑战，我们提出了4DRadar-GS框架，这是一个针对动态驾驶场景设计的、利用4D雷达增强的自监督3D重建方法。具体而言，首先我们提出了一种4D雷达辅助高斯初始化方案，该方案利用了4D雷达的速度和空间信息来分割动态对象并恢复单目深度尺度，从而生成精确的高斯点表示。 此外，我们还提出了一个速度引导的点追踪（Velocity-guided PointTrack，VGPT）模型。此模型与重建管道一起在场景流监督下联合训练，以跟踪精细的动态轨迹，并构造时间上一致的表示形式。 我们在OmniHD-Scenes数据集上对该方法进行了评估，结果表明4DRadar-GS在动态驾驶场景3D重建方面达到了最先进的性能水平。
URL
https://arxiv.org/abs/2509.12931
PDF
-
World Modeling with Probabilistic Structure Integration
2025-09-10 18:01:04Klemen Kotar, Wanhee Lee, Rahul Venkatesh, Honglin Chen, Daniel Bear, Jared Watrous, Simon Kim, Khai Loong Aw, Lilian Naing Chen, Stefan Stojanov, Kevin Feigelis, Imran Thobani, Alex Durango, Khaled Jedoui, Atlas Kazemian, Dan YaminsAbstract
We present Probabilistic Structure Integration (PSI), a system for learning richly controllable and flexibly promptable world models from data. PSI consists of a three-step cycle. The first step, Probabilistic prediction, involves building a probabilistic graphical model Psi of the data, in the form of a random-access autoregressive sequence model. Psi supports a complete set of learned conditional distributions describing the dependence of any variables in the data on any other set of variables. In step 2, Structure extraction, we show how to extract underlying low-dimensional properties in the data, corresponding to a diverse set of meaningful "intermediate structures", in a zero-shot fashion via causal inference on Psi. Step 3, Integration, completes the cycle by converting these structures into new token types that are then continually mixed back into the training diet as conditioning signals and prediction targets. Each such cycle augments the capabilities of Psi, both allowing it to model the underlying data better, and creating new control handles -- akin to an LLM-like universal prompting language. We train an instance of Psi on 1.4 trillion tokens of internet video data; we use it to perform a variety of useful video prediction and understanding inferences; we extract state-of-the-art optical flow, self-supervised depth and object segmentation; and we use these structures to support a full cycle of predictive improvements.
Abstract (translated)
我们提出了概率结构集成（PSI），这是一个从数据中学习丰富可控且灵活可提示的世界模型的系统。PSI由一个三步循环组成。 第一步，概率预测，涉及构建数据的概率图模型Psi，形式为随机访问自回归序列模型。Psi支持一组完整的学习条件分布，描述了数据中的任何变量对其他任意集合变量的依赖关系。 第二步，结构提取，在此步骤中我们展示了如何通过在Psi上进行因果推理来以零样本方式抽取数据中的底层低维属性，对应于一系列有意义的“中间结构”。 第三步，集成，该循环通过将这些结构转换为新的标记类型完成，然后将其不断混合回训练过程中作为条件信号和预测目标。每次这样的循环都会增强Psi的能力，使其更好地建模基础数据，并创建新的控制手柄——类似于类似大型语言模型（LLM）的通用提示语言。 我们在1.4万亿个互联网视频令牌的数据上对Psi的一个实例进行了训练；我们使用它来执行各种有用的视频预测和理解推理；提取最先进的光流、自我监督深度和对象分割技术；并利用这些结构支持整个预测改进循环。
URL
https://arxiv.org/abs/2509.09737
PDF
-
Computational Imaging for Enhanced Computer Vision
2025-09-10 16:02:42Humera Shaikh, Kaur JashanpreetAbstract
This paper presents a comprehensive survey of computational imaging (CI) techniques and their transformative impact on computer vision (CV) applications. Conventional imaging methods often fail to deliver high-fidelity visual data in challenging conditions, such as low light, motion blur, or high dynamic range scenes, thereby limiting the performance of state-of-the-art CV systems. Computational imaging techniques, including light field imaging, high dynamic range (HDR) imaging, deblurring, high-speed imaging, and glare mitigation, address these limitations by enhancing image acquisition and reconstruc- tion processes. This survey systematically explores the synergies between CI techniques and core CV tasks, including object detection, depth estimation, optical flow, face recognition, and keypoint detection. By analyzing the relationships between CI methods and their practical contributions to CV applications, this work highlights emerging opportunities, challenges, and future research directions. We emphasize the potential for task-specific, adaptive imaging pipelines that improve robustness, accuracy, and efficiency in real-world scenarios, such as autonomous navigation, surveillance, augmented reality, and robotics.
Abstract (translated)
本文全面回顾了计算成像（CI）技术及其对计算机视觉（CV）应用的变革性影响。传统成像方法在诸如低光、运动模糊或高动态范围场景等挑战条件下，往往无法提供高质量的视觉数据，从而限制了当前最先进的CV系统的表现能力。计算成像技术，包括光场成像、高动态范围（HDR）成像、去模糊处理、高速成像和消除眩光等方法，在增强图像采集和重建过程中解决了这些问题。 本文系统地探讨了CI技术和核心CV任务之间的协同效应，这些任务包括物体检测、深度估计、光学流分析、人脸识别以及关键点检测。通过分析CI方法与其在实际CV应用中的贡献关系，本研究强调了新兴的机会、挑战及未来的研究方向。我们特别关注特定任务的自适应成像管道的潜力，这将提高真实场景（如自主导航、监控、增强现实和机器人技术）中系统的鲁棒性、准确性和效率。
URL
https://arxiv.org/abs/2509.08712
PDF
-
FractalPINN-Flow: A Fractal-Inspired Network for Unsupervised Optical Flow Estimation with Total Variation Regularization
2025-09-10 15:05:51Sara Behnamian, Rasoul Khaksarinezhad, Andreas LangerAbstract
We present FractalPINN-Flow, an unsupervised deep learning framework for dense optical flow estimation that learns directly from consecutive grayscale frames without requiring ground truth. The architecture centers on the Fractal Deformation Network (FDN) - a recursive encoder-decoder inspired by fractal geometry and self-similarity. Unlike traditional CNNs with sequential downsampling, FDN uses repeated encoder-decoder nesting with skip connections to capture both fine-grained details and long-range motion patterns. The training objective is based on a classical variational formulation using total variation (TV) regularization. Specifically, we minimize an energy functional that combines $L^1$ and $L^2$ data fidelity terms to enforce brightness constancy, along with a TV term that promotes spatial smoothness and coherent flow fields. Experiments on synthetic and benchmark datasets show that FractalPINN-Flow produces accurate, smooth, and edge-preserving optical flow fields. The model is especially effective for high-resolution data and scenarios with limited annotations.
Abstract (translated)
我们提出了一种无监督深度学习框架FractalPINN-Flow，用于从连续的灰度帧中直接估计稠密光流，而无需依赖真实标签。该架构的核心是分形变形网络（FDN），这是一种受分形几何和自相似性启发的递归编码器-解码器结构。与传统的使用顺序下采样的CNN不同，FDN通过重复的编码器-解码器嵌套以及跳跃连接来捕捉细粒度细节和长距离运动模式。 训练目标基于经典的变分方法，该方法采用了总变化（TV）正则化。具体而言，我们最小化一个能量泛函，该函数结合了$L^1$和$L^2$数据保真项以强制执行亮度恒常性，并且加入了一个TV项来促进空间平滑度和一致的流场。 在合成数据集和基准数据集上的实验表明，FractalPINN-Flow能够生成准确、平滑并保持边缘特性的光流场。该模型对于高分辨率数据和标注有限的情景特别有效。
URL
https://arxiv.org/abs/2509.08670
PDF
-
Deep Visual Odometry for Stereo Event Cameras
2025-09-10 02:30:18Sheng Zhong, Junkai Niu, Yi ZhouAbstract
Event-based cameras are bio-inspired sensors with pixels that independently and asynchronously respond to brightness changes at microsecond resolution, offering the potential to handle state estimation tasks involving motion blur and high dynamic range (HDR) illumination conditions. However, the versatility of event-based visual odometry (VO) relying on handcrafted data association (either direct or indirect methods) is still unreliable, especially in field robot applications under low-light HDR conditions, where the dynamic range can be enormous and the signal-to-noise ratio is spatially-and-temporally varying. Leveraging deep neural networks offers new possibilities for overcoming these challenges. In this paper, we propose a learning-based stereo event visual odometry. Building upon Deep Event Visual Odometry (DEVO), our system (called Stereo-DEVO) introduces a novel and efficient static-stereo association strategy for sparse depth estimation with almost no additional computational burden. By integrating it into a tightly coupled bundle adjustment (BA) optimization scheme, and benefiting from the recurrent network's ability to perform accurate optical flow estimation through voxel-based event representations to establish reliable patch associations, our system achieves high-precision pose estimation in metric scale. In contrast to the offline performance of DEVO, our system can process event data of \zs{Video Graphics Array} (VGA) resolution in real time. Extensive evaluations on multiple public real-world datasets and self-collected data justify our system's versatility, demonstrating superior performance compared to state-of-the-art event-based VO methods. More importantly, our system achieves stable pose estimation even in large-scale nighttime HDR scenarios.
Abstract (translated)
基于事件的相机是受生物启发的传感器，其像素可以在微秒级分辨率下独立且异步地响应亮度变化，这为处理包含运动模糊和高动态范围（HDR）照明条件的状态估计任务提供了潜力。然而，依赖于手工制作的数据关联（直接或间接方法）的基于事件的视觉测距（VO）在低光HDR现场机器人应用中的可靠性仍然不足，因为在这种情况下，动态范围可能非常大且信噪比会随时间和空间变化而波动。利用深度神经网络为克服这些挑战提供了新的可能性。 本文中，我们提出了一种学习型立体事件视觉测距方法。基于深度事件视觉测距（DEVO），我们的系统（称为Stereo-DEVO）引入了一个新颖且高效的静态立体关联策略，用于稀疏深度估计，并几乎无需额外的计算负担。通过将其整合到紧密耦合的整体调整优化方案中，并利用递归网络的能力通过对体素表示的事件进行准确的光流估算来建立可靠的补丁关联，我们的系统能够在度量尺度上实现高精度的姿态估计。与DEVO的离线性能相比，我们的系统可以实时处理VGA分辨率的事件数据。 在多个公共真实世界的数据集和自采集数据上的广泛评估证明了我们系统的多功能性，并展示了优于现有最先进的基于事件的VO方法的卓越性能。更重要的是，我们的系统即使在大规模夜间HDR场景中也能实现稳定的姿态估计。
URL
https://arxiv.org/abs/2509.08235
PDF
