Optical_Flow
-
MIORe & VAR-MIORe: Benchmarks to Push the Boundaries of Restoration
2025-09-08 15:34:31George Ciubotariu, Zhuyun Zhou, Zongwei Wu, Radu TimofteAbstract
We introduce MIORe and VAR-MIORe, two novel multi-task datasets that address critical limitations in current motion restoration benchmarks. Designed with high-frame-rate (1000 FPS) acquisition and professional-grade optics, our datasets capture a broad spectrum of motion scenarios, which include complex ego-camera movements, dynamic multi-subject interactions, and depth-dependent blur effects. By adaptively averaging frames based on computed optical flow metrics, MIORe generates consistent motion blur, and preserves sharp inputs for video frame interpolation and optical flow estimation. VAR-MIORe further extends by spanning a variable range of motion magnitudes, from minimal to extreme, establishing the first benchmark to offer explicit control over motion amplitude. We provide high-resolution, scalable ground truths that challenge existing algorithms under both controlled and adverse conditions, paving the way for next-generation research of various image and video restoration tasks.
Abstract (translated)
我们介绍了两个新颖的多任务数据集MIORe和VAR-MIORe，这两个数据集旨在解决当前运动恢复基准测试中的关键局限性。我们的数据集采用高帧率（1000 FPS）采集技术和专业级光学设备设计，能够捕捉广泛的运动场景，包括复杂的自拍相机移动、动态的多主体交互以及依赖于深度的模糊效果。 通过根据计算出的光流指标自适应地平均帧，MIORe生成了一致的运动模糊，并保留了清晰的输入数据，以用于视频帧插值和光流估计。VAR-MIORe进一步扩展，涵盖了从最小到极端的不同范围的运动幅度，从而建立了第一个能够提供对运动幅度明确控制的基准。 我们提供了高分辨率、可扩展的真实数据，在受控条件和不利条件下都挑战现有的算法，为各种图像和视频恢复任务的下一代研究铺平了道路。
URL
https://arxiv.org/abs/2509.06803
PDF
-
Motion Aware ViT-based Framework for Monocular 6-DoF Spacecraft Pose Estimation
2025-09-07 10:15:55Jose Sosa, Dan Pineau, Arunkumar Rathinam, Abdelrahman Shabayek, Djamila AouadaAbstract
Monocular 6-DoF pose estimation plays an important role in multiple spacecraft missions. Most existing pose estimation approaches rely on single images with static keypoint localisation, failing to exploit valuable temporal information inherent to space operations. In this work, we adapt a deep learning framework from human pose estimation to the spacecraft pose estimation domain that integrates motion-aware heatmaps and optical flow to capture motion dynamics. Our approach combines image features from a Vision Transformer (ViT) encoder with motion cues from a pre-trained optical flow model to localise 2D keypoints. Using the estimates, a Perspective-n-Point (PnP) solver recovers 6-DoF poses from known 2D-3D correspondences. We train and evaluate our method on the SPADES-RGB dataset and further assess its generalisation on real and synthetic data from the SPARK-2024 dataset. Overall, our approach demonstrates improved performance over single-image baselines in both 2D keypoint localisation and 6-DoF pose estimation. Furthermore, it shows promising generalisation capabilities when testing on different data distributions.
Abstract (translated)
单目六自由度姿态估计在多航天器任务中扮演着重要角色。大多数现有的姿态估计算法依赖于仅包含静态关键点定位的单一图像，未能充分利用空间操作固有的宝贵时间信息。在这项工作中，我们将一个源自人体姿态估计算法的深度学习框架进行了改进，并将其应用于航天器的姿态估计领域，该框架集成了运动感知热图和光学流以捕捉运动动力学。我们的方法结合了来自视觉变压器（ViT）编码器的图像特征与预训练光学流模型提供的运动线索来定位2D关键点。利用这些估算结果，透视-n-点(PnP)求解器可以从已知的二维三维对应关系中恢复出六自由度姿态。我们在SPADES-RGB数据集上对我们的方法进行了训练和评估，并进一步在来自SPARK-2024的数据集中真实和合成数据上的泛化能力进行了测试。总体而言，与单图像基线相比，我们提出的方法在二维关键点定位和六自由度姿态估计方面都表现出更优的性能；同时，它还展示了在不同数据分布下的良好泛化能力。
URL
https://arxiv.org/abs/2509.06000
PDF
-
FlowSeek: Optical Flow Made Easier with Depth Foundation Models and Motion Bases
Abstract
We present FlowSeek, a novel framework for optical flow requiring minimal hardware resources for training. FlowSeek marries the latest advances on the design space of optical flow networks with cutting-edge single-image depth foundation models and classical low-dimensional motion parametrization, implementing a compact, yet accurate architecture. FlowSeek is trained on a single consumer-grade GPU, a hardware budget about 8x lower compared to most recent methods, and still achieves superior cross-dataset generalization on Sintel Final and KITTI, with a relative improvement of 10 and 15% over the previous state-of-the-art SEA-RAFT, as well as on Spring and LayeredFlow datasets.
Abstract (translated)
我们介绍了FlowSeek，这是一个新颖的光流框架，它在训练时所需的硬件资源非常有限。FlowSeek将最新的光学流网络设计空间进展与最先进的单图像深度基础模型以及经典的低维运动参数化相结合，实现了一个既紧凑又准确的架构。FlowSeek可以在单一消费级GPU上进行训练，其硬件预算比最近大多数方法低约8倍，但在Sintel Final和KITTI数据集上的跨数据集泛化性能仍然优于之前最先进的SEA-RAFT方法，相对改进分别达到10%和15%，并且在Spring和LayeredFlow数据集上也表现出色。
URL
https://arxiv.org/abs/2509.05297
PDF
-
A biologically inspired separable learning vision model for real-time traffic object perception in Dark
2025-09-05 11:22:52Hulin Li, Qiliang Ren, Jun Li, Hanbing Wei, Zheng Liu, Linfang FanAbstract
Fast and accurate object perception in low-light traffic scenes has attracted increasing attention. However, due to severe illumination degradation and the lack of reliable visual cues, existing perception models and methods struggle to quickly adapt to and accurately predict in low-light environments. Moreover, there is the absence of available large-scale benchmark specifically focused on low-light traffic scenes. To bridge this gap, we introduce a physically grounded illumination degradation method tailored to real-world low-light settings and construct Dark-traffic, the largest densely annotated dataset to date for low-light traffic scenes, supporting object detection, instance segmentation, and optical flow estimation. We further propose the Separable Learning Vision Model (SLVM), a biologically inspired framework designed to enhance perception under adverse lighting. SLVM integrates four key components: a light-adaptive pupillary mechanism for illumination-sensitive feature extraction, a feature-level separable learning strategy for efficient representation, task-specific decoupled branches for multi-task separable learning, and a spatial misalignment-aware fusion module for precise multi-feature alignment. Extensive experiments demonstrate that SLVM achieves state-of-the-art performance with reduced computational overhead. Notably, it outperforms RT-DETR by 11.2 percentage points in detection, YOLOv12 by 6.1 percentage points in instance segmentation, and reduces endpoint error (EPE) of baseline by 12.37% on Dark-traffic. On the LIS benchmark, the end-to-end trained SLVM surpasses Swin Transformer+EnlightenGAN and ConvNeXt-T+EnlightenGAN by an average of 11 percentage points across key metrics, and exceeds Mask RCNN (with light enhancement) by 3.1 percentage points. The Dark-traffic dataset and complete code is released at this https URL.
Abstract (translated)
在低光照交通场景中，快速而准确的对象感知吸引了越来越多的关注。然而，由于严重的照明退化和缺乏可靠视觉线索，现有的感知模型和方法难以迅速适应并准确预测在这种环境下的表现。此外，目前还没有专注于低光交通场景的大规模基准数据集可用。为了填补这一空白，我们引入了一种基于物理原理的光照退化方法，并针对现实世界的低光设置构建了Dark-traffic，这是迄今为止最大的密集标注的低光交通场景数据集，支持目标检测、实例分割和光流估计。我们进一步提出了分离学习视觉模型（SLVM），这是一种生物启发式的框架，旨在增强在恶劣照明条件下的感知能力。SLVM整合了四个关键组成部分：一种光照适应性的瞳孔机制，用于提取对光照敏感的特征；一种有效的表示策略——基于特征级别的可分离学习方法；针对特定任务解耦分支，进行多任务可分离学习；以及一种空间错位意识融合模块，实现精确的多特征对齐。广泛的实验表明，SLVM在保持较低计算开销的同时达到了最先进的性能。值得注意的是，在Dark-traffic数据集上，它比RT-DETR的目标检测性能高出11.2个百分点，比YOLOv12的实例分割性能高出6.1个百分点，并将基准线模型的端点误差（EPE）降低了12.37%。在LIS基准测试中，全卷积训练后的SLVM在关键指标上平均超过了Swin Transformer+EnlightenGAN和ConvNeXt-T+EnlightenGAN 11个百分点，比带光照增强功能的Mask RCNN高出3.1个百分点。Dark-traffic数据集及完整代码已在此网址发布。
URL
https://arxiv.org/abs/2509.05012
PDF
-
Motion-Refined DINOSAUR for Unsupervised Multi-Object Discovery
2025-09-02 17:44:49Xinrui Gong, Oliver Hahn, Christoph Reich, Krishnakant Singh, Simone Schaub-Meyer, Daniel Cremers, Stefan RothAbstract
Unsupervised multi-object discovery (MOD) aims to detect and localize distinct object instances in visual scenes without any form of human supervision. Recent approaches leverage object-centric learning (OCL) and motion cues from video to identify individual objects. However, these approaches use supervision to generate pseudo labels to train the OCL model. We address this limitation with MR-DINOSAUR -- Motion-Refined DINOSAUR -- a minimalistic unsupervised approach that extends the self-supervised pre-trained OCL model, DINOSAUR, to the task of unsupervised multi-object discovery. We generate high-quality unsupervised pseudo labels by retrieving video frames without camera motion for which we perform motion segmentation of unsupervised optical flow. We refine DINOSAUR's slot representations using these pseudo labels and train a slot deactivation module to assign slots to foreground and background. Despite its conceptual simplicity, MR-DINOSAUR achieves strong multi-object discovery results on the TRI-PD and KITTI datasets, outperforming the previous state of the art despite being fully unsupervised.
Abstract (translated)
无监督多对象发现（MOD）的目标是在没有任何形式的人类监督的情况下，检测和定位视觉场景中的不同物体实例。最近的方法利用了以对象为中心的学习（OCL）以及视频中运动线索来识别单独的物体。然而，这些方法使用监督信息生成伪标签来训练OCL模型。我们通过MR-DINOSAUR——即“Motion-Refined DINOSAUR”——解决了这一限制，这是一种极简主义的无监督方法，它将自我监督预训练的OCL模型DINOSAUR扩展到无监督多对象发现任务上。 我们通过检索没有相机运动的视频帧来生成高质量的无监督伪标签，并在此基础上执行无监督光流的运动分割。使用这些伪标签细化了DINOSAUR的槽位表示，训练了一个槽位去激活模块，以将槽位分配给前景和背景对象。尽管概念上简单，MR-DINOSAUR在TRI-PD和KITTI数据集上的多对象发现结果中表现出了强大的性能，并且在完全无监督的情况下超过了之前的最佳水平。
URL
https://arxiv.org/abs/2509.02545
PDF
-
Encoder-Only Image Registration
2025-08-30 10:45:39Xiang Chen, Renjiu Hu, Jinwei Zhang, Yuxi Zhang, Xinyao Yue, Min Liu, Yaonan Wang, Hang ZhangAbstract
Learning-based techniques have significantly improved the accuracy and speed of deformable image registration. However, challenges such as reducing computational complexity and handling large deformations persist. To address these challenges, we analyze how convolutional neural networks (ConvNets) influence registration performance using the Horn-Schunck optical flow equation. Supported by prior studies and our empirical experiments, we observe that ConvNets play two key roles in registration: linearizing local intensities and harmonizing global contrast variations. Based on these insights, we propose the Encoder-Only Image Registration (EOIR) framework, designed to achieve a better accuracy-efficiency trade-off. EOIR separates feature learning from flow estimation, employing only a 3-layer ConvNet for feature extraction and a set of 3-layer flow estimators to construct a Laplacian feature pyramid, progressively composing diffeomorphic deformations under a large-deformation model. Results on five datasets across different modalities and anatomical regions demonstrate EOIR's effectiveness, achieving superior accuracy-efficiency and accuracy-smoothness trade-offs. With comparable accuracy, EOIR provides better efficiency and smoothness, and vice versa. The source code of EOIR will be publicly available on this https URL.
Abstract (translated)
基于学习的技术显著提高了可变形图像配准的准确性和速度。然而，减少计算复杂度和处理大形变等问题仍然存在挑战。为了解决这些问题，我们分析了卷积神经网络（ConvNets）如何通过Horn-Schunck光流方程影响配准性能。基于先前的研究和我们的实验证据，我们观察到ConvNets在图像配准中扮演两个关键角色：线性化局部强度以及协调全局对比度变化。根据这些见解，我们提出了编码器仅有的图像配准（EOIR）框架，旨在实现更好的准确性和效率之间的权衡。EOIR将特征学习与流估计分离，使用一个三层的卷积网络进行特征提取，并采用一组三层流估算器来构建拉普拉斯特征金字塔，在大形变模型下逐步组合微分同胚变形。 在五种不同模式和解剖区域的数据集上的结果表明了EOIR的有效性，实现了更优的准确性和效率权衡以及准确性和平滑度之间的最佳平衡。与相当的准确性相比，EOIR提供更好的效率和平滑度；反之亦然。 EOIR的源代码将在[此链接](https://example.com)公开发布（请将URL替换为实际网站地址）。
URL
https://arxiv.org/abs/2509.00451
PDF
-
Observer Design for Optical Flow-Based Visual-Inertial Odometry with Almost-Global Convergence
2025-08-28 18:55:16Tarek Bouazza, Soulaimane Berkane, Minh-Duc Hua, Tarek HamelAbstract
This paper presents a novel cascaded observer architecture that combines optical flow and IMU measurements to perform continuous monocular visual-inertial odometry (VIO). The proposed solution estimates body-frame velocity and gravity direction simultaneously by fusing velocity direction information from optical flow measurements with gyro and accelerometer data. This fusion is achieved using a globally exponentially stable Riccati observer, which operates under persistently exciting translational motion conditions. The estimated gravity direction in the body frame is then employed, along with an optional magnetometer measurement, to design a complementary observer on $\mathbf{SO}(3)$ for attitude estimation. The resulting interconnected observer architecture is shown to be almost globally asymptotically stable. To extract the velocity direction from sparse optical flow data, a gradient descent algorithm is developed to solve a constrained minimization problem on the unit sphere. The effectiveness of the proposed algorithms is validated through simulation results.
Abstract (translated)
这篇论文提出了一种新颖的级联观测器架构，该架构结合了光流和惯性测量单元（IMU）的数据来执行连续单目视觉惯性里程计（VIO）。所提出的解决方案通过融合来自光流测量的速度方向信息与陀螺仪和加速度计数据，在同一个框架内同时估计机体坐标系中的速度和重力方向。这种融合是利用一个在持续激励平移运动条件下全局指数稳定的里卡蒂观测器来实现的。 然后，以机体系中估算出的重力方向为基础，并可选择性地结合磁力计测量值，设计了一个定义在$\mathbf{SO}(3)$上的互补观测器来进行姿态估计。所得到的互联观测器架构被证明是几乎全局渐近稳定的。 为了从稀疏光流数据中提取速度方向，开发了一种梯度下降算法来解决单位球面上的一个约束最小化问题。通过仿真结果验证了所提出的算法的有效性。
URL
https://arxiv.org/abs/2508.21163
PDF
-
AutoQ-VIS: Improving Unsupervised Video Instance Segmentation via Automatic Quality Assessment
2025-08-27 11:52:41Kaixuan Lu, Mehmet Onurcan Kaya, Dim P. PapadopoulosAbstract
Video Instance Segmentation (VIS) faces significant annotation challenges due to its dual requirements of pixel-level masks and temporal consistency labels. While recent unsupervised methods like VideoCutLER eliminate optical flow dependencies through synthetic data, they remain constrained by the synthetic-to-real domain gap. We present AutoQ-VIS, a novel unsupervised framework that bridges this gap through quality-guided self-training. Our approach establishes a closed-loop system between pseudo-label generation and automatic quality assessment, enabling progressive adaptation from synthetic to real videos. Experiments demonstrate state-of-the-art performance with 52.6 $\text{AP}_{50}$ on YouTubeVIS-2019 val set, surpassing the previous state-of-the-art VideoCutLER by 4.4$\%$, while requiring no human annotations. This demonstrates the viability of quality-aware self-training for unsupervised VIS. The source code of our method is available at this https URL.
Abstract (translated)
视频实例分割（VIS）面临着像素级掩码和时间一致性标签的双重标注挑战。尽管最近的一些无监督方法，如VideoCutLER，通过合成数据消除了对光流依赖的需求，但它们仍然受限于从合成域到真实域的差距。我们提出了AutoQ-VIS，这是一种新的无监督框架，通过质量导向的自我训练来弥合这一差距。我们的方法在伪标签生成和自动质量评估之间建立了闭循环系统，从而能够逐步适应从合成视频到真实视频的过程。实验表明，在YouTubeVIS-2019验证集上，AutoQ-VIS达到了52.6%的$\text{AP}_{50}$成绩，比之前的最佳方法VideoCutLER高出4.4%，且无需人工标注。这证明了质量感知自我训练在无监督视频实例分割中的可行性。我们的方法源代码可在以下网址获得：[https URL]。 （提示：请将上述URL部分的实际链接地址替换为正确的项目网址以获取完整信息）
URL
https://arxiv.org/abs/2508.19808
PDF
-
Context-aware Sparse Spatiotemporal Learning for Event-based Vision
2025-08-27 11:48:03Shenqi Wang, Guangzhi TangAbstract
Event-based camera has emerged as a promising paradigm for robot perception, offering advantages with high temporal resolution, high dynamic range, and robustness to motion blur. However, existing deep learning-based event processing methods often fail to fully leverage the sparse nature of event data, complicating their integration into resource-constrained edge applications. While neuromorphic computing provides an energy-efficient alternative, spiking neural networks struggle to match of performance of state-of-the-art models in complex event-based vision tasks, like object detection and optical flow. Moreover, achieving high activation sparsity in neural networks is still difficult and often demands careful manual tuning of sparsity-inducing loss terms. Here, we propose Context-aware Sparse Spatiotemporal Learning (CSSL), a novel framework that introduces context-aware thresholding to dynamically regulate neuron activations based on the input distribution, naturally reducing activation density without explicit sparsity constraints. Applied to event-based object detection and optical flow estimation, CSSL achieves comparable or superior performance to state-of-the-art methods while maintaining extremely high neuronal sparsity. Our experimental results highlight CSSL's crucial role in enabling efficient event-based vision for neuromorphic processing.
Abstract (translated)
基于事件的相机已成为机器人感知领域的一个有前景的范式，它提供了高时间分辨率、宽动态范围以及对运动模糊的强大鲁棒性。然而，现有的基于深度学习的事件处理方法往往未能充分利用事件数据的稀疏特性，从而使其难以在资源受限的应用中集成。虽然神经形态计算提供了一种节能替代方案，但脉冲神经网络（SNN）在复杂基于事件的视觉任务，如物体检测和光流估计上，仍难以达到当前最佳模型的表现水平。此外，在神经网络中实现高激活稀疏性仍然具有挑战性，并且通常需要对诱导稀疏性的损失项进行仔细的手动调整。 为此，我们提出了一种新颖的框架——感知上下文的稀疏时空学习（CSSL），该框架引入了基于输入分布动态调节神经元激活的感知上下文阈值法。这种方法自然减少了激活密度，而无需明确的稀疏约束条件。在基于事件的物体检测和光流估计的应用中，CSSL实现了与现有最佳方法相当或更优的表现水平，并且保持了极高的神经元稀疏性。 实验结果强调了CSSL在实现高效的基于事件视觉处理以支持神经形态计算中的关键作用。
URL
https://arxiv.org/abs/2508.19806
PDF
-
FlowVLA: Thinking in Motion with a Visual Chain of Thought
2025-08-25 17:59:21Zhide Zhong, Haodong Yan, Junfeng Li, Xiangchen Liu, Xin Gong, Wenxuan Song, Jiayi Chen, Haoang LiAbstract
Many Vision-Language-Action (VLA) models rely on an internal world model trained via next-frame prediction. This approach, however, struggles with physical reasoning as it entangles static appearance with dynamic motion, often resulting in implausible visual forecasts and inefficient policy learning. To address these limitations, we introduce the Visual Chain of Thought (Visual CoT): a pre-training framework that encourages a model to reason about how a scene evolves before predicting what it will look like. We instantiate this principle in FlowVLA, which predicts a future frame ($v_{t+1}$) only after generating an intermediate optical flow representation ($f_t$) that encodes motion dynamics. This ``$v_t \rightarrow f_t \rightarrow v_{t+1}$'' reasoning process is implemented within a single autoregressive Transformer, guiding the model to learn disentangled dynamics. As a result, FlowVLA produces coherent visual predictions and facilitates more efficient policy learning. Experiments on challenging robotics manipulation benchmarks demonstrate state-of-the-art performance with substantially improved sample efficiency, pointing toward a more principled foundation for world modeling. Project page: this https URL
Abstract (translated)
许多视觉-语言-行动（VLA）模型依赖于通过下一帧预测训练的内部世界模型。然而，这种做法在物理推理方面存在困难，因为它将静态外观与动态运动纠缠在一起，通常会导致不可信的视觉预测和低效的行为学习。为了克服这些限制，我们引入了视觉思维链（Visual CoT）：一个预训练框架，鼓励模型先推断场景如何演变，然后再预测其未来的样子。我们将这一原则在FlowVLA中实现，在生成编码运动动态的中间光流表示($f_t$)之后才预测未来的帧($v_{t+1}$)。这种"$v_t \rightarrow f_t \rightarrow v_{t+1}$"推理过程在一个单一的自回归Transformer内部实施，指导模型学习解耦的动力学。因此，FlowVLA能够生成连贯的视觉预测，并促进更高效的行为学习。 在具有挑战性的机器人操作基准测试中的实验表明，与现有方法相比，FlowVLA表现出卓越的性能和显著提高的样本效率，这指向了一个更为原则化的世界建模基础。项目页面：[此网址](https://this-url.com/)
URL
https://arxiv.org/abs/2508.18269
PDF
-
MESTI-MEGANet: Micro-expression Spatio-Temporal Image and Micro-expression Gradient Attention Networks for Micro-expression Recognition
2025-08-25 09:49:05Luu Tu Nguyen, Vu Tram Anh Khuong, Thanh Ha Le, Thi Duyen NgoAbstract
Micro-expression recognition (MER) is a challenging task due to the subtle and fleeting nature of micro-expressions. Traditional input modalities, such as Apex Frame, Optical Flow, and Dynamic Image, often fail to adequately capture these brief facial movements, resulting in suboptimal performance. In this study, we introduce the Micro-expression Spatio-Temporal Image (MESTI), a novel dynamic input modality that transforms a video sequence into a single image while preserving the essential characteristics of micro-movements. Additionally, we present the Micro-expression Gradient Attention Network (MEGANet), which incorporates a novel Gradient Attention block to enhance the extraction of fine-grained motion features from micro-expressions. By combining MESTI and MEGANet, we aim to establish a more effective approach to MER. Extensive experiments were conducted to evaluate the effectiveness of MESTI, comparing it with existing input modalities across three CNN architectures (VGG19, ResNet50, and EfficientNetB0). Moreover, we demonstrate that replacing the input of previously published MER networks with MESTI leads to consistent performance improvements. The performance of MEGANet, both with MESTI and Dynamic Image, is also evaluated, showing that our proposed network achieves state-of-the-art results on the CASMEII and SAMM datasets. The combination of MEGANet and MESTI achieves the highest accuracy reported to date, setting a new benchmark for micro-expression recognition. These findings underscore the potential of MESTI as a superior input modality and MEGANet as an advanced recognition network, paving the way for more effective MER systems in a variety of applications.
Abstract (translated)
微表情识别（MER）是一项具有挑战性的任务，原因在于微表情的微妙性和短暂性。传统的输入模式，如Apex Frame、光学流和动态图像，常常无法充分捕捉这些短暂的面部运动，从而导致性能不佳。在这项研究中，我们引入了一种新的动态输入模式——微表情时空图（MESTI），该模式能够将视频序列转换为单张图片，并保留了细微动作的关键特性。此外，我们还提出了微表情梯度注意力网络（MEGANet），它包含一个新颖的梯度注意力块来增强从微表情中提取细粒度运动特征的能力。 通过结合MESTI和MEGANet，我们的目标是建立一种更有效的MER方法。为了评估MESTI的有效性，我们在三种CNN架构（VGG19、ResNet50和EfficientNetB0）上进行了广泛的实验，并将其与现有的输入模式进行了比较。此外，我们还展示了用MESTI替换先前发布的MER网络的输入会导致性能的一致提高。我们也对MEGANet在使用MESTI和动态图像时的表现进行了评估，结果表明我们的提议网络在CASMEII和SAMM数据集上实现了最先进的成果。 结合了MEGANet与MESTI的方法取得了迄今为止报告的最高准确率，并为微表情识别设定了新的基准。这些发现强调了MESTI作为更优输入模式以及MEGANet作为高级别识别网络的潜力，为各种应用场景中的MER系统铺平了道路。
URL
https://arxiv.org/abs/2509.00056
PDF
-
6-DoF Object Tracking with Event-based Optical Flow and Frames
2025-08-20 15:22:51Zhichao Li, Arren Glover, Chiara Bartolozzi, Lorenzo NataleAbstract
Tracking the position and orientation of objects in space (i.e., in 6-DoF) in real time is a fundamental problem in robotics for environment interaction. It becomes more challenging when objects move at high-speed due to frame rate limitations in conventional cameras and motion blur. Event cameras are characterized by high temporal resolution, low latency and high dynamic range, that can potentially overcome the impacts of motion blur. Traditional RGB cameras provide rich visual information that is more suitable for the challenging task of single-shot object pose estimation. In this work, we propose using event-based optical flow combined with an RGB based global object pose estimator for 6-DoF pose tracking of objects at high-speed, exploiting the core advantages of both types of vision sensors. Specifically, we propose an event-based optical flow algorithm for object motion measurement to implement an object 6-DoF velocity tracker. By integrating the tracked object 6-DoF velocity with low frequency estimated pose from the global pose estimator, the method can track pose when objects move at high-speed. The proposed algorithm is tested and validated on both synthetic and real world data, demonstrating its effectiveness, especially in high-speed motion scenarios.
Abstract (translated)
在机器人与环境交互中，实时跟踪空间内物体的位置和姿态（即6自由度）是一项基本问题。当物体以高速移动时，由于传统摄像头的帧率限制和运动模糊效应，这一任务变得更加困难。事件相机因其高时间分辨率、低延迟和宽动态范围等特点，能够克服运动模糊的影响。传统的RGB相机则提供了丰富的视觉信息，更适合单次拍摄下的目标姿态估计这一挑战性任务。 在本研究中，我们提出了一种结合基于事件的光流与基于RGB的全局物体姿态估算器的方法，用于高速移动物体的6自由度姿态跟踪，从而充分利用了这两种视觉传感器的核心优势。具体来说，我们提出了一个基于事件的光流算法来测量目标运动，并实现了一个对象6自由度速度追踪器。通过将跟踪到的对象6自由度速度与来自全局姿态估计器的低频估计姿态相结合，该方法能够在高速移动时进行姿态追踪。 所提出的算法已在合成数据和真实世界数据上进行了测试和验证，展示了其在高速运动场景中的有效性和优势。
URL
https://arxiv.org/abs/2508.14776
PDF
-
Multiscale Video Transformers for Class Agnostic Segmentation in Autonomous Driving
2025-08-20 14:23:11Leila Cheshmi, Mennatullah SiamAbstract
Ensuring safety in autonomous driving is a complex challenge requiring handling unknown objects and unforeseen driving scenarios. We develop multiscale video transformers capable of detecting unknown objects using only motion cues. Video semantic and panoptic segmentation often relies on known classes seen during training, overlooking novel categories. Recent visual grounding with large language models is computationally expensive, especially for pixel-level output. We propose an efficient video transformer trained end-to-end for class-agnostic segmentation without optical flow. Our method uses multi-stage multiscale query-memory decoding and a scale-specific random drop-token to ensure efficiency and accuracy, maintaining detailed spatiotemporal features with a shared, learnable memory module. Unlike conventional decoders that compress features, our memory-centric design preserves high-resolution information at multiple scales. We evaluate on DAVIS'16, KITTI, and Cityscapes. Our method consistently outperforms multiscale baselines while being efficient in GPU memory and run-time, demonstrating a promising direction for real-time, robust dense prediction in safety-critical robotics.
Abstract (translated)
确保自动驾驶的安全是一项复杂的挑战，需要处理未知物体和不可预见的驾驶场景。我们开发了一种多尺度视频变压器，仅通过运动线索就能检测到未知对象。视频语义分割和全景分割通常依赖于训练期间见到的已知类别，忽略了新出现的类别。最近使用大型语言模型进行视觉定位在计算上成本高昂，尤其是在生成像素级输出时。为此，我们提出了一种高效端到端训练的视频变压器，用于无类别的分割，并且不需要光学流信息。我们的方法采用了多阶段多尺度查询-内存解码器和特定于比例尺的随机丢弃令牌策略，确保效率与准确性的同时还能保持详细的空间时间特征。通过一个共享、可学习的记忆模块，我们设计保留了不同分辨率下的高分辨率信息，这不同于传统的压缩特征的解码器。 我们在DAVIS'16、KITTI以及Cityscapes数据集上进行了评估，我们的方法在性能上始终优于多尺度基线模型，并且在GPU内存和运行时间方面表现高效。这一研究方向展示了实时、鲁棒密集预测在未来安全关键型机器人技术中的巨大潜力。
URL
https://arxiv.org/abs/2508.14729
PDF
-
Reliable Smoke Detection via Optical Flow-Guided Feature Fusion and Transformer-Based Uncertainty Modeling
2025-08-20 10:28:53Nitish Kumar Mahala, Muzammil Khan, Pushpendra KumarAbstract
Fire outbreaks pose critical threats to human life and infrastructure, necessitating high-fidelity early-warning systems that detect combustion precursors such as smoke. However, smoke plumes exhibit complex spatiotemporal dynamics influenced by illumination variability, flow kinematics, and environmental noise, undermining the reliability of traditional detectors. To address these challenges without the logistical complexity of multi-sensor arrays, we propose an information-fusion framework by integrating smoke feature representations extracted from monocular imagery. Specifically, a Two-Phase Uncertainty-Aware Shifted Windows Transformer for robust and reliable smoke detection, leveraging a novel smoke segmentation dataset, constructed via optical flow-based motion encoding, is proposed. The optical flow estimation is performed with a four-color-theorem-inspired dual-phase level-set fractional-order variational model, which preserves motion discontinuities. The resulting color-encoded optical flow maps are fused with appearance cues via a Gaussian Mixture Model to generate binary segmentation masks of the smoke regions. These fused representations are fed into the novel Shifted-Windows Transformer, which is augmented with a multi-scale uncertainty estimation head and trained under a two-phase learning regimen. First learning phase optimizes smoke detection accuracy, while during the second phase, the model learns to estimate plausibility confidence in its predictions by jointly modeling aleatoric and epistemic uncertainties. Extensive experiments using multiple evaluation metrics and comparative analysis with state-of-the-art approaches demonstrate superior generalization and robustness, offering a reliable solution for early fire detection in surveillance, industrial safety, and autonomous monitoring applications.
Abstract (translated)
火灾爆发对人类生命和基础设施构成重大威胁，需要高精度的早期预警系统来检测如烟雾等燃烧前兆。然而，烟羽表现出复杂的时空动态特性，受到照明变化、流体动力学以及环境噪声的影响，从而降低了传统探测器的可靠性。为了解决这些问题而不增加多传感器阵列的复杂性，我们提出了一种信息融合框架，通过将单目图像中的烟雾特征表示进行整合来实现这一目标。具体来说，我们提出了一个两阶段不确定性感知移位窗口变换模型（Two-Phase Uncertainty-Aware Shifted Windows Transformer），用于鲁棒且可靠的烟雾检测，并利用基于光流编码的新型烟雾分割数据集来构建该模型。光学流动估计使用一种受四色定理启发的双相水平集合分数阶变分模型完成，这种模型能够保留运动间的不连续性。 生成的颜色编码光流图通过高斯混合模型与外观线索相结合，生成烟雾区域的二值分割掩模。这些融合后的表示形式被输入到新的移位窗口变换器中，该变换器配备了多尺度不确定性估计头，并在两阶段学习方案下进行训练。第一阶段优化烟雾检测精度；第二阶段则通过共同建模随机不确定性和系统不确定性来学习对其预测的可能性信心。 使用多种评估指标及与现有最佳方法的比较分析显示，该方法具有优越的一般化能力和鲁棒性，在监控、工业安全和自主监测应用中早期火灾检测方面提供了一种可靠的解决方案。
URL
https://arxiv.org/abs/2508.14597
PDF
-
MF-LPR$^2$: Multi-Frame License Plate Image Restoration and Recognition using Optical Flow
2025-08-19 07:18:14Kihyun Na, Junseok Oh, Youngkwan Cho, Bumjin Kim, Sungmin Cho, Jinyoung Choi, Injung KimAbstract
License plate recognition (LPR) is important for traffic law enforcement, crime investigation, and surveillance. However, license plate areas in dash cam images often suffer from low resolution, motion blur, and glare, which make accurate recognition challenging. Existing generative models that rely on pretrained priors cannot reliably restore such poor-quality images, frequently introducing severe artifacts and distortions. To address this issue, we propose a novel multi-frame license plate restoration and recognition framework, MF-LPR$^2$, which addresses ambiguities in poor-quality images by aligning and aggregating neighboring frames instead of relying on pretrained knowledge. To achieve accurate frame alignment, we employ a state-of-the-art optical flow estimator in conjunction with carefully designed algorithms that detect and correct erroneous optical flow estimations by leveraging the spatio-temporal consistency inherent in license plate image sequences. Our approach enhances both image quality and recognition accuracy while preserving the evidential content of the input images. In addition, we constructed a novel Realistic LPR (RLPR) dataset to evaluate MF-LPR$^2$. The RLPR dataset contains 200 pairs of low-quality license plate image sequences and high-quality pseudo ground-truth images, reflecting the complexities of real-world scenarios. In experiments, MF-LPR$^2$ outperformed eight recent restoration models in terms of PSNR, SSIM, and LPIPS by significant margins. In recognition, MF-LPR$^2$ achieved an accuracy of 86.44%, outperforming both the best single-frame LPR (14.04%) and the multi-frame LPR (82.55%) among the eleven baseline models. The results of ablation studies confirm that our filtering and refinement algorithms significantly contribute to these improvements.
Abstract (translated)
车牌识别（LPR）对于交通执法、犯罪调查和监控至关重要。然而，行车记录仪图像中的车牌区域常常分辨率低、模糊不清且有眩光，这使得准确识别变得困难。现有的依赖于预训练先验的生成模型无法可靠地恢复这些质量较差的图像，经常引入严重的伪影和失真。为解决这一问题，我们提出了一种新颖的多帧车牌修复与识别框架——MF-LPR$^2$，该框架通过将相邻帧对齐并聚合而不是依赖于预训练知识来处理低质量图像中的模糊性。为了实现精确的帧对齐，我们采用最先进的光流估计器，并结合精心设计的算法利用车牌图像序列中固有的时空一致性检测和校正错误的光流估计。我们的方法在增强图像质量和识别准确度的同时保留了输入图像的证据内容。 此外，为评估MF-LPR$^2$，我们构建了一个新颖的真实车牌识别（RLPR）数据集。该RLPR数据集包含200对低质量车牌图像序列和高质量伪真实标签图像，反映了现实场景中的复杂性。在实验中，MF-LPR$^2$以显著的性能差距，在PSNR、SSIM和LPIPS等指标上超越了八种最新的修复模型。识别方面，MF-LPR$^2$达到了86.44%的准确率，优于十一种基线模型中的最佳单帧车牌识别（14.04%）和多帧车牌识别（82.55%）。消融研究结果确认我们的过滤和细化算法对此性能改进有显著贡献。
URL
https://arxiv.org/abs/2508.14797
PDF
-
Deformation of the panoramic sphere into an ellipsoid to induce self-motion in telepresence users
2025-08-18 13:48:15Eetu Laukka, Evan G. Center, Timo Ojala, Steven M. LaValle, Matti PoukeAbstract
Mobile telepresence robots allow users to feel present and explore remote environments using technology. Traditionally, these systems are implemented using a camera onboard a mobile robot that can be controlled. Although high-immersion technologies, such as 360-degree cameras, can increase situational awareness and presence, they also introduce significant challenges. Additional processing and bandwidth requirements often result in latencies of up to seconds. The current delay with a 360-degree camera streaming over the internet makes real-time control of these systems difficult. Working with high-latency systems requires some form of assistance to the users. This study presents a novel way to utilize optical flow to create an illusion of self-motion to the user during the latency period between user sending motion commands to the robot and seeing the actual motion through the 360-camera stream. We find no significant benefit of using the self-motion illusion to performance or accuracy of controlling a telepresence robot with a latency of 500 ms, as measured by the task completion time and collisions into objects. Some evidence is shown that the method might increase virtual reality (VR) sickness, as measured by the simulator sickness questionnaire (SSQ). We conclude that further adjustments are necessary in order to render the method viable.
Abstract (translated)
移动远程呈现机器人允许用户通过技术在远程环境中感受存在并进行探索。传统上，这些系统使用可控制的车载摄像头来实现。尽管像360度摄像头这样的高沉浸感技术可以提高情境意识和存在感，但它们也会带来显著的技术挑战。额外的数据处理和带宽需求通常会导致延迟长达几秒。当前使用360度摄像头通过互联网流媒体传输时的延迟使得实时控制这些系统变得困难。与高延迟系统合作需要某种形式的支持以帮助用户。 本研究提出了一种新的方法，利用光学流动来在用户发送移动命令给机器人和看到实际运动之间的延迟期间为用户提供一种自我移动的错觉感。我们发现，在500毫秒延迟的情况下使用这种自我移动错觉对远程呈现机器人的控制性能或准确性（通过任务完成时间和碰撞次数衡量）并无显著益处。此外，研究显示这种方法可能会增加虚拟现实（VR）病状的可能性，这一点可以通过模拟器病状问卷（SSQ）来测量。 我们得出结论，在使该方法变得可行之前，需要进行进一步的调整。
URL
https://arxiv.org/abs/2508.12925
PDF
-
Cooperative Face Liveness Detection from Optical Flow
2025-08-14 16:13:22Artem Sokolov, Mikhail Nikitin, Anton KonushinAbstract
In this work, we proposed a novel cooperative video-based face liveness detection method based on a new user interaction scenario where participants are instructed to slowly move their frontal-oriented face closer to the camera. This controlled approaching face protocol, combined with optical flow analysis, represents the core innovation of our approach. By designing a system where users follow this specific movement pattern, we enable robust extraction of facial volume information through neural optical flow estimation, significantly improving discrimination between genuine faces and various presentation attacks (including printed photos, screen displays, masks, and video replays). Our method processes both the predicted optical flows and RGB frames through a neural classifier, effectively leveraging spatial-temporal features for more reliable liveness detection compared to passive methods.
Abstract (translated)
在这项工作中，我们提出了一种基于新型用户交互场景的创新合作视频面部活体检测方法。在这个新场景中，参与者被指示缓慢地将面向摄像头的脸部逐渐靠近镜头。这种受控接近脸部协议结合了光流分析，构成了我们方法的核心创新点。通过设计一个要求用户遵循特定移动模式的系统，我们能够利用神经网络光流估计技术稳健地提取面部体积信息，从而显著提升了真实人脸与各种展示攻击（包括打印照片、屏幕显示、面具和视频重播）之间的鉴别能力。 我们的方法处理预测的光流以及RGB帧，并通过一个神经分类器来分析这些数据。这种方法有效地利用了空间-时间特征，相比被动的方法而言，能够实现更可靠的活体检测。
URL
https://arxiv.org/abs/2508.10786
PDF
-
DiTVR: Zero-Shot Diffusion Transformer for Video Restoration
2025-08-11 09:54:45Sicheng Gao, Nancy Mehta, Zongwei Wu, Radu TimofteAbstract
Video restoration aims to reconstruct high quality video sequences from low quality inputs, addressing tasks such as super resolution, denoising, and deblurring. Traditional regression based methods often produce unrealistic details and require extensive paired datasets, while recent generative diffusion models face challenges in ensuring temporal consistency. We introduce DiTVR, a zero shot video restoration framework that couples a diffusion transformer with trajectory aware attention and a wavelet guided, flow consistent sampler. Unlike prior 3D convolutional or frame wise diffusion approaches, our attention mechanism aligns tokens along optical flow trajectories, with particular emphasis on vital layers that exhibit the highest sensitivity to temporal dynamics. A spatiotemporal neighbour cache dynamically selects relevant tokens based on motion correspondences across frames. The flow guided sampler injects data consistency only into low-frequency bands, preserving high frequency priors while accelerating convergence. DiTVR establishes a new zero shot state of the art on video restoration benchmarks, demonstrating superior temporal consistency and detail preservation while remaining robust to flow noise and occlusions.
Abstract (translated)
视频恢复的目标是从低质量的输入中重建高质量的视频序列，解决超分辨率、去噪和去模糊等问题。传统的回归方法通常会产生不真实的细节，并且需要大量的配对数据集，而最近的生成扩散模型在确保时间一致性方面面临挑战。我们引入了DiTVR（Diffusion Transformer with Trajectory-aware Video Restoration），这是一种零样本视频恢复框架，结合了一个具有轨迹感知注意力机制和小波引导、流一致采样的扩散变换器。 与之前的3D卷积或帧级扩散方法不同，我们的注意机制沿光学流动轨迹对齐令牌，并特别强调最敏感于时间动态的关键层。一个时空邻域缓存根据帧之间的运动对应关系动态选择相关令牌。受流引导的采样器仅在低频带中注入数据一致性，同时保留高频先验并加速收敛。 DiTVR在视频恢复基准测试上建立了新的零样本状态最佳表现，展示了卓越的时间一致性和细节保持能力，并且对流动噪声和遮挡具有鲁棒性。
URL
https://arxiv.org/abs/2508.07811
PDF
-
Fast Motion Estimation and Context-Aware Refinement for Efficient Bayer-Domain Video Vision
2025-08-08 03:55:19Haichao Wang, Xinyue Xi, Jiangtao Wen, Yuxing HanAbstract
The efficiency of video computer vision system remains a challenging task due to the high temporal redundancy inside a video. Existing works have been proposed for efficient vision computer vision. However, they do not fully reduce the temporal redundancy and neglect the front end computation overhead. In this paper, we propose an efficient video computer vision system. First, image signal processor is removed and Bayer-format data is directly fed into video computer vision models, thus saving the front end computation. Second, instead of optical flow models and video codecs, a fast block matching-based motion estimation algorithm is proposed specifically for efficient video computer vision, with a MV refinement module. To correct the error, context-aware block refinement network is introduced to refine regions with large error. To further balance the accuracy and efficiency, a frame selection strategy is employed. Experiments on multiple video computer vision tasks demonstrate that our method achieves significant acceleration with slight performance loss.
Abstract (translated)
视频计算机视觉系统的效率仍然是一个挑战，因为视频内部存在较高的时间冗余。尽管已有研究提出了一些高效的方法来解决这一问题，但它们并未完全消除时间冗余，并且忽视了前端计算的开销。在本文中，我们提出了一种高效的视频计算机视觉系统。首先，移除图像信号处理器，并直接将Bayer格式数据输入到视频计算机视觉模型中，从而节省前端计算资源。其次，我们提出一种基于快速块匹配的运动估计算法以取代现有的光学流模型和视频编解码器，同时引入了MV（运动向量）精炼模块。为了纠正错误，我们还引入了一种上下文感知的块精炼网络来修正误差较大的区域。为平衡准确性和效率，我们采用了帧选择策略。在多个视频计算机视觉任务上的实验表明，我们的方法能够在性能略有下降的情况下实现显著加速。
URL
https://arxiv.org/abs/2508.05990
PDF
-
Revealing Latent Information: A Physics-inspired Self-supervised Pre-training Framework for Noisy and Sparse Events
2025-08-07 15:38:36Lin Zhu, Ruonan Liu, Xiao Wang, Lizhi Wang, Hua HuangAbstract
Event camera, a novel neuromorphic vision sensor, records data with high temporal resolution and wide dynamic range, offering new possibilities for accurate visual representation in challenging scenarios. However, event data is inherently sparse and noisy, mainly reflecting brightness changes, which complicates effective feature extraction. To address this, we propose a self-supervised pre-training framework to fully reveal latent information in event data, including edge information and texture cues. Our framework consists of three stages: Difference-guided Masked Modeling, inspired by the event physical sampling process, reconstructs temporal intensity difference maps to extract enhanced information from raw event data. Backbone-fixed Feature Transition contrasts event and image features without updating the backbone to preserve representations learned from masked modeling and stabilizing their effect on contrastive learning. Focus-aimed Contrastive Learning updates the entire model to improve semantic discrimination by focusing on high-value regions. Extensive experiments show our framework is robust and consistently outperforms state-of-the-art methods on various downstream tasks, including object recognition, semantic segmentation, and optical flow estimation. The code and dataset are available at this https URL.
Abstract (translated)
事件相机是一种新型的神经形态视觉传感器，它以高时间分辨率和宽动态范围记录数据，在具有挑战性的场景中为准确的视觉表示提供了新的可能性。然而，由于事件数据本身稀疏且噪声较多，并主要反映亮度变化，这使得有效特征提取变得复杂。为了应对这一问题，我们提出了一种自我监督的预训练框架，旨在从事件数据中充分揭示潜在信息，包括边缘信息和纹理线索。 我们的框架包含三个阶段： 1. 差异导向的掩码建模（Difference-guided Masked Modeling），受事件物理采样过程启发，重建时间强度差异图以增强从原始事件数据提取的信息。 2. 背骨固定的特征转换（Backbone-fixed Feature Transition），对比事件和图像特征而不更新背骨部分，以此保持通过掩码建模所学到的表示，并稳定其对对比学习的影响。 3. 目标导向的对比性学习（Focus-aimed Contrastive Learning），通过专注于高价值区域来更新整个模型以提高语义判别能力。 广泛的实验表明，我们的框架具有鲁棒性，在各种下游任务上始终优于现有方法，包括物体识别、语义分割和光流估计。代码和数据集可在此网址获得（请将“https URL”替换为实际链接）。
URL
https://arxiv.org/abs/2508.05507
PDF
