Optical_Flow
-
A Modular Zero-Shot Pipeline for Accident Detection, Localization, and Classification in Traffic Surveillance Video
2026-04-05 05:18:19Amey Thakur, Sarvesh TaleleAbstract
We describe a zero-shot pipeline developed for the ACCIDENT @ CVPR 2026 challenge. The challenge requires predicting when, where, and what type of traffic accident occurs in surveillance video, without labeled real-world training data. Our method separates the problem into three independent modules. The first module localizes the collision in time by running peak detection on z-score normalized frame-difference signals. The second module finds the impact location by computing the weighted centroid of cumulative dense optical flow magnitude maps using the Farneback algorithm. The third module classifies collision type by measuring cosine similarity between CLIP image embeddings of frames near the detected peak and text embeddings built from multi-prompt natural language descriptions of each collision category. No domain-specific fine-tuning is involved; the pipeline processes each video using only pre-trained model weights. Our implementation is publicly available as a Kaggle notebook.
Abstract (translated)
URL
https://arxiv.org/abs/2604.09685
PDF
-
SBF: An Effective Representation to Augment Skeleton for Video-based Human Action Recognition
2026-04-04 04:41:08Zhuoxuan Peng, Yiyi Ding, Yang Lin, S. -H. Gary ChanAbstract
Many modern video-based human action recognition (HAR) approaches use 2D skeleton as the intermediate representation in their prediction pipelines. Despite overall encouraging results, these approaches still struggle in many common scenes, mainly because the skeleton does not capture critical action-related information pertaining to the depth of the joints, contour of the human body, and interaction between the human and objects. To address this, we propose an effective approach to augment skeleton with a representation capturing action-related information in the pipeline of HAR. The representation, termed Scale-Body-Flow (SBF), consists of three distinct components, namely a scale map volume given by the scale (and hence depth information) of each joint, a body map outlining the human subject, and a flow map indicating human-object interaction given by pixel-wise optical flow values. To predict SBF, we further present SFSNet, a novel segmentation network supervised by the skeleton and optical flow without extra annotation overhead beyond the existing skeleton extraction. Extensive experiments across different datasets demonstrate that our pipeline based on SBF and SFSNet achieves significantly higher HAR accuracy with similar compactness and efficiency as compared with the state-of-the-art skeleton-only approaches.
Abstract (translated)
许多现代基于视频的人体动作识别（HAR）方法在预测流程中使用2D骨架作为中间表示。尽管整体结果令人鼓舞，但这些方法在许多常见场景中仍面临挑战，主要原因在于骨架无法捕捉与关节深度、人体轮廓以及人与物体交互相关的关键动作信息。为解决这一问题，我们提出了一种有效方法，在HAR流程中通过一种能捕捉动作相关信息的表示来增强骨架。该表示称为尺度-体素-流（SBF），包含三个不同组件：由每个关节的尺度（因而包含深度信息）给出的尺度图体积、勾勒人体主体的体素图，以及通过像素级光流值表示的人-物交互流图。为预测SBF，我们进一步提出了SFSNet——一种通过骨架和光流进行监督的新型分割网络，无需在现有骨架提取之外增加额外标注开销。在不同数据集上的大量实验表明，基于SBF和SFSNet的流程相比最先进的纯骨架方法，在保持相似紧凑性和效率的同时，显著提升了HAR准确率。
URL
https://arxiv.org/abs/2604.03590
PDF
-
HyVGGT-VO: Tightly Coupled Hybrid Dense Visual Odometry with Feed-Forward Models
2026-04-02 14:35:59Junxiang Pan, Lipu Zhou, Baojie ChenAbstract
Dense visual odometry (VO), which provides pose estimation and dense 3D reconstruction, serves as the cornerstone for applications ranging from robotics to augmented reality. Recently, feed-forward models have demonstrated remarkable capabilities in dense mapping. However, when these models are used in dense visual SLAM systems, their heavy computational burden restricts them to yielding sparse pose outputs at keyframes while still failing to achieve real-time pose estimation. In contrast, traditional sparse methods provide high computational efficiency and high-frequency pose outputs, but lack the capability for dense reconstruction. To address these limitations, we propose HyVGGT-VO, a novel framework that combines the computational efficiency of sparse VO with the dense reconstruction capabilities of feed-forward models. To the best of our knowledge, this is the first work to tightly couple a traditional VO framework with VGGT, a state-of-the-art feed-forward model. Specifically, we design an adaptive hybrid tracking frontend that dynamically switches between traditional optical flow and the VGGT tracking head to ensure robustness. Furthermore, we introduce a hierarchical optimization framework that jointly refines VO poses and the scale of VGGT predictions to ensure global scale consistency. Our approach achieves an approximately 5x processing speedup compared to existing VGGT-based methods, while reducing the average trajectory error by 85% on the indoor EuRoC dataset and 12% on the outdoor KITTI benchmark. Our code will be publicly available upon acceptance. Project page: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.02107
PDF
-
MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction
2026-04-02 12:20:45Xilai Li, Weijun Jiang, Xiaosong Li, Yang Liu, Hongbin Wang, Tao Ye, Huafeng Li, Haishu TanAbstract
Infrared and visible video fusion combines the object saliency from infrared images with the texture details from visible images to produce semantically rich fusion results. However, most existing methods are designed for static image fusion and cannot effectively handle frame-to-frame motion in videos. Current video fusion methods improve temporal consistency by introducing interactions across frames, but they often require high computational cost. To mitigate these challenges, we propose MAVFusion, an end-to-end video fusion framework featuring a motion-aware sparse interaction mechanism that enhances efficiency while maintaining superior fusion quality. Specifically, we leverage optical flow to identify dynamic regions in multi-modal sequences, adaptively allocating computationally intensive cross-modal attention to these sparse areas to capture salient transitions and facilitate inter-modal information exchange. For static background regions, a lightweight weak interaction module is employed to maintain structural and appearance integrity. By decoupling the processing of dynamic and static regions, MAVFusion simultaneously preserves temporal consistency and fine-grained details while significantly accelerating inference. Extensive experiments demonstrate that MAVFusion achieves state-of-the-art performance on multiple infrared and visible video benchmarks, achieving a speed of 14.16\,FPS at $640 \times 480$ resolution. The source code will be available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.01958
PDF
-
PTC-Depth: Pose-Refined Monocular Depth Estimation with Temporal Consistency
2026-04-02 09:03:02Leezy Han, Seunggyu Kim, Dongseok Shim, Hyeonbeom LeeAbstract
Monocular depth estimation (MDE) has been widely adopted in the perception systems of autonomous vehicles and mobile robots. However, existing approaches often struggle to maintain temporal consistency in depth estimation across consecutive frames. This inconsistency not only causes jitter but can also lead to estimation failures when the depth range changes abruptly. To address these challenges, this paper proposes a consistency-aware monocular depth estimation framework that leverages wheel odometry from a mobile robot to achieve stable and coherent depth predictions over time. Specifically, we estimate camera pose and sparse depth from triangulation using optical flow between consecutive frames. The sparse depth estimates are used to update a recursive Bayesian estimate of the metric scale, which is then applied to rescale the relative depth predicted by a pre-trained depth estimation foundation model. The proposed method is evaluated on the KITTI, TartanAir, MS2, and our own dataset, demonstrating robust and accurate depth estimation performance.
Abstract (translated)
单目深度估计（MDE）已被广泛应用于自动驾驶车辆和移动机器人的感知系统中。然而，现有方法在连续帧间的深度估计中往往难以保持时序一致性。这种不一致性不仅会导致抖动，还可能在深度范围突变时引发估计失败。为应对这些挑战，本文提出了一种感知一致性的单目深度估计框架，该框架利用移动机器人的轮式里程计信息，实现随时间稳定且连贯的深度预测。具体而言，我们通过连续帧间的光流估计相机位姿，并利用三角测量获得稀疏深度。这些稀疏深度估计值用于更新度量尺度的递归贝叶斯估计，进而对预训练深度估计基础模型预测的相对深度进行重缩放。所提方法在KITTI、TartanAir、MS2及自建数据集上进行了评估，展现了 robust 且精确的深度估计性能。
URL
https://arxiv.org/abs/2604.01791
PDF
-
Setup-Independent Full Projector Compensation
2026-04-02 08:00:22Haibo Li, Qingyue Deng, Jijiang Li, Haibin Ling, Bingyao HuangAbstract
Projector compensation seeks to correct geometric and photometric distortions that occur when images are projected onto nonplanar or textured surfaces. However, most existing methods are highly setup-dependent, requiring fine-tuning or retraining whenever the surface, lighting, or projector-camera pose changes. Progress has been limited by two key challenges: (1) the absence of large, diverse training datasets and (2) existing geometric correction models are typically constrained by specific spatial setups; without further retraining or fine-tuning, they often fail to generalize directly to novel geometric configurations. We introduce SIComp, the first Setup-Independent framework for full projector Compensation, capable of generalizing to unseen setups without fine-tuning or retraining. To enable this, we construct a large-scale real-world dataset spanning 277 distinct projector-camera setups. SIComp adopts a co-adaptive design that decouples geometry and photometry: A carefully tailored optical flow module performs online geometric correction, while a novel photometric network handles photometric compensation. To further enhance robustness under varying illumination, we integrate intensity-varying surface priors into the network design. Extensive experiments demonstrate that SIComp consistently produces high-quality compensation across diverse unseen setups, substantially outperforming existing methods in terms of generalization ability and establishing the first generalizable solution to projector compensation. The code and dataset are available on our project page: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2604.01736
PDF
-
Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding
2026-04-02 06:29:53Yuheng Jiang, Yiwen Cai, Zihao Wang, Yize Wu, Sicheng Li, Zhuo Su, Shaohui Jiao, Lan XuAbstract
Volumetric video seeks to model dynamic scenes as temporally coherent 4D representations. While recent Gaussian-based approaches achieve impressive rendering fidelity, they primarily emphasize appearance but are largely agnostic to instance-level structure, limiting stable tracking and semantic reasoning in highly dynamic scenarios. In this paper, we present Director, a unified spatio-temporal Gaussian representation that jointly models human performance, high-fidelity rendering, and instance-level semantics. Our key insight is that embedding instance-consistent semantics naturally complements 4D modeling, enabling more accurate scene decomposition while supporting robust dynamic scene understanding. To this end, we leverage temporally aligned instance masks and sentence embeddings derived from Multimodal Large Language Models to supervise the learnable semantic features of each Gaussian via two MLP decoders, enabling language-aligned 4D representations and enforcing identity consistency over time. To enhance temporal stability, we bridge 2D optical flow with 4D Gaussians and finetune their motions, yielding reliable initialization and reducing drift. For the training, we further introduce a geometry-aware SDF constraints, along with regularization terms that enforces surface continuity, enhancing temporal coherence in dynamic foreground modeling. Experiments demonstrate that Director achieves temporally coherent 4D reconstructions while simultaneously enabling instance segmentation and open-vocabulary querying.
Abstract (translated)
体积视频旨在将动态场景建模为时间连贯的4D表示。尽管近期基于高斯的方法实现了令人印象深刻的渲染保真度，但它们主要侧重于外观表现，而在很大程度上忽略了实例级结构，这限制了在高度动态场景中的稳定追踪与语义推理。本文提出了一种名为Director的统一时空高斯表示，可联合建模人体动作、高保真渲染及实例级语义。我们的核心见解是：嵌入实例一致的语义能自然补充4D建模，从而在实现更精确场景分解的同时，支持稳健的动态场景理解。为此，我们利用多模态大语言模型生成的时间对齐实例掩码与句子嵌入，通过两个多层感知机解码器监督每个高斯点的可学习语义特征，实现语言对齐的4D表示并确保时间上身份一致性。为增强时间稳定性，我们将2D光流与4D高斯相连接并微调其运动，从而获得可靠的初始化并减少漂移。训练中，我们进一步引入几何感知的符号距离场约束，以及强制表面连续性的正则项，以提升动态前景建模的时间连贯性。实验表明，Director在实现时间连贯4D重建的同时，还能支持实例分割与开放词汇查询。
URL
https://arxiv.org/abs/2604.01678
PDF
-
DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Data
2026-04-02 06:12:38Wonjoon Jin, Jiyun Won, Janghyeok Han, Qi Dai, Chong Luo, Seung-Hwan Baek, Sunghyun ChoAbstract
Despite recent progress, video diffusion models still struggle to synthesize realistic videos involving highly dynamic motions or requiring fine-grained motion controllability. A central limitation lies in the scarcity of such examples in commonly used training datasets. To address this, we introduce DynaVid, a video synthesis framework that leverages synthetic motion data in training, which is represented as optical flow and rendered using computer graphics pipelines. This approach offers two key advantages. First, synthetic motion offers diverse motion patterns and precise control signals that are difficult to obtain from real data. Second, unlike rendered videos with artificial appearances, rendered optical flow encodes only motion and is decoupled from appearance, thereby preventing models from reproducing the unnatural look of synthetic videos. Building on this idea, DynaVid adopts a two-stage generation framework: a motion generator first synthesizes motion, and then a motion-guided video generator produces video frames conditioned on that motion. This decoupled formulation enables the model to learn dynamic motion patterns from synthetic data while preserving visual realism from real-world videos. We validate our framework on two challenging scenarios, vigorous human motion generation and extreme camera motion control, where existing datasets are particularly limited. Extensive experiments demonstrate that DynaVid improves the realism and controllability in dynamic motion generation and camera motion control.
Abstract (translated)
尽管近期取得了进展，视频扩散模型在合成涉及高度动态运动或需要细粒度运动可控性的逼真视频时仍面临挑战。其核心限制在于常用训练数据集中此类样本的匮乏。为此，我们提出了DynaVid——一个利用合成运动数据进行训练的视频合成框架，该数据以光流形式表示并通过计算机图形管线渲染。该方法具有两大优势：首先，合成运动能提供多样化的运动模式和精确的控制信号，而这些从真实数据中难以获取；其次，与外观人工的渲染视频不同，渲染光流仅编码运动信息且与外观解耦，从而避免模型复现合成视频的不自然外观。基于此思路，DynaVid采用两阶段生成框架：运动生成器先合成运动，随后运动引导的视频生成器以该运动为条件生成视频帧。这种解耦设计使模型能从合成数据中学习动态运动模式，同时保留真实视频的视觉真实性。我们在两个挑战性场景中验证了该框架——剧烈人体运动生成与极端摄像机运动控制（现有数据集在这些领域尤为匮乏）。大量实验表明，DynaVid提升了动态运动生成与摄像机运动控制的真实性与可控性。
URL
https://arxiv.org/abs/2604.01666
PDF
-
ReFlow: Self-correction Motion Learning for Dynamic Scene Reconstruction
2026-04-02 03:14:26Yanzhe Liang, Ruijie Zhu, Hanzhi Chang, Zhuoyuan Li, Jiahao Lu, Tianzhu ZhangAbstract
We present ReFlow, a unified framework for monocular dynamic scene reconstruction that learns 3D motion in a novel self-correction manner from raw video. Existing methods often suffer from incomplete scene initialization for dynamic regions, leading to unstable reconstruction and motion estimation, which often resorts to external dense motion guidance such as pre-computed optical flow to further stabilize and constrain the reconstruction of dynamic components. However, this introduces additional complexity and potential error propagation. To address these issues, ReFlow integrates a Complete Canonical Space Construction module for enhanced initialization of both static and dynamic regions, and a Separation-Based Dynamic Scene Modeling module that decouples static and dynamic components for targeted motion supervision. The core of ReFlow is a novel self-correction flow matching mechanism, consisting of Full Flow Matching to align 3D scene flow with time-varying 2D observations, and Camera Flow Matching to enforce multi-view consistency for static objects. Together, these modules enable robust and accurate dynamic scene reconstruction. Extensive experiments across diverse scenarios demonstrate that ReFlow achieves superior reconstruction quality and robustness, establishing a novel self-correction paradigm for monocular 4D reconstruction.
Abstract (translated)
我们提出ReFlow，一个用于单目动态场景重建的统一框架，它以新颖的自校正方式从原始视频中学习3D运动。现有方法常因动态区域场景初始化不完整而导致重建与运动估计不稳定，往往依赖外部稠密运动引导（如预计算光流）来进一步稳定和约束动态部件的重建。然而这引入了额外的复杂性和潜在误差传播。为应对这些问题，ReFlow集成了完整规范空间构建模块以增强静态与动态区域的初始化，并采用基于分离的动态场景建模模块解耦静态与动态组件以实现针对性运动监督。ReFlow的核心是一种新颖的自校正流匹配机制，包含全流匹配（将3D场景流与时变2D观测对齐）和相机流匹配（对静态物体施加多视图一致性）。这些模块共同实现了鲁棒准确的动态场景重建。在多样化场景上的大量实验表明，ReFlow达到了卓越的重建质量与鲁棒性，为单目4D重建建立了新颖的自校正范式。
URL
https://arxiv.org/abs/2604.01561
PDF
-
VADMamba++: Efficient Video Anomaly Detection via Hybrid Modeling in Grayscale Space
2026-04-01 01:14:31Jihao Lyu, Minghua Zhao, Jing Hu, Yifei Chen, Shuangli Du, Cheng ShiAbstract
VADMamba pioneered the introduction of Mamba to Video Anomaly Detection (VAD), achieving high accuracy and fast inference through hybrid proxy tasks. Nevertheless, its heavy reliance on optical flow as auxiliary input and inter-task fusion scoring constrains its applicability to a single proxy task. In this paper, we introduce VADMamba++, an efficient VAD method based on the Gray-to-RGB paradigm that enforces a Single-Channel to Three-Channel reconstruction mapping, designed for a single proxy task and operating without auxiliary inputs. This paradigm compels inferring color appearances from grayscale structures, allowing anomalies to be more effectively revealed through dual inconsistencies between structure and chromatic cues. Specifically, VADMamba++ reconstructs grayscale frames into the RGB space to simultaneously discriminate structural geometry and chromatic fidelity, thereby enhancing sensitivity to explicit visual anomalies. We further design a hybrid modeling backbone that integrates Mamba, CNN, and Transformer modules to capture diverse normal patterns while suppressing the appearance of anomalies. Furthermore, an intra-task fusion scoring strategy integrates explicit future-frame prediction errors with implicit quantized feature errors, further improving accuracy under a single task setting. Extensive experiments on three benchmark datasets demonstrate that VADMamba++ outperforms state-of-the-art methods while meeting performance and efficiency, especially under a strict single-task setting with only frame-level inputs.
Abstract (translated)
URL
https://arxiv.org/abs/2604.00360
PDF
-
FlowIt: Global Matching for Optical Flow with Confidence-Guided Refinement
2026-03-30 17:58:12Sadra Safadoust, Fabio Tosi, Matteo Poggi, Fatma G\"uneyAbstract
We present FlowIt, a novel architecture for optical flow estimation designed to robustly handle large pixel displacements. At its core, FlowIt leverages a hierarchical transformer architecture that captures extensive global context, enabling the model to effectively model long-range correspondences. To overcome the limitations of localized matching, we formulate the flow initialization as an optimal transport problem. This formulation yields a highly robust initial flow field, alongside explicitly derived occlusion and confidence maps. These cues are then seamlessly integrated into a guided refinement stage, where the network actively propagates reliable motion estimates from high-confidence regions into ambiguous, low-confidence areas. Extensive experiments across the Sintel, KITTI, Spring, and LayeredFlow datasets validate the efficacy of our approach. FlowIt achieves state-of-the-art results on the competitive Sintel and KITTI benchmarks, while simultaneously establishing new state-of-the-art cross-dataset zero-shot generalization performance on Sintel, Spring, and LayeredFlow.
Abstract (translated)
URL
https://arxiv.org/abs/2603.28759
PDF
-
RehearsalNeRF: Decoupling Intrinsic Neural Fields of Dynamic Illuminations for Scene Editing
2026-03-30 01:49:36Changyeon Won, Hyunjun Jung, Jungu Cho, Seonmi Park, Chi-Hoon Lee, Hae-Gon JeonAbstract
Although there has been significant progress in neural radiance fields, an issue on dynamic illumination changes still remains unsolved. Different from relevant works that parameterize time-variant/-invariant components in scenes, subjects' radiance is highly entangled with their own emitted radiance and lighting colors in spatio-temporal domain. In this paper, we present a new effective method to learn disentangled neural fields under the severe illumination changes, named RehearsalNeRF. Our key idea is to leverage scenes captured under stable lighting like rehearsal stages, easily taken before dynamic illumination occurs, to enforce geometric consistency between the different lighting conditions. In particular, RehearsalNeRF employs a learnable vector for lighting effects which represents illumination colors in a temporal dimension and is used to disentangle projected light colors from scene radiance. Furthermore, our RehearsalNeRF is also able to reconstruct the neural fields of dynamic objects by simply adopting off-the-shelf interactive masks. To decouple the dynamic objects, we propose a new regularization leveraging optical flow, which provides coarse supervision for the color disentanglement. We demonstrate the effectiveness of RehearsalNeRF by showing robust performances on novel view synthesis and scene editing under dynamic illumination conditions. Our source code and video datasets will be publicly available.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27948
PDF
-
Dual-View Optical Flow for 4D Micro-Expression Recognition - A Multi-Stream Fusion Attention Approach
2026-03-27 10:07:59Luu Tu Nguyen, Thi Bich Phuong Man, Vu Tram Anh Khuong, Thanh Ha Le, Thi Duyen NgoAbstract
Micro-expression recognition is vital for affective computing but remains challenging due to the extremely brief, low-intensity facial motions involved and the high-dimensional nature of 4D mesh data. To address these challenges, we introduce a dual-view optical flow approach that simplifies mesh processing by capturing each micro-expression sequence from two synchronized viewpoints and computing optical flow to represent motion. Our pipeline begins with view separation and sequence-wise face cropping to ensure spatial consistency, followed by automatic apex-frame detection based on peak motion intensity in both views. We decompose each sequence into onset-apex and apex-offset phases, extracting horizontal, vertical, and magnitude flow channels for each phase. These are fed into our Triple-Stream MicroAttNet, which employs a fusion attention module to adaptively weight modality-specific features and a squeeze-and-excitation block to enhance magnitude representations. Training uses focal loss to mitigate class imbalance and the Adam optimizer with early stopping. Evaluated on the multi-label 4DME dataset, comprising 24 subjects and five emotion categories, in the 4DMR IJCAI Workshop Challenge 2025, our method achieves a macro-UF1 score of 0.536, outperforming the official baseline by over 50\% and securing first place. Ablation studies confirm that both the fusion attention and SE components each contribute up to 3.6 points of UF1 gain. These results demonstrate that dual-view, phase-aware optical flow combined with multi-stream fusion yields a robust and interpretable solution for 4D micro-expression recognition.
Abstract (translated)
URL
https://arxiv.org/abs/2603.26849
PDF
-
ANVIL: Accelerator-Native Video Interpolation via Codec Motion Vector Priors
Abstract
Mobile displays refresh at 90-120 Hz, yet most video is encoded at 24-30 frames per second; real-time frame-rate doubling requires each synthesized frame within 33.3 ms on mobile neural processing units. We show that mainstream flow-based video frame interpolation faces three structural deployment barriers on mobile accelerators: spatial sampling operators exceed the frame budget or lack hardware support, iterative flow refinement collapses under 8-bit post-training quantization, and memory-bound operators dominate the inference graph. ANVIL addresses these barriers by reusing motion vectors already computed by the H.264 decoder to prealign input frames, removing learned optical flow, spatial sampling, and iterative accumulation from the accelerator graph. The remaining residual is refined by a convolution-dominated network whose inference graph is composed almost entirely of compute-bound operators. On a Snapdragon 8 Gen 3 device, ANVIL achieves 12.8 ms 1080p network inference in 8-bit integer precision; an open-source Android player sustains 28.4 ms median end-to-end latency per interpolated frame pair over 54,623 consecutively logged samples during 30-minute continuous playback. Per-operator causal analysis identifies quantized accumulation on recurrent flow states as a key mechanism behind integer quantization failure in iterative methods. The current design targets H.264 playback scenarios with decoder-exposed motion vectors.
Abstract (translated)
URL
https://arxiv.org/abs/2603.26835
PDF
-
MegaFlow: Zero-Shot Large Displacement Optical Flow
2026-03-26 17:59:51Dingxi Zhang, Fangjinhua Wang, Marc Pollefeys, Haofei XuAbstract
Accurate estimation of large displacement optical flow remains a critical challenge. Existing methods typically rely on iterative local search or/and domain-specific fine-tuning, which severely limits their performance in large displacement and zero-shot generalization scenarios. To overcome this, we introduce MegaFlow, a simple yet powerful model for zero-shot large displacement optical flow. Rather than relying on highly complex, task-specific architectural designs, MegaFlow adapts powerful pre-trained vision priors to produce temporally consistent motion fields. In particular, we formulate flow estimation as a global matching problem by leveraging pre-trained global Vision Transformer features, which naturally capture large displacements. This is followed by a few lightweight iterative refinements to further improve the sub-pixel accuracy. Extensive experiments demonstrate that MegaFlow achieves state-of-the-art zero-shot performance across multiple optical flow benchmarks. Moreover, our model also delivers highly competitive zero-shot performance on long-range point tracking benchmarks, demonstrating its robust transferability and suggesting a unified paradigm for generalizable motion estimation. Our project page is at: this https URL.
Abstract (translated)
大位移光流的精确估计仍是一项关键挑战。现有方法通常依赖迭代局部搜索和/或领域特定的微调，这严重限制了其在大位移和零样本泛化场景下的性能。为克服此问题，我们提出了MegaFlow——一个简单而强大的零样本大位移光流模型。MegaFlow不依赖高度复杂、任务特定的架构设计，而是通过适配强大的预训练视觉先验来生成时序一致的运动场。具体而言，我们利用预训练的全局视觉Transformer特征将光流估计建模为全局匹配问题，这些特征天然能够捕捉大位移；随后通过少量轻量级迭代优化进一步提升亚像素精度。大量实验表明，MegaFlow在多个光流基准测试中达到了最先进的零样本性能。此外，我们的模型在长程点跟踪基准测试中也展现出极具竞争力的零样本表现，体现了其强大的迁移能力，并暗示了可泛化运动估计的统一范式。项目页面地址：此https链接。
URL
https://arxiv.org/abs/2603.25739
PDF
-
LILAC: Language-Conditioned Object-Centric Optical Flow for Open-Loop Trajectory Generation
2026-03-26 14:21:22Motonari Kambara, Koki Seno, Tomoya Kaichi, Yanan Wang, Komei SugiuraAbstract
We address language-conditioned robotic manipulation using flow-based trajectory generation, which enables training on human and web videos of object manipulation and requires only minimal embodiment-specific data. This task is challenging, as object trajectory generation from pre-manipulation images and natural language instructions requires appropriate instruction-flow alignment. To tackle this challenge, we propose the flow-based Language Instruction-guided open-Loop ACtion generator (LILAC). This flow-based Vision-Language-Action model (VLA) generates object-centric 2D optical flow from an RGB image and a natural language instruction, and converts the flow into a 6-DoF manipulator trajectory. LILAC incorporates two key components: Semantic Alignment Loss, which strengthens language conditioning to generate instruction-aligned optical flow, and Prompt-Conditioned Cross-Modal Adapter, which aligns learned visual prompts with image and text features to provide rich cues for flow generation. Experimentally, our method outperformed existing approaches in generated flow quality across multiple benchmarks. Furthermore, in physical object manipulation experiments using free-form instructions, LILAC demonstrated a superior task success rate compared to existing methods. The project page is available at this https URL.
Abstract (translated)
我们采用基于流式的轨迹生成方法来解决语言条件机器人操作问题，该方法支持利用人类操作和网络视频进行训练，且仅需极少量的具身数据。由于需要根据操作前图像和自然语言指令生成物体轨迹，并确保指令与流场的对齐，该任务具有挑战性。为应对这一挑战，我们提出了基于流式的语言指令引导开环动作生成器（LILAC）。这一基于流式的视觉-语言-动作模型（VLA）可从RGB图像和自然语言指令中生成以物体为中心的二维光流，并将光流转换为6自由度机械臂轨迹。LILAC包含两个关键组件：语义对齐损失（用于强化语言条件以生成与指令对齐的光流）和提示条件跨模态适配器（用于将学习到的视觉提示与图像及文本特征对齐，为流场生成提供丰富线索）。实验表明，我们的方法在多个基准测试中生成的流场质量优于现有方法。此外，在使用自由形式指令的物理物体操作实验中，LILAC的任务成功率也显著高于现有方法。项目页面可通过此链接访问。
URL
https://arxiv.org/abs/2603.25481
PDF
-
MoRGS: Efficient Per-Gaussian Motion Reasoning for Streamable Dynamic 3D Scenes
2026-03-26 05:25:11Wonjoon Lee, Sungmin Woo, Donghyeong Kim, Jungho Lee, Sangheon Park, Sangyoun LeeAbstract
Online reconstruction of dynamic scenes aims to learn from streaming multi-view inputs under low-latency constraints. The fast training and real-time rendering capabilities of 3D Gaussian Splatting have made on-the-fly reconstruction practically feasible, enabling online 4D reconstruction. However, existing online approaches, despite their efficiency and visual quality, fail to learn per-Gaussian motion that reflects true scene dynamics. Without explicit motion cues, appearance and motion are optimized solely under photometric loss, causing per-Gaussian motion to chase pixel residuals rather than true 3D motion. To address this, we propose MoRGS, an efficient online per-Gaussian motion reasoning framework that explicitly models per-Gaussian motion to improve 4D reconstruction quality. Specifically, we leverage optical flow on a sparse set of key views as lightweight motion cues that regularize per-Gaussian motion beyond photometric supervision. To compensate for the sparsity of flow supervision, we learn a per-Gaussian motion offset field that reconciles discrepancies between projected 3D motion and observed flow across views and time. In addition, we introduce a per-Gaussian motion confidence that separates dynamic from static Gaussians and weights Gaussian attribute residual updates, thereby suppressing redundant motion in static regions for better temporal consistency and accelerating the modeling of large motions. Extensive experiments demonstrate that MoRGS achieves state-of-the-art reconstruction quality and motion fidelity among online methods, while maintaining streamable performance.
Abstract (translated)
在线动态场景重建旨在低延迟约束下从流式多视角输入中学习。3D高斯泼溅（3D Gaussian Splatting）的快速训练与实时渲染能力使即时重建成为可能，从而支持在线4D重建。然而，现有在线方法尽管具备效率与视觉质量优势，却未能学习反映真实场景动态的每高斯运动。缺乏显式运动线索时，外观与运动仅通过光度损失进行优化，导致每高斯运动追逐像素残差而非真实3D运动。为解决此问题，我们提出MoRGS——一种高效的在线每高斯运动推理框架，通过显式建模每高斯运动来提升4D重建质量。具体而言，我们利用稀疏关键视角上的光流作为轻量级运动线索，以超越光度监督来约束每高斯运动。为弥补流监督的稀疏性，我们学习每高斯运动偏移场，以协调跨视角与时间上投影3D运动与观测流之间的差异。此外，我们引入每高斯运动置信度，区分动态与静态高斯，并对高斯属性残差更新进行加权，从而抑制静态区域的冗余运动以获得更佳时间一致性，并加速大运动建模。大量实验表明，MoRGS在在线方法中实现了最先进的重建质量与运动保真度，同时保持流式处理性能。
URL
https://arxiv.org/abs/2603.25042
PDF
-
Relaxed Rigidity with Ray-based Grouping for Dynamic Gaussian Splatting
2026-03-26 03:38:38Junoh Leea, Junmyeong Lee, Yeon-Ji Song, Inhwan Bae, Jisu Shin, Hae-Gon Jeon, Jin-Hwa KimAbstract
The reconstruction of dynamic 3D scenes using 3D Gaussian Splatting has shown significant promise. A key challenge, however, remains in modeling realistic motion, as most methods fail to align the motion of Gaussians with real-world physical dynamics. This misalignment is particularly problematic for monocular video datasets, where failing to maintain coherent motion undermines local geometric structure, ultimately leading to degraded reconstruction quality. Consequently, many state-of-the-art approaches rely heavily on external priors, such as optical flow or 2D tracks, to enforce temporal coherence. In this work, we propose a novel method to explicitly preserve the local geometric structure of Gaussians across time in 4D scenes. Our core idea is to introduce a view-space ray grouping strategy that clusters Gaussians intersected by the same ray, considering only those whose $\alpha$-blending weights exceed a threshold. We then apply constraints to these groups to maintain a consistent spatial distribution, effectively preserving their local geometry. This approach enforces a more physically plausible motion model by ensuring that local geometry remains stable over time, eliminating the reliance on external guidance. We demonstrate the efficacy of our method by integrating it into two distinct baseline models. Extensive experiments on challenging monocular datasets show that our approach significantly outperforms existing methods, achieving superior temporal consistency and reconstruction quality.
Abstract (translated)
利用3D高斯泼溅技术重建动态3D场景已展现出显著前景。然而，一个关键挑战仍在于模拟逼真的运动——大多数方法未能使高斯元的运动与真实物理动力学保持一致。这种偏差在单目视频数据集中尤为突出，因为无法维持连贯的运动会破坏局部几何结构，最终导致重建质量下降。因此，许多先进方法严重依赖光流或二维轨迹等外部先验信息来强制实现时间一致性。本研究提出一种新方法，旨在4D场景中显式保持高斯元跨时间的局部几何结构。核心思路是引入视图空间射线分组策略：将同一射线相交的高斯元进行聚类，仅考虑α混合权重超过阈值的高斯元，随后对这些分组施加约束以维持稳定的空间分布，从而有效保留其局部几何形态。该方法通过确保局部几何随时间保持稳定，构建了更符合物理规律的运动模型，并消除了对外部引导的依赖。我们将该方法集成到两个不同的基线模型中验证其有效性，在具有挑战性的单目数据集上的大量实验表明，本方案显著优于现有方法，在时间一致性和重建质量两方面均达到更优表现。
URL
https://arxiv.org/abs/2603.24994
PDF
-
CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning
2026-03-25 06:37:12Hieu Hoang, Dung Trung Tran, Hong Nguyen, Nam-Phong NguyenAbstract
Online Action Detection (OAD) systems face two primary challenges: high computational cost and insufficient modeling of discriminative temporal dynamics against background motion. Adding optical flow could provides strong motion cues but it incurs significant computational overhead. We propose CAKE, a OAD Flow-based distillation framework to transfer motion knowledge into RGB models. We propose Dynamic Motion Adapter (DMA) to suppress static background noise and emphasize pixel changes, effectively approximating optical flow without explicit computation. The framework also integrates a Floating Contrastive Learning strategy to distinguish informative motion dynamics from temporal background. Various experiments conducted on the TVSeries, THUMOS'14, Kinetics-400 datasets show effectiveness of our model. CAKE achieves a standout mAP compared with SOTA while using the same backbone. Our model operates at over 72 FPS on a single CPU, making it highly suitable for resource-constrained systems.
Abstract (translated)
在线动作检测（OAD）系统面临两大主要挑战：计算成本高昂，以及对背景运动中的判别性时序动态建模不足。添加光流虽能提供强运动线索，但会带来巨大的计算开销。我们提出CAKE，一个基于光流蒸馏的OAD框架，用于将运动知识迁移至RGB模型。我们设计了动态运动适配器（DMA）以抑制静态背景噪声并突出像素变化，从而在无需显式计算的情况下有效近似光流。该框架还集成了浮动对比学习策略，以区分具有信息量的运动动态与时序背景。在TVSeries、THUMOS'14和Kinetics-400数据集上的多项实验表明了我们模型的有效性。在使用相同主干网络的情况下，CAKE实现了相较于SOTA方法的显著平均精度均值提升。我们的模型在单CPU上运行速度超过72 FPS，非常适合资源受限的系统。
URL
https://arxiv.org/abs/2603.23988
PDF
-
DA-Flow: Degradation-Aware Optical Flow Estimation with Diffusion Models
2026-03-24 17:59:13Jaewon Min, Jaeeun Lee, Yeji Choi, Paul Hyunbin Cho, Jin Hyeon Kim, Tae-Young Lee, Jongsik Ahn, Hwayeong Lee, Seonghyun Park, Seungryong KimAbstract
Optical flow models trained on high-quality data often degrade severely when confronted with real-world corruptions such as blur, noise, and compression artifacts. To overcome this limitation, we formulate Degradation-Aware Optical Flow, a new task targeting accurate dense correspondence estimation from real-world corrupted videos. Our key insight is that the intermediate representations of image restoration diffusion models are inherently corruption-aware but lack temporal awareness. To address this limitation, we lift the model to attend across adjacent frames via full spatio-temporal attention, and empirically demonstrate that the resulting features exhibit zero-shot correspondence capabilities. Based on this finding, we present DA-Flow, a hybrid architecture that fuses these diffusion features with convolutional features within an iterative refinement framework. DA-Flow substantially outperforms existing optical flow methods under severe degradation across multiple benchmarks.
Abstract (translated)
在高质量数据上训练的光流模型，在面对模糊、噪声和压缩伪影等真实世界退化时，性能往往会严重下降。为克服这一限制，我们提出了“退化感知光流”（Degradation-Aware Optical Flow）这一新任务，旨在从真实世界退化的视频中实现准确的稠密对应估计。我们的核心见解是：图像修复扩散模型的中间表示天然具备退化感知能力，但缺乏时序感知能力。为解决此问题，我们通过全时空注意力机制使模型能够跨相邻帧进行关注，并实证表明由此产生的特征具有零样本对应能力。基于这一发现，我们提出了DA-Flow——一种混合架构，它在迭代精炼框架内融合了这些扩散特征与卷积特征。在多个基准测试的严重退化场景下，DA-Flow的表现均显著优于现有光流方法。
URL
https://arxiv.org/abs/2603.23499
PDF
