Optical_Flow
-
Dy3DGS-SLAM: Monocular 3D Gaussian Splatting SLAM for Dynamic Environments
2025-06-06 10:43:41Mingrui Li, Yiming Zhou, Hongxing Zhou, Xinggang Hu, Florian Roemer, Hongyu Wang, Ahmad OsmanAbstract
Current Simultaneous Localization and Mapping (SLAM) methods based on Neural Radiance Fields (NeRF) or 3D Gaussian Splatting excel in reconstructing static 3D scenes but struggle with tracking and reconstruction in dynamic environments, such as real-world scenes with moving elements. Existing NeRF-based SLAM approaches addressing dynamic challenges typically rely on RGB-D inputs, with few methods accommodating pure RGB input. To overcome these limitations, we propose Dy3DGS-SLAM, the first 3D Gaussian Splatting (3DGS) SLAM method for dynamic scenes using monocular RGB input. To address dynamic interference, we fuse optical flow masks and depth masks through a probabilistic model to obtain a fused dynamic mask. With only a single network iteration, this can constrain tracking scales and refine rendered geometry. Based on the fused dynamic mask, we designed a novel motion loss to constrain the pose estimation network for tracking. In mapping, we use the rendering loss of dynamic pixels, color, and depth to eliminate transient interference and occlusion caused by dynamic objects. Experimental results demonstrate that Dy3DGS-SLAM achieves state-of-the-art tracking and rendering in dynamic environments, outperforming or matching existing RGB-D methods.
Abstract (translated)
当前基于神经辐射场（NeRF）或三维高斯点阵（3D Gaussian Splatting）的同步定位与建图（SLAM）方法在重建静态三维场景方面表现出色，但在处理包含移动元素的真实世界动态环境时却显得力不从心。现有的针对动态挑战进行优化的基于NeRF的SLAM方法大多依赖于RGB-D输入，而能够适应纯RGB输入的方法则相对较少。为了克服这些限制，我们提出了Dy3DGS-SLAM，这是首个专门用于处理动态场景并采用单目RGB输入的三维高斯点阵（3DGS）SLAM方法。 为了解决动态干扰问题，我们通过概率模型融合光流掩码和深度掩码以获得一个融合后的动态掩码。仅需一次网络迭代即可借助这一融合后的动态掩码限制跟踪尺度并优化渲染几何形状。在设计约束姿态估计网络进行跟踪的新型运动损失时，我们亦基于该融合后的动态掩码进行了创新。 在建图过程中，我们将动像素的渲染损失、颜色和深度相结合以消除由动态对象引起的瞬态干扰与遮挡问题。实验结果表明，在处理动态环境时，Dy3DGS-SLAM不仅能够实现最先进的跟踪与渲染性能，并且超越或匹敌现有的RGB-D方法。
URL
https://arxiv.org/abs/2506.05965
PDF
-
DualX-VSR: Dual Axial Spatial$times$Temporal Transformer for Real-World Video Super-Resolution without Motion Compensation
2025-06-05 09:53:44Shuo Cao, Yihao Liu, Xiaohui Li. Yuanting Gao. Yu Zhou, Chao DongAbstract
Transformer-based models like ViViT and TimeSformer have advanced video understanding by effectively modeling spatiotemporal dependencies. Recent video generation models, such as Sora and Vidu, further highlight the power of transformers in long-range feature extraction and holistic spatiotemporal modeling. However, directly applying these models to real-world video super-resolution (VSR) is challenging, as VSR demands pixel-level precision, which can be compromised by tokenization and sequential attention mechanisms. While recent transformer-based VSR models attempt to address these issues using smaller patches and local attention, they still face limitations such as restricted receptive fields and dependence on optical flow-based alignment, which can introduce inaccuracies in real-world settings. To overcome these issues, we propose Dual Axial Spatial$\times$Temporal Transformer for Real-World Video Super-Resolution (DualX-VSR), which introduces a novel dual axial spatial$\times$temporal attention mechanism that integrates spatial and temporal information along orthogonal directions. DualX-VSR eliminates the need for motion compensation, offering a simplified structure that provides a cohesive representation of spatiotemporal information. As a result, DualX-VSR achieves high fidelity and superior performance in real-world VSR task.
Abstract (translated)
基于Transformer的模型，如ViViT和TimeSformer，通过有效地建模空间-时间依赖性，极大地推进了视频理解的发展。最近的视频生成模型，例如Sora和Vidu，则进一步突显了变压器在长程特征提取和整体空间-时间建模方面的强大能力。然而，直接将这些模型应用于现实世界的视频超分辨率（VSR）颇具挑战，因为VSR需要像素级别的精度，而令牌化和序列注意机制可能会削弱这一点。尽管最近基于Transformer的VSR模型试图通过使用较小的补丁和局部注意力来解决这些问题，但它们仍然面临着诸如受限感受野以及依赖于光流对齐所带来的可能在现实世界设置中引入不准确性的限制。 为了解决这些挑战，我们提出了用于真实世界视频超分辨率（DualX-VSR）的双重轴向空间$\times$时间变压器。该模型引入了一种新颖的双轴向空间$\times$时间注意力机制，能够沿正交方向整合空间和时间信息。通过这种方法，DualX-VSR消除了对运动补偿的需求，提供了一个结构简化、能为时空信息提供连贯表示的方法。因此，在真实世界的VSR任务中，DualX-VSR实现了高保真度，并取得了卓越的性能表现。 简而言之，DualX-VSR通过引入新颖的空间$\times$时间注意力机制来优化视频超分辨率技术，这种方法不仅可以更有效地处理时空信息，还可以简化结构并避免运动补偿带来的误差。这使得模型在真实世界的应用中能表现出更高的精确性和更好的性能。
URL
https://arxiv.org/abs/2506.04830
PDF
-
JointSplat: Probabilistic Joint Flow-Depth Optimization for Sparse-View Gaussian Splatting
2025-06-04 12:04:40Yang Xiao, Guoan Xu, Qiang Wu, Wenjing JiaAbstract
Reconstructing 3D scenes from sparse viewpoints is a long-standing challenge with wide applications. Recent advances in feed-forward 3D Gaussian sparse-view reconstruction methods provide an efficient solution for real-time novel view synthesis by leveraging geometric priors learned from large-scale multi-view datasets and computing 3D Gaussian centers via back-projection. Despite offering strong geometric cues, both feed-forward multi-view depth estimation and flow-depth joint estimation face key limitations: the former suffers from mislocation and artifact issues in low-texture or repetitive regions, while the latter is prone to local noise and global inconsistency due to unreliable matches when ground-truth flow supervision is unavailable. To overcome this, we propose JointSplat, a unified framework that leverages the complementarity between optical flow and depth via a novel probabilistic optimization mechanism. Specifically, this pixel-level mechanism scales the information fusion between depth and flow based on the matching probability of optical flow during training. Building upon the above mechanism, we further propose a novel multi-view depth-consistency loss to leverage the reliability of supervision while suppressing misleading gradients in uncertain areas. Evaluated on RealEstate10K and ACID, JointSplat consistently outperforms state-of-the-art (SOTA) methods, demonstrating the effectiveness and robustness of our proposed probabilistic joint flow-depth optimization approach for high-fidelity sparse-view 3D reconstruction.
Abstract (translated)
从稀疏视角重建三维场景是一个具有广泛应用的长期挑战。近期，基于前馈的三维高斯稀疏视图重建方法取得了进展，通过利用大规模多视图数据集中学习到的几何先验，并计算通过反向投影获得的三维高斯中心，提供了一种实时新视角合成的有效解决方案。尽管这些方法提供了强大的几何线索，但前馈多视图深度估计和光流-深度联合估计都面临关键限制：前者在低纹理或重复区域容易出现定位错误和伪影问题；后者由于缺乏地面实况光流监督而导致不准确的匹配时，则会受到局部噪声和全局一致性差的影响。为克服这些问题，我们提出了JointSplat，这是一个统一框架，通过新颖的概率优化机制利用了光流与深度之间的互补性。具体来说，这种像素级别的机制根据训练过程中的光流匹配概率来调整深度和光流的信息融合规模。基于上述机制，我们进一步提出了一种新的多视图深度一致性损失方法，以利用监督的可靠性并抑制不确定区域中误导性的梯度影响。 在RealEstate10K和ACID数据集上的评估表明，JointSplat始终优于最先进的（SOTA）方法，这证明了我们的概率联合光流-深度优化方法对于高保真稀疏视图三维重建的有效性和鲁棒性。
URL
https://arxiv.org/abs/2506.03872
PDF
-
EDCFlow: Exploring Temporally Dense Difference Maps for Event-based Optical Flow Estimation
2025-06-04 02:55:04Daikun Liu, Lei Cheng, Teng Wang, changyin SunAbstract
Recent learning-based methods for event-based optical flow estimation utilize cost volumes for pixel matching but suffer from redundant computations and limited scalability to higher resolutions for flow refinement. In this work, we take advantage of the complementarity between temporally dense feature differences of adjacent event frames and cost volume and present a lightweight event-based optical flow network (EDCFlow) to achieve high-quality flow estimation at a higher resolution. Specifically, an attention-based multi-scale temporal feature difference layer is developed to capture diverse motion patterns at high resolution in a computation-efficient manner. An adaptive fusion of high-resolution difference motion features and low-resolution correlation motion features is performed to enhance motion representation and model generalization. Notably, EDCFlow can serve as a plug-and-play refinement module for RAFT-like event-based methods to enhance flow details. Extensive experiments demonstrate that EDCFlow achieves better performance with lower complexity compared to existing methods, offering superior generalization.
Abstract (translated)
最近基于学习的方法用于事件驱动的光流估计时，虽然利用了成本体（cost volumes）来进行像素匹配，但仍然面临着冗余计算和难以扩展到更高分辨率的问题。在这项工作中，我们充分利用了相邻事件帧之间时间密集特征差与成本体之间的互补性，并提出了一种轻量级的基于事件的光流网络（EDCFlow），旨在实现高分辨率下的高质量光流估计。 具体而言，我们开发了一个基于注意力机制的多尺度时间特征差异层，以高效地捕捉不同运动模式在高分辨率下的表现。此外，还进行了一项自适应融合操作，将高分辨率差动运动特征与低分辨率相关运动特征相结合，以此增强对运动的表示和模型泛化能力。 值得注意的是，EDCFlow可以作为插件式的细化模块应用于类似RAFT的方法中，以提高光流细节的质量。通过广泛的实验，我们证明了相较于现有方法，EDCFlow在性能和复杂度之间实现了更好的平衡，并且具有更优越的泛化能力。
URL
https://arxiv.org/abs/2506.03512
PDF
-
Learning Optical Flow Field via Neural Ordinary Differential Equation
2025-06-03 18:30:14Leyla Mirvakhabova, Hong Cai, Jisoo Jeong, Hanno Ackermann, Farhad Zanjani, Fatih PorikliAbstract
Recent works on optical flow estimation use neural networks to predict the flow field that maps positions of one image to positions of the other. These networks consist of a feature extractor, a correlation volume, and finally several refinement steps. These refinement steps mimic the iterative refinements performed by classical optimization algorithms and are usually implemented by neural layers (e.g., GRU) which are recurrently executed for a fixed and pre-determined number of steps. However, relying on a fixed number of steps may result in suboptimal performance because it is not tailored to the input data. In this paper, we introduce a novel approach for predicting the derivative of the flow using a continuous model, namely neural ordinary differential equations (ODE). One key advantage of this approach is its capacity to model an equilibrium process, dynamically adjusting the number of compute steps based on the data at hand. By following a particular neural architecture, ODE solver, and associated hyperparameters, our proposed model can replicate the exact same updates as recurrent cells used in existing works, offering greater generality. Through extensive experimental analysis on optical flow benchmarks, we demonstrate that our approach achieves an impressive improvement over baseline and existing models, all while requiring only a single refinement step.
Abstract (translated)
最近关于光流估计的工作利用神经网络来预测将一张图像的位置映射到另一张图像位置的流动场。这些网络通常包含特征提取器、相关体积（correlation volume）以及几个后续的细化步骤。这些细化步骤模仿了经典优化算法中迭代改进的过程，并且通常通过递归执行一定次数的神经层（如GRU）来实现。然而，依赖于固定数量的步骤可能导致次优性能，因为它没有针对输入数据进行定制调整。 在本文中，我们提出了一种使用连续模型——即神经常微分方程（ODE）来预测流动场导数的新方法。这种方法的一个关键优势在于它能够建模一个平衡过程，并且可以根据手头的数据动态地调整计算步骤的数量。通过遵循特定的神经网络架构、ODE求解器以及相关的超参数，我们的提议模型可以复制现有工作中使用的递归单元所执行的确切更新操作，从而提供更大的通用性。 在广泛的光流基准实验分析中，我们展示了该方法比基线和现有的模型实现了显著改进，并且只需要一个细化步骤即可实现。
URL
https://arxiv.org/abs/2506.03290
PDF
-
LinkTo-Anime: A 2D Animation Optical Flow Dataset from 3D Model Rendering
2025-06-03 10:50:20Xiaoyi Feng, Kaifeng Zou, Caichun Cen, Tao Huang, Hui Guo, Zizhou Huang, Yingli Zhao, Mingqing Zhang, Diwei Wang, Yuntao Zou, Dagang LiAbstract
Existing optical flow datasets focus primarily on real-world simulation or synthetic human motion, but few are tailored to Celluloid(cel) anime character motion: a domain with unique visual and motion characteristics. To bridge this gap and facilitate research in optical flow estimation and downstream tasks such as anime video generation and line drawing colorization, we introduce LinkTo-Anime, the first high-quality dataset specifically designed for cel anime character motion generated with 3D model rendering. LinkTo-Anime provides rich annotations including forward and backward optical flow, occlusion masks, and Mixamo Skeleton. The dataset comprises 395 video sequences, totally 24,230 training frames, 720 validation frames, and 4,320 test frames. Furthermore, a comprehensive benchmark is constructed with various optical flow estimation methods to analyze the shortcomings and limitations across multiple datasets.
Abstract (translated)
现有的光学流数据集主要关注现实世界的模拟或合成人体运动，但很少有数据集专门针对赛璐珞（cel）动画角色的运动：这是一个具有独特视觉和运动特征的领域。为了填补这一空白，并促进光学流估算以及下游任务如动漫视频生成和线条画着色的研究，我们引入了LinkTo-Anime，这是第一个专门为使用3D模型渲染的赛璐珞动画角色运动设计的高质量数据集。LinkTo-Anime提供了丰富的注释，包括前向和后向光学流、遮挡掩码以及Mixamo骨架信息。该数据集包含395个视频序列，总计24,230帧用于训练，720帧用于验证，以及4,320帧用于测试。此外，还构建了一个全面的基准测试平台，使用各种光学流估算方法来分析不同数据集中的不足和限制。
URL
https://arxiv.org/abs/2506.02733
PDF
-
LumosFlow: Motion-Guided Long Video Generation
2025-06-03 06:25:00Jiahao Chen, Hangjie Yuan, Yichen Qian, Jingyun Liang, Jiazheng Xing, Pengwei Liu, Weihua Chen, Fan Wang, Bing SuAbstract
Long video generation has gained increasing attention due to its widespread applications in fields such as entertainment and simulation. Despite advances, synthesizing temporally coherent and visually compelling long sequences remains a formidable challenge. Conventional approaches often synthesize long videos by sequentially generating and concatenating short clips, or generating key frames and then interpolate the intermediate frames in a hierarchical manner. However, both of them still remain significant challenges, leading to issues such as temporal repetition or unnatural transitions. In this paper, we revisit the hierarchical long video generation pipeline and introduce LumosFlow, a framework introduce motion guidance explicitly. Specifically, we first employ the Large Motion Text-to-Video Diffusion Model (LMTV-DM) to generate key frames with larger motion intervals, thereby ensuring content diversity in the generated long videos. Given the complexity of interpolating contextual transitions between key frames, we further decompose the intermediate frame interpolation into motion generation and post-hoc refinement. For each pair of key frames, the Latent Optical Flow Diffusion Model (LOF-DM) synthesizes complex and large-motion optical flows, while MotionControlNet subsequently refines the warped results to enhance quality and guide intermediate frame generation. Compared with traditional video frame interpolation, we achieve 15x interpolation, ensuring reasonable and continuous motion between adjacent frames. Experiments show that our method can generate long videos with consistent motion and appearance. Code and models will be made publicly available upon acceptance. Our project page: this https URL
Abstract (translated)
长时间视频生成因其在娱乐和仿真等领域的广泛应用而受到了越来越多的关注。尽管已经取得了进展，但合成时间上连贯且视觉吸引人的长序列仍然是一个艰巨的挑战。传统的方法通常通过依次生成并连接短片段来构建长视频，或者通过分层次的方式生成关键帧再插值中间帧。然而这两种方法仍然存在显著的问题，导致诸如时间上的重复或过渡不自然等问题。 在本文中，我们重新审视了分层长时间视频生成流水线，并引入了LumosFlow框架，该框架明确地引入运动引导机制。具体而言，我们首先利用大型动作文本到视频扩散模型（Large Motion Text-to-Video Diffusion Model, LMTV-DM）来生成具有较大动作间隔的关键帧，从而确保生成的长视频内容多样性。鉴于在关键帧之间进行上下文过渡插值的复杂性，我们将中间帧插值进一步分解为运动生成和事后精炼两个阶段。对于每一对关键帧，Latent Optical Flow Diffusion Model（LOF-DM）会合成复杂的、大动作位移流，而随后通过MotionControlNet对形变结果进行细化以提升质量，并指导中间帧的生成。 与传统的视频帧插值相比，我们实现了15倍的插值，确保了相邻帧之间的合理且连贯的动作。实验表明，我们的方法可以生成具有一致运动和外观的长视频。代码和模型将在被接受后公开发布。项目页面：[请在此处插入实际链接]。 请注意，原文末尾提到的实际链接需在正式发布时添加具体的URL地址以便访问相关资源。
URL
https://arxiv.org/abs/2506.02497
PDF
-
MS-RAFT-3D: A Multi-Scale Architecture for Recurrent Image-Based Scene Flow
2025-06-02 08:59:05Jakob Schmid, Azin Jahedi, Noah Berenguel Senn, Andr\'es BruhnAbstract
Although multi-scale concepts have recently proven useful for recurrent network architectures in the field of optical flow and stereo, they have not been considered for image-based scene flow so far. Hence, based on a single-scale recurrent scene flow backbone, we develop a multi-scale approach that generalizes successful hierarchical ideas from optical flow to image-based scene flow. By considering suitable concepts for the feature and the context encoder, the overall coarse-to-fine framework and the training loss, we succeed to design a scene flow approach that outperforms the current state of the art on KITTI and Spring by 8.7%(3.89 vs. 4.26) and 65.8% (9.13 vs. 26.71), respectively. Our code is available at this https URL.
Abstract (translated)
尽管多尺度概念最近在光流和立体视觉领域被证明对递归网络架构非常有用，但迄今为止它们尚未应用于基于图像的场景流量。因此，我们以单尺度递归场景流量主干为基础，开发了一种多尺度方法，将光学领域的成功分层理念推广到基于图像的场景流量中。通过考虑特征编码器和上下文编码器中的合适概念、整体从粗到精框架以及训练损失，我们设计出一种在KITTI数据集上优于当前最先进技术8.7%（3.89 vs. 4.26）并在Spring数据集中表现更好，提高了65.8%（9.13 vs. 26.71）的场景流量方法。我们的代码可在[此链接](https://example.com)获取。
URL
https://arxiv.org/abs/2506.01443
PDF
-
MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows
2025-06-01 18:53:27Hong Nguyen, Dung Tran, Hieu Hoang, Phong Nguyen, Shrikanth NarayananAbstract
Many motion-centric video analysis tasks, such as atomic actions, detecting atypical motor behavior in individuals with autism, or analyzing articulatory motion in real-time MRI of human speech, require efficient and interpretable temporal modeling. Capturing temporal dynamics is a central challenge in video analysis, often requiring significant computational resources and fine-grained annotations that are not widely available. This paper presents MOOSE (Motion Flow Over Spatial Space), a novel temporally-centric video encoder explicitly integrating optical flow with spatial embeddings to model temporal information efficiently, inspired by human perception of motion. Unlike prior models, MOOSE takes advantage of rich, widely available pre-trained visual and optical flow encoders instead of training video models from scratch. This significantly reduces computational complexity while enhancing temporal interpretability. Our primary contributions includes (1) proposing a computationally efficient temporally-centric architecture for video understanding (2) demonstrating enhanced interpretability in modeling temporal dynamics; and (3) achieving state-of-the-art performance on diverse benchmarks, including clinical, medical, and standard action recognition datasets, confirming the broad applicability and effectiveness of our approach.
Abstract (translated)
许多以运动为中心的视频分析任务，如原子动作识别、自闭症个体中非典型性运动行为检测以及实时MRI下人类言语发音动作的分析，都需要高效的可解释的时间模型。捕捉时间动态是视频分析中的一个核心挑战，这通常需要大量的计算资源和精细标注的数据，而这些数据并不广泛可用。本文介绍了MOOSE（Motion Flow Over Spatial Space），这是一种新颖的时间为中心的视频编码器，它通过将光流与空间嵌入相结合来高效地建模时间信息，灵感来源于人类对运动的理解方式。不同于先前的模型，MOOSE利用了丰富的、广泛可用的预训练视觉和光流编码器，而不是从头开始训练视频模型。这大大减少了计算复杂性，并提高了时间可解释性。 我们的主要贡献包括： 1. 提出了一种计算效率高的以时间为中心的架构，用于视频理解； 2. 展示了在建模时间动态方面增强了可解释性；以及 3. 在各种基准测试中达到了最先进的性能，这些基准涵盖了临床、医疗和标准动作识别数据集，这证实了我们的方法具有广泛适用性和有效性。
URL
https://arxiv.org/abs/2506.01119
PDF
-
Improving Optical Flow and Stereo Depth Estimation by Leveraging Uncertainty-Based Learning Difficulties
2025-05-31 00:37:12Jisoo Jeong, Hong Cai, Jamie Menjay Lin, Fatih PorikliAbstract
Conventional training for optical flow and stereo depth models typically employs a uniform loss function across all pixels. However, this one-size-fits-all approach often overlooks the significant variations in learning difficulty among individual pixels and contextual regions. This paper investigates the uncertainty-based confidence maps which capture these spatially varying learning difficulties and introduces tailored solutions to address them. We first present the Difficulty Balancing (DB) loss, which utilizes an error-based confidence measure to encourage the network to focus more on challenging pixels and regions. Moreover, we identify that some difficult pixels and regions are affected by occlusions, resulting from the inherently ill-posed matching problem in the absence of real correspondences. To address this, we propose the Occlusion Avoiding (OA) loss, designed to guide the network into cycle consistency-based confident regions, where feature matching is more reliable. By combining the DB and OA losses, we effectively manage various types of challenging pixels and regions during training. Experiments on both optical flow and stereo depth tasks consistently demonstrate significant performance improvements when applying our proposed combination of the DB and OA losses.
Abstract (translated)
传统的光流和立体深度模型训练通常采用在整个图像的所有像素上统一应用的损失函数。然而，这种一刀切的方法往往忽视了各个像素及其上下文区域在学习难度上的显著差异。本文研究了一种基于不确定性的置信度图，该图捕捉到了这些空间变化的学习困难，并提出了定制化解决方案来应对这些问题。 首先，我们介绍了难度平衡（DB）损失函数，它使用一种基于误差的置信度测量方法，鼓励网络更加关注那些具有挑战性的像素和区域。此外，我们发现一些难以处理的像素和区域受到遮挡的影响，这是由于在缺乏真实对应关系的情况下匹配问题本质上是病态的。为了解决这个问题，我们提出了避免遮挡（OA）损失函数，它旨在引导网络进入基于循环一致性的可靠特征匹配区域。 通过结合DB和OA损失函数，在训练过程中有效地管理各种类型的具有挑战性的像素和区域。实验结果表明，在光流和立体深度任务上应用我们提出的DB和OA损失组合方法时，性能有显著提升。
URL
https://arxiv.org/abs/2506.00324
PDF
-
Towards a Generalizable Bimanual Foundation Policy via Flow-based Video Prediction
2025-05-30 03:01:21Chenyou Fan, Fangzheng Yan, Chenjia Bai, Jiepeng Wang, Chi Zhang, Zhen Wang, Xuelong LiAbstract
Learning a generalizable bimanual manipulation policy is extremely challenging for embodied agents due to the large action space and the need for coordinated arm movements. Existing approaches rely on Vision-Language-Action (VLA) models to acquire bimanual policies. However, transferring knowledge from single-arm datasets or pre-trained VLA models often fails to generalize effectively, primarily due to the scarcity of bimanual data and the fundamental differences between single-arm and bimanual manipulation. In this paper, we propose a novel bimanual foundation policy by fine-tuning the leading text-to-video models to predict robot trajectories and training a lightweight diffusion policy for action generation. Given the lack of embodied knowledge in text-to-video models, we introduce a two-stage paradigm that fine-tunes independent text-to-flow and flow-to-video models derived from a pre-trained text-to-video model. Specifically, optical flow serves as an intermediate variable, providing a concise representation of subtle movements between images. The text-to-flow model predicts optical flow to concretize the intent of language instructions, and the flow-to-video model leverages this flow for fine-grained video prediction. Our method mitigates the ambiguity of language in single-stage text-to-video prediction and significantly reduces the robot-data requirement by avoiding direct use of low-level actions. In experiments, we collect high-quality manipulation data for real dual-arm robot, and the results of simulation and real-world experiments demonstrate the effectiveness of our method.
Abstract (translated)
学习一种通用的双臂操作策略对于具身代理来说极为具有挑战性，原因在于庞大的动作空间以及需要协调的手臂运动。现有的方法依赖于视觉-语言-行动（VLA）模型来获取双臂策略。然而，从单臂数据集或预训练的VLA模型中转移知识往往难以有效泛化，主要是由于双臂数据稀缺和单臂与双臂操作之间存在本质差异所致。 在本文中，我们提出了一种新颖的双臂基础策略，通过微调领先的文本到视频模型来预测机器人轨迹，并训练一个轻量级扩散策略用于动作生成。鉴于文本到视频模型缺乏具身知识，我们引入了一个两阶段范式，以预训练的文本到视频模型为基础，对独立的文本到流（flow）和流到视频模型进行微调。具体来说，光学流动充当中间变量，提供图像之间细微运动的简洁表示。文本到流模型预测光学流动，将语言指令的具体意图形式化；而流到视频模型则利用此流动实现精细的视频预测。我们的方法减轻了一步式文本到视频预测中语言模糊性的问题，并通过避免直接使用低级动作显著减少了对机器人数据的需求。 在实验中，我们为实际的双臂机器人收集了高质量的操作数据，模拟和真实世界实验的结果展示了该方法的有效性。
URL
https://arxiv.org/abs/2505.24156
PDF
-
Zero-to-Hero: Zero-Shot Initialization Empowering Reference-Based Video Appearance Editing
2025-05-29 06:10:16Tongtong Su, Chengyu Wang, Jun Huang, Dongming LuAbstract
Appearance editing according to user needs is a pivotal task in video editing. Existing text-guided methods often lead to ambiguities regarding user intentions and restrict fine-grained control over editing specific aspects of objects. To overcome these limitations, this paper introduces a novel approach named {Zero-to-Hero}, which focuses on reference-based video editing that disentangles the editing process into two distinct problems. It achieves this by first editing an anchor frame to satisfy user requirements as a reference image and then consistently propagating its appearance across other frames. We leverage correspondence within the original frames to guide the attention mechanism, which is more robust than previously proposed optical flow or temporal modules in memory-friendly video generative models, especially when dealing with objects exhibiting large motions. It offers a solid ZERO-shot initialization that ensures both accuracy and temporal consistency. However, intervention in the attention mechanism results in compounded imaging degradation with over-saturated colors and unknown blurring issues. Starting from Zero-Stage, our Hero-Stage Holistically learns a conditional generative model for vidEo RestOration. To accurately evaluate the consistency of the appearance, we construct a set of videos with multiple appearances using Blender, enabling a fine-grained and deterministic evaluation. Our method outperforms the best-performing baseline with a PSNR improvement of 2.6 dB. The project page is at this https URL.
Abstract (translated)
根据用户需求进行外观编辑是视频编辑中的关键任务之一。现有的文本引导方法常常导致关于用户意图的模糊性，并限制了对特定对象方面进行精细控制的能力。为了克服这些局限，本文提出了一种名为{Zero-to-Hero}的新方法，该方法专注于基于参考的视频编辑，将编辑过程分解为两个不同的问题：首先编辑一个锚帧以满足用户的需要作为参考图像，然后在其他帧中一致地传播其外观。我们利用原始帧之间的对应关系来指导注意力机制，在处理具有大幅度运动的对象时，这种方法比以前提出的光流或时间模块更稳健，尤其是在内存友好的视频生成模型中更为适用。它提供了一个坚实的零样本初始化，确保了准确性和时间一致性。然而，干预注意力机制会导致图像退化的问题加重，表现为过饱和的颜色和未知的模糊问题。 从Zero-Stage开始，我们的Hero-Stage全面学习了一种条件生成模型用于视频恢复。为了精确评估外观的一致性，我们使用Blender构建了一系列具有多种外观变化的视频集，从而进行精细且确定性的评估。我们的方法超越了最佳基线，在PSNR（峰值信噪比）上提高了2.6分贝。 该项目页面可在以下链接访问：[https URL] （请注意，实际URL需要被替换为项目的具体网址）。
URL
https://arxiv.org/abs/2505.23134
PDF
-
Object Concepts Emerge from Motion
2025-05-27 18:09:02Haoqian Liang, Xiaohui Wang, Zhichao Li, Ya Yang, Naiyan WangAbstract
Object concepts play a foundational role in human visual cognition, enabling perception, memory, and interaction in the physical world. Inspired by findings in developmental neuroscience - where infants are shown to acquire object understanding through observation of motion - we propose a biologically inspired framework for learning object-centric visual representations in an unsupervised manner. Our key insight is that motion boundary serves as a strong signal for object-level grouping, which can be used to derive pseudo instance supervision from raw videos. Concretely, we generate motion-based instance masks using off-the-shelf optical flow and clustering algorithms, and use them to train visual encoders via contrastive learning. Our framework is fully label-free and does not rely on camera calibration, making it scalable to large-scale unstructured video data. We evaluate our approach on three downstream tasks spanning both low-level (monocular depth estimation) and high-level (3D object detection and occupancy prediction) vision. Our models outperform previous supervised and self-supervised baselines and demonstrate strong generalization to unseen scenes. These results suggest that motion-induced object representations offer a compelling alternative to existing vision foundation models, capturing a crucial but overlooked level of abstraction: the visual instance. The corresponding code will be released upon paper acceptance.
Abstract (translated)
物体概念在人类视觉认知中扮演着基础性的角色，使感知、记忆和与物理世界的互动成为可能。受到发育神经科学发现的启发——即婴儿通过观察运动来获得对物体的理解——我们提出了一种生物启发式的框架，在无监督的方式下学习以对象为中心的视觉表示。我们的核心洞察是，运动边界可以作为用于将同一场景中的不同实例分组的强大信号，这可以从原始视频中推导出伪样本标签。具体来说，我们使用现成的光流和聚类算法生成基于运动的实例掩模，并利用它们通过对比学习训练视觉编码器。我们的框架完全不需要标签，也不依赖于相机校准，使其能够应用于大规模无结构视频数据。 我们在三个下游任务上评估了我们的方法，这些任务涵盖了从低级（单目深度估计）到高级（3D物体检测和占用预测）的视觉处理。我们的模型在之前监督学习及自监督学习基准测试中表现出色，并且对未见过的场景具有强大的泛化能力。这些结果表明，运动诱导的对象表示为现有视觉基础模型提供了一种有吸引力的替代方案，捕捉到了一个关键但被忽视的抽象层次：视觉实例。相应的代码将在论文接受后发布。
URL
https://arxiv.org/abs/2505.21635
PDF
-
A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarking
2025-05-26 11:45:10Zixiang Zhao, Haowen Bai, Bingxin Ke, Yukun Cui, Lilun Deng, Yulun Zhang, Kai Zhang, Konrad SchindlerAbstract
The real world is dynamic, yet most image fusion methods process static frames independently, ignoring temporal correlations in videos and leading to flickering and temporal inconsistency. To address this, we propose Unified Video Fusion (UniVF), a novel framework for temporally coherent video fusion that leverages multi-frame learning and optical flow-based feature warping for informative, temporally coherent video fusion. To support its development, we also introduce Video Fusion Benchmark (VF-Bench), the first comprehensive benchmark covering four video fusion tasks: multi-exposure, multi-focus, infrared-visible, and medical fusion. VF-Bench provides high-quality, well-aligned video pairs obtained through synthetic data generation and rigorous curation from existing datasets, with a unified evaluation protocol that jointly assesses the spatial quality and temporal consistency of video fusion. Extensive experiments show that UniVF achieves state-of-the-art results across all tasks on VF-Bench. Project page: this https URL.
Abstract (translated)
现实世界是动态的，但大多数图像融合方法处理的是静态帧，并且忽略了视频中的时间相关性，导致闪烁和时间上的一致性问题。为了解决这个问题，我们提出了统一视频融合（Unified Video Fusion, UniVF），这是一种新颖的框架，用于生成具有时间一致性的视频融合结果，它利用多帧学习和基于光学流的特征扭曲技术来实现信息丰富且时间上连贯的视频融合。为了支持其开发，我们也引入了视频融合基准测试（Video Fusion Benchmark, VF-Bench），这是第一个全面的基准测试集，涵盖了四种视频融合任务：多曝光、多焦点、红外-可见光和医学图像融合。VF-Bench提供了高质量、精确对齐的视频配对，这些数据通过合成数据生成并从现有数据集中严格筛选出来，并提供了一个统一的评估协议，用于共同评估视频融合的空间质量和时间一致性。广泛的实验表明，在VF-Bench的所有任务上，UniVF都达到了最先进的结果。 项目页面：[此链接](https://this-url.com)。
URL
https://arxiv.org/abs/2505.19858
PDF
-
Brightness-Invariant Tracking Estimation in Tagged MRI
2025-05-23 20:48:23Zhangxing Bian, Shuwen Wei, Xiao Liang, Yuan-Chiao Lu, Samuel W. Remedios, Fangxu Xing, Jonghye Woo, Dzung L. Pham, Aaron Carass, Philip V. Bayly, Jiachen Zhuo, Ahmed Alshareef, Jerry L. PrinceAbstract
Magnetic resonance (MR) tagging is an imaging technique for noninvasively tracking tissue motion in vivo by creating a visible pattern of magnetization saturation (tags) that deforms with the tissue. Due to longitudinal relaxation and progression to steady-state, the tags and tissue brightnesses change over time, which makes tracking with optical flow methods error-prone. Although Fourier methods can alleviate these problems, they are also sensitive to brightness changes as well as spectral spreading due to motion. To address these problems, we introduce the brightness-invariant tracking estimation (BRITE) technique for tagged MRI. BRITE disentangles the anatomy from the tag pattern in the observed tagged image sequence and simultaneously estimates the Lagrangian motion. The inherent ill-posedness of this problem is addressed by leveraging the expressive power of denoising diffusion probabilistic models to represent the probabilistic distribution of the underlying anatomy and the flexibility of physics-informed neural networks to estimate biologically-plausible motion. A set of tagged MR images of a gel phantom was acquired with various tag periods and imaging flip angles to demonstrate the impact of brightness variations and to validate our method. The results show that BRITE achieves more accurate motion and strain estimates as compared to other state of the art methods, while also being resistant to tag fading.
Abstract (translated)
磁共振标记（MR 标记）是一种用于非侵入性追踪活体内组织运动的成像技术，通过创建一个可视化的磁饱和模式（标签），该模式会随着组织变形而变化。由于纵向弛豫和向稳态发展的过程，标签及组织亮度随时间发生变化，这使得使用光学流方法进行跟踪容易产生误差。虽然傅立叶变换可以缓解这些问题，但它们对亮度变化同样敏感，并且会因运动导致频谱扩散问题。为了应对这些挑战，我们引入了一种名为亮度不变追踪估计（BRITE）的技术用于标记 MRI。BRITE 技术能够从观察到的标记图像序列中分离出解剖结构和标签模式，并同时估算拉格朗日运动。通过利用去噪扩散概率模型来表示潜在解剖结构的概率分布以及采用物理信息神经网络来估算生物合理性运动，这种方法解决了该问题内在的不适定性。我们使用一组不同标记周期和成像翻转角获得的凝胶模型标记 MR 图像数据集，以演示亮度变化的影响并验证我们的方法的有效性。实验结果表明，与现有的先进技术相比，BRITE 能够提供更准确的运动和应变估计，并且对标签褪色具有更强的抗干扰能力。
URL
https://arxiv.org/abs/2505.18365
PDF
-
CTRL-GS: Cascaded Temporal Residue Learning for 4D Gaussian Splatting
Abstract
Recently, Gaussian Splatting methods have emerged as a desirable substitute for prior Radiance Field methods for novel-view synthesis of scenes captured with multi-view images or videos. In this work, we propose a novel extension to 4D Gaussian Splatting for dynamic scenes. Drawing on ideas from residual learning, we hierarchically decompose the dynamic scene into a "video-segment-frame" structure, with segments dynamically adjusted by optical flow. Then, instead of directly predicting the time-dependent signals, we model the signal as the sum of video-constant values, segment-constant values, and frame-specific residuals, as inspired by the success of residual learning. This approach allows more flexible models that adapt to highly variable scenes. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets, with the greatest improvements on complex scenes with large movements, occlusions, and fine details, where current methods degrade most.
Abstract (translated)
最近，高斯点置法（Gaussian Splatting）作为一种新型方法，在使用多视角图像或视频捕获的场景的新视图合成中，成为之前辐射场（Radiance Field）方法的理想替代方案。在本项工作中，我们提出了一种针对动态场景扩展4D高斯点置法的创新方法。借鉴残差学习的思想，我们将动态场景层次化地分解成“视频片段帧”结构，并通过光流进行动态调整。然后，我们不像直接预测时间依赖信号那样处理问题，而是将信号建模为视频常量值、片段常量值和特定帧残差的总和，这受到残差学习成功的启发。这种方法使得模型能够更灵活地适应变化多端的场景。我们在几个已建立的数据集上展示了最先进的视觉质量和实时渲染性能，并且在包含大量移动、遮挡以及细节复杂场景的情况下改进最大，这是当前方法表现最差的地方。
URL
https://arxiv.org/abs/2505.18306
PDF
-
Real-time Traffic Accident Anticipation with Feature Reuse
2025-05-23 04:09:26Inpyo Song, Jangwon LeeAbstract
This paper addresses the problem of anticipating traffic accidents, which aims to forecast potential accidents before they happen. Real-time anticipation is crucial for safe autonomous driving, yet most methods rely on computationally heavy modules like optical flow and intermediate feature extractors, making real-world deployment challenging. In this paper, we thus introduce RARE (Real-time Accident anticipation with Reused Embeddings), a lightweight framework that capitalizes on intermediate features from a single pre-trained object detector. By eliminating additional feature-extraction pipelines, RARE significantly reduces latency. Furthermore, we introduce a novel Attention Score Ranking Loss, which prioritizes higher attention on accident-related objects over non-relevant ones. This loss enhances both accuracy and interpretability. RARE demonstrates a 4-8 times speedup over existing approaches on the DAD and CCD benchmarks, achieving a latency of 13.6ms per frame (73.3 FPS) on an RTX 6000. Moreover, despite its reduced complexity, it attains state-of-the-art Average Precision and reliably anticipates imminent collisions in real time. These results highlight RARE's potential for safety-critical applications where timely and explainable anticipation is essential.
Abstract (translated)
本文解决的是预测交通事故的问题，旨在在事故发生前进行预测。实时预测对于自动驾驶的安全性至关重要，然而大多数方法依赖于计算量大的模块（如光流和中间特征提取器），这使得实际部署变得颇具挑战。为此，在本文中我们提出了RARE（Real-time Accident anticipation with Reused Embeddings），这是一个轻量级框架，利用了单一预训练对象检测器中的中间特征。通过消除额外的特征提取管道，RARE显著减少了延迟。 此外，我们还引入了一种新的注意力得分排序损失函数，该损失函数优先关注与事故相关的物体而非无关物体，从而提高了准确性和可解释性。在DAD和CCD基准测试上，RARE比现有的方法快4到8倍，在RTX 6000上实现了每帧13.6毫秒（73.3 FPS）的延迟。并且尽管其复杂度降低，它仍达到了最先进的平均精度，并且能够实时可靠地预测即将发生的碰撞。 这些结果突显了RARE在需要及时和可解释性的安全关键应用中的潜力。
URL
https://arxiv.org/abs/2505.17449
PDF
-
Efficient Correlation Volume Sampling for Ultra-High-Resolution Optical Flow Estimation
2025-05-22 17:30:38Karlis Martins Briedis, Markus Gross, Christopher SchroersAbstract
Recent optical flow estimation methods often employ local cost sampling from a dense all-pairs correlation volume. This results in quadratic computational and memory complexity in the number of pixels. Although an alternative memory-efficient implementation with on-demand cost computation exists, this is slower in practice and therefore prior methods typically process images at reduced resolutions, missing fine-grained details. To address this, we propose a more efficient implementation of the all-pairs correlation volume sampling, still matching the exact mathematical operator as defined by RAFT. Our approach outperforms on-demand sampling by up to 90% while maintaining low memory usage, and performs on par with the default implementation with up to 95% lower memory usage. As cost sampling makes up a significant portion of the overall runtime, this can translate to up to 50% savings for the total end-to-end model inference in memory-constrained environments. Our evaluation of existing methods includes an 8K ultra-high-resolution dataset and an additional inference-time modification of the recent SEA-RAFT method. With this, we achieve state-of-the-art results at high resolutions both in accuracy and efficiency.
Abstract (translated)
最近的光流估计方法通常采用从密集的所有成对相关体积中抽取局部成本的方法。这导致计算和内存复杂度随像素数量呈二次增长。虽然存在一种记忆高效的实现方式，可以在需要时进行成本计算，但这种方法在实践中速度较慢，因此以前的方法通常会以降低图像分辨率的方式来处理图像，从而忽略了细粒度的细节。为了解决这个问题，我们提出了一种更高效的全部成对相关体积采样实现方法，在保持与RAFT定义的确切数学操作一致的同时，比按需成本计算方式快最多90%，同时保持低内存使用率，并且相比默认实现方式内存消耗减少了高达95%。由于成本采样占整个运行时间的重要部分，这可以在内存受限的环境中将总体端到端模型推理节省多达50%的内存。 我们对现有方法的评估包括一个8K超高分辨率的数据集以及最近SEA-RAFT方法在推断时的一个额外修改版本。通过这种方法，在高分辨率下，我们在准确性和效率方面均达到了最先进的成果。
URL
https://arxiv.org/abs/2505.16942
PDF
-
V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation
2025-05-22 15:38:12Hanyue Lou, Jinxiu Liang, Minggui Teng, Yi Wang, Boxin ShiAbstract
Event-based cameras offer unique advantages such as high temporal resolution, high dynamic range, and low power consumption. However, the massive storage requirements and I/O burdens of existing synthetic data generation pipelines and the scarcity of real data prevent event-based training datasets from scaling up, limiting the development and generalization capabilities of event vision models. To address this challenge, we introduce Video-to-Voxel (V2V), an approach that directly converts conventional video frames into event-based voxel grid representations, bypassing the storage-intensive event stream generation entirely. V2V enables a 150 times reduction in storage requirements while supporting on-the-fly parameter randomization for enhanced model robustness. Leveraging this efficiency, we train several video reconstruction and optical flow estimation model architectures on 10,000 diverse videos totaling 52 hours--an order of magnitude larger than existing event datasets, yielding substantial improvements.
Abstract (translated)
基于事件的相机提供了诸如高时间分辨率、宽动态范围和低功耗等独特优势。然而，现有合成数据生成管道的巨大存储需求和输入输出(I/O)负担，以及真实数据的稀缺性，限制了基于事件的训练数据集的扩展能力，从而阻碍了事件视觉模型的发展和泛化能力。为了解决这一挑战，我们引入了一种名为视频到体素（Video-to-Voxel, V2V）的方法，该方法可以直接将传统的视频帧转换为基于事件的体素网格表示，完全绕过了存储密集型的事件流生成过程。通过采用V2V技术，我们可以实现150倍的存储需求减少，并支持实时参数随机化以增强模型鲁棒性。利用这一效率，我们在总计长达52小时（包含10,000多个多样视频）的数据集上训练了几个视频重建和光流估计模型架构——这比现有的基于事件数据集大了一个数量级，从而显著提高了性能。
URL
https://arxiv.org/abs/2505.16797
PDF
-
RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers
2025-05-19 16:50:26Ahmet Berke Gokmen, Yigit Ekin, Bahri Batuhan Bilecen, Aysegul DundarAbstract
We propose RoPECraft, a training-free video motion transfer method for diffusion transformers that operates solely by modifying their rotary positional embeddings (RoPE). We first extract dense optical flow from a reference video, and utilize the resulting motion offsets to warp the complex-exponential tensors of RoPE, effectively encoding motion into the generation process. These embeddings are then further optimized during denoising time steps via trajectory alignment between the predicted and target velocities using a flow-matching objective. To keep the output faithful to the text prompt and prevent duplicate generations, we incorporate a regularization term based on the phase components of the reference video's Fourier transform, projecting the phase angles onto a smooth manifold to suppress high-frequency artifacts. Experiments on benchmarks reveal that RoPECraft outperforms all recently published methods, both qualitatively and quantitatively.
Abstract (translated)
我们提出了一种名为RoPECraft的方法，这是一种无需训练的视频运动转移方法，专门针对扩散变换器。该方法仅通过修改其旋转位置嵌入（RoPE）来操作。首先从参考视频中提取密集光流，并利用由此产生的运动偏移量对RoPE的复指数张量进行变形处理，从而将运动编码到生成过程中。在去噪阶段，我们进一步优化这些嵌入，通过预测速度和目标速度之间的轨迹对齐，采用流动匹配的目标函数来完成这一过程。为了确保输出忠于文本提示并防止重复生成，我们在参考视频傅里叶变换的相位成分基础上引入了一项正则化项，将相位角度投影到平滑流形上以抑制高频伪影。 在基准测试中的实验表明，RoPECraft无论是在定性还是定量评估中都优于所有最近发表的方法。
URL
https://arxiv.org/abs/2505.13344
PDF
