Optical_Flow
-
WAFT: Warping-Alone Field Transforms for Optical Flow
Abstract
We introduce Warping-Alone Field Transforms (WAFT), a simple and effective method for optical flow. WAFT is similar to RAFT but replaces cost volume with high-resolution warping, achieving better accuracy with lower memory cost. This design challenges the conventional wisdom that constructing cost volumes is necessary for strong performance. WAFT is a simple and flexible meta-architecture with minimal inductive biases and reliance on custom designs. Compared with existing methods, WAFT ranks 1st on Spring and KITTI benchmarks, achieves the best zero-shot generalization on KITTI, while being up to 4.1x faster than methods with similar performance. Code and model weights are available at this https URL.
Abstract (translated)
我们介绍了Warping-Alone Field Transforms（WAFT），这是一种简单而有效的方法，用于光流分析。WAFT与RAFT类似，但用高分辨率的图像扭曲操作替代了成本体(volume of cost)，从而在降低内存消耗的同时提高了准确性。这一设计挑战了传统的观点，即构建成本体积是实现高性能所必需的。WAFT是一种简单且灵活的元架构，具有最小的归纳偏差和对定制设计的依赖性。 与现有方法相比，在Spring和KITTI基准测试中，WAFT排名第一，并在KITTI上实现了最佳的零样本泛化能力，同时比性能相似的方法快4.1倍。代码和模型权重可在该链接（此URL）获取。
URL
https://arxiv.org/abs/2506.21526
PDF
-
EndoFlow-SLAM: Real-Time Endoscopic SLAM with Flow-Constrained Gaussian Splatting
2025-06-26 16:06:46Taoyu Wu, Yiyi Miao, Zhuoxiao Li, Haocheng Zhao, Kang Dang, Jionglong Su, Limin Yu, Haoang LiAbstract
Efficient three-dimensional reconstruction and real-time visualization are critical in surgical scenarios such as endoscopy. In recent years, 3D Gaussian Splatting (3DGS) has demonstrated remarkable performance in efficient 3D reconstruction and rendering. Most 3DGS-based Simultaneous Localization and Mapping (SLAM) methods only rely on the appearance constraints for optimizing both 3DGS and camera poses. However, in endoscopic scenarios, the challenges include photometric inconsistencies caused by non-Lambertian surfaces and dynamic motion from breathing affects the performance of SLAM systems. To address these issues, we additionally introduce optical flow loss as a geometric constraint, which effectively constrains both the 3D structure of the scene and the camera motion. Furthermore, we propose a depth regularisation strategy to mitigate the problem of photometric inconsistencies and ensure the validity of 3DGS depth rendering in endoscopic scenes. In addition, to improve scene representation in the SLAM system, we improve the 3DGS refinement strategy by focusing on viewpoints corresponding to Keyframes with suboptimal rendering quality frames, achieving better rendering results. Extensive experiments on the C3VD static dataset and the StereoMIS dynamic dataset demonstrate that our method outperforms existing state-of-the-art methods in novel view synthesis and pose estimation, exhibiting high performance in both static and dynamic surgical scenes. The source code will be publicly available upon paper acceptance.
Abstract (translated)
在手术场景（如内窥镜）中，高效的三维重建和实时可视化至关重要。近年来，3D高斯点阵法 (3DGaussian Splatting, 3DGS) 在高效三维重建和渲染方面表现出了卓越的性能。大多数基于3DGS的同时定位与地图构建(Simultaneous Localization and Mapping, SLAM)方法仅依赖于外观约束来优化3DGS以及相机姿态。然而，在内窥镜场景中，非朗伯表面引起的光度不一致性和呼吸导致的动态运动会对SLAM系统的性能产生挑战。 为了解决这些问题，我们额外引入了光学流损失作为几何约束，该约束有效限制了场景中的三维结构和摄像机移动。此外，我们提出了一种深度正则化策略来缓解光度不一致性的问题，并确保3DGS在内窥镜场景中渲染的深度的有效性。为了提高SLAM系统中的场景表示，我们改进了3DGS优化策略，重点关注关键帧对应的子优图像视点，从而获得更好的渲染效果。 在C3VD静态数据集和StereoMIS动态数据集上的广泛实验表明，我们的方法在新视角合成和姿态估计方面超越现有的最先进的技术，在静态和动态手术场景中都表现出色。论文被接受后，源代码将公开发布。
URL
https://arxiv.org/abs/2506.21420
PDF
-
Feature Hallucination for Self-supervised Action Recognition
2025-06-25 11:50:23Lei Wang, Piotr KoniuszAbstract
Understanding human actions in videos requires more than raw pixel analysis; it relies on high-level semantic reasoning and effective integration of multimodal features. We propose a deep translational action recognition framework that enhances recognition accuracy by jointly predicting action concepts and auxiliary features from RGB video frames. At test time, hallucination streams infer missing cues, enriching feature representations without increasing computational overhead. To focus on action-relevant regions beyond raw pixels, we introduce two novel domain-specific descriptors. Object Detection Features (ODF) aggregate outputs from multiple object detectors to capture contextual cues, while Saliency Detection Features (SDF) highlight spatial and intensity patterns crucial for action recognition. Our framework seamlessly integrates these descriptors with auxiliary modalities such as optical flow, Improved Dense Trajectories, skeleton data, and audio cues. It remains compatible with state-of-the-art architectures, including I3D, AssembleNet, Video Transformer Network, FASTER, and recent models like VideoMAE V2 and InternVideo2. To handle uncertainty in auxiliary features, we incorporate aleatoric uncertainty modeling in the hallucination step and introduce a robust loss function to mitigate feature noise. Our multimodal self-supervised action recognition framework achieves state-of-the-art performance on multiple benchmarks, including Kinetics-400, Kinetics-600, and Something-Something V2, demonstrating its effectiveness in capturing fine-grained action dynamics.
Abstract (translated)
理解视频中的人类动作不仅仅是对原始像素的分析；它还需要高层次的语义推理和有效整合多模态特征。我们提出了一种深度转换式动作识别框架，通过共同预测RGB视频帧的动作概念和辅助特征来提高识别准确性。在测试时，幻觉流可以推断缺失的线索，丰富特征表示而不增加计算开销。为了超越原始像素关注与动作相关区域，我们引入了两种新颖的领域特定描述符：物体检测特征（ODF）聚合来自多个对象检测器的输出以捕捉上下文线索；而注意力检测特征（SDF）则突出空间和强度模式，这些模式对动作识别至关重要。我们的框架可以无缝地将这些描述符与光流、改进的密集轨迹、骨骼数据以及音频提示等辅助模态整合在一起，并且兼容包括I3D、AssembleNet、视频变换网络（Video Transformer Network）、FASTER以及最近的模型如VideoMAE V2和InternVideo2在内的最先进架构。为了处理辅助特征中的不确定性，我们在幻觉步骤中引入了经验不确定性建模，并提出了一种稳健的损失函数以减少特征噪声。我们的多模态自监督动作识别框架在Kinetics-400、Kinetics-600以及Something-Something V2等多个基准测试上取得了最先进的性能，证明其能够有效捕捉细粒度的动作动态变化。
URL
https://arxiv.org/abs/2506.20342
PDF
-
Online camera-pose-free stereo endoscopic tissue deformation recovery with tissue-invariant vision-biomechanics consistency
2025-06-24 07:32:57Jiahe Chen, Naoki Tomii, Ichiro Sakuma, Etsuko KobayashiAbstract
Tissue deformation recovery based on stereo endoscopic images is crucial for tool-tissue interaction analysis and benefits surgical navigation and autonomous soft tissue manipulation. Previous research suffers from the problems raised from camera motion, occlusion, large tissue deformation, lack of tissue-specific biomechanical priors, and reliance on offline processing. Unlike previous studies where the tissue geometry and deformation are represented by 3D points and displacements, the proposed method models tissue geometry as the 3D point and derivative map and tissue deformation as the 3D displacement and local deformation map. For a single surface point, 6 parameters are used to describe its rigid motion and 3 parameters for its local deformation. The method is formulated under the camera-centric setting, where all motions are regarded as the scene motion with respect to the camera. Inter-frame alignment is realized by optimizing the inter-frame deformation, making it unnecessary to estimate camera pose. The concept of the canonical map is introduced to optimize tissue geometry and deformation in an online approach. Quantitative and qualitative experiments were conducted using in vivo and ex vivo laparoscopic datasets. With the inputs of depth and optical flow, the method stably models tissue geometry and deformation even when the tissue is partially occluded or moving outside the field of view. Results show that the 3D reconstruction accuracy in the non-occluded and occluded areas reaches 0.37$\pm$0.27 mm and 0.39$\pm$0.21 mm in terms of surface distance, respectively. The method can also estimate surface strain distribution during various manipulations as an extra modality for mechanical-based analysis.
Abstract (translated)
基于立体内窥镜图像的组织变形恢复对于工具-组织交互分析至关重要，能够改善手术导航和自主软组织操作。然而，以往的研究面临摄像机运动、遮挡、组织大范围变形、缺乏特定生物力学先验知识以及依赖于离线处理等问题。不同于之前研究中用3D点和位移来表示组织几何形状和变形的方式，所提出的方法将组织的几何结构建模为3D点及其导数图，并且将组织的变形建模为3D位移及局部变形图。对于单个表面点，使用6个参数描述其刚性运动，并用3个参数描述其局部变形。该方法以摄像机为中心设置进行构建，在此框架下所有运动都被视为相对于摄像机的场景运动。通过优化帧间变形实现帧间对齐，从而无需估计摄像机姿态。引入“标准映射”的概念来在线优化组织几何形状和变形。 使用体内和体外腹腔镜数据集进行了定量和定性实验，在输入深度信息和光流的情况下，即使在部分遮挡或运动超出视野范围时，该方法也能稳定地建模组织的几何形状和变形。结果显示，在未被遮挡和被遮挡区域中的3D重建准确度分别为0.37±0.27毫米和0.39±0.21毫米（以表面距离为衡量标准）。此外，该方法还可以估算在各种操作期间的表面应变分布，作为基于力学分析的一个额外模态。
URL
https://arxiv.org/abs/2506.19388
PDF
-
Multimodal Fusion SLAM with Fourier Attention
2025-06-22 23:44:07Youjie Zhou, Guofeng Mei, Yiming Wang, Yi Wan, Fabio PoiesiAbstract
Visual SLAM is particularly challenging in environments affected by noise, varying lighting conditions, and darkness. Learning-based optical flow algorithms can leverage multiple modalities to address these challenges, but traditional optical flow-based visual SLAM approaches often require significant computational this http URL overcome this limitation, we propose FMF-SLAM, an efficient multimodal fusion SLAM method that utilizes fast Fourier transform (FFT) to enhance the algorithm efficiency. Specifically, we introduce a novel Fourier-based self-attention and cross-attention mechanism to extract features from RGB and depth signals. We further enhance the interaction of multimodal features by incorporating multi-scale knowledge distillation across modalities. We also demonstrate the practical feasibility of FMF-SLAM in real-world scenarios with real time performance by integrating it with a security robot by fusing with a global positioning module GNSS-RTK and global Bundle Adjustment. Our approach is validated using video sequences from TUM, TartanAir, and our real-world datasets, showcasing state-of-the-art performance under noisy, varying lighting, and dark this http URL code and datasets are available at this https URL.
Abstract (translated)
视觉SLAM在受噪声影响、光照变化以及黑暗环境下的表现特别具有挑战性。基于学习的光流算法可以通过利用多模态信息来应对这些挑战，但传统的基于光流的视觉SLAM方法往往需要大量的计算资源才能克服这一限制。为了克服这个局限性，我们提出了FMF-SLAM（快速傅里叶变换多模态融合SLAM），这是一种高效的多模态融合定位和建图算法，它利用快速傅里叶变换来提高算法效率。具体来说，我们引入了一种新的基于傅里叶的自注意力机制以及跨模态注意机制，用于从RGB信号和深度信号中提取特征。此外，通过在各尺度上进行跨模态知识蒸馏，我们也加强了多模态特征之间的交互作用。 为了展示FMF-SLAM在实际场景中的可行性并保持实时性能，我们将其与一个安防机器人集成，并融合了全球定位模块GNSS-RTK和全局捆绑调整。我们的方法通过使用TUM、TartanAir以及我们自己的真实数据集进行了验证，在嘈杂环境、光照变化大及黑暗环境下均表现出领先于现有技术的性能。 有关代码和数据集可在此网址获取：[请提供具体的URL]（假设原文本中的 "this https URL" 应为实际链接地址）。
URL
https://arxiv.org/abs/2506.18204
PDF
-
EndoMUST: Monocular Depth Estimation for Robotic Endoscopy via End-to-end Multi-step Self-supervised Training
2025-06-19 04:31:59Liangjing Shao, Linxin Bai, Chenkang Du, Xinrong ChenAbstract
Monocular depth estimation and ego-motion estimation are significant tasks for scene perception and navigation in stable, accurate and efficient robot-assisted endoscopy. To tackle lighting variations and sparse textures in endoscopic scenes, multiple techniques including optical flow, appearance flow and intrinsic image decomposition have been introduced into the existing methods. However, the effective training strategy for multiple modules are still critical to deal with both illumination issues and information interference for self-supervised depth estimation in endoscopy. Therefore, a novel framework with multistep efficient finetuning is proposed in this work. In each epoch of end-to-end training, the process is divided into three steps, including optical flow registration, multiscale image decomposition and multiple transformation alignments. At each step, only the related networks are trained without interference of irrelevant information. Based on parameter-efficient finetuning on the foundation model, the proposed method achieves state-of-the-art performance on self-supervised depth estimation on SCARED dataset and zero-shot depth estimation on Hamlyn dataset, with 4\%$\sim$10\% lower error. The evaluation code of this work has been published on this https URL.
Abstract (translated)
单目深度估计和自运动估计对于稳定、准确且高效的机器人辅助内窥镜感知与导航任务至关重要。为了应对内窥镜场景中的光照变化和纹理稀疏问题，现有的方法已经引入了包括光流（optical flow）、外观流（appearance flow）和内在图像分解（intrinsic image decomposition）在内的多种技术。然而，对于自监督深度估计而言，在处理照明问题与信息干扰的同时，有效的多模块训练策略仍然至关重要。因此，本文提出了一种具有多步骤高效微调的新框架。 在端到端训练的每个周期中，过程被分为三个阶段：光流配准（optical flow registration）、多尺度图像分解（multiscale image decomposition）和多重变换对齐（multiple transformation alignments）。在每一阶段中，只有相关的网络会被训练，并且不会受到无关信息的干扰。基于基础模型上的参数高效微调方法，该方法在SCARED数据集上实现了自监督深度估计领域的最先进性能，在Hamlyn数据集上也实现了零样本深度估计方面的领先表现，误差降低了4%到10%。 本文的工作代码已发布在此[URL]（请将方括号内的文本替换为实际的网址链接）。 这种描述提供了对所提出方法技术细节和优势的一个清晰概览，并强调了其在特定数据集上的性能提升。
URL
https://arxiv.org/abs/2506.16017
PDF
-
MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution
2025-06-17 13:35:06Zhiwen Shao, Yifan Cheng, Feiran Li, Yong Zhou, Xuequan Lu, Yuan Xie, Lizhuang MaAbstract
Facial micro-expression recognition (MER) is a challenging problem, due to transient and subtle micro-expression (ME) actions. Most existing methods depend on hand-crafted features, key frames like onset, apex, and offset frames, or deep networks limited by small-scale and low-diversity datasets. In this paper, we propose an end-to-end micro-action-aware deep learning framework with advantages from transformer, graph convolution, and vanilla convolution. In particular, we propose a novel F5C block composed of fully-connected convolution and channel correspondence convolution to directly extract local-global features from a sequence of raw frames, without the prior knowledge of key frames. The transformer-style fully-connected convolution is proposed to extract local features while maintaining global receptive fields, and the graph-style channel correspondence convolution is introduced to model the correlations among feature patterns. Moreover, MER, optical flow estimation, and facial landmark detection are jointly trained by sharing the local-global features. The two latter tasks contribute to capturing facial subtle action information for MER, which can alleviate the impact of insufficient training data. Extensive experiments demonstrate that our framework (i) outperforms the state-of-the-art MER methods on CASME II, SAMM, and SMIC benchmarks, (ii) works well for optical flow estimation and facial landmark detection, and (iii) can capture facial subtle muscle actions in local regions associated with MEs. The code is available at this https URL.
Abstract (translated)
面部微表情识别（MER）是一个具有挑战性的问题，由于短暂和细微的微表情动作。大多数现有方法依赖于手工设计特征、关键帧如起始点、顶峰点及结束点，或受制于小规模且低多样性数据集的深度网络。在本文中，我们提出了一种结合了变压器、图卷积以及普通卷积优势的端到端微动作感知深度学习框架。 特别地，我们提出了一种新颖的F5C（全连接卷积与通道对应卷积组成）模块，可以直接从一系列原始帧序列提取局部-全局特征，而无需事先知道关键帧的信息。提出的变压器风格的全连接卷积旨在提取局部特征的同时保持全局感受野，图样式通道对应卷积则被引入以建模特征模式之间的相关性。 此外，MER、光流估计和面部标志点检测通过共享局部-全局特征进行联合训练。后两项任务有助于捕捉对微表情识别有用的细微面部动作信息，从而缓解因训练数据不足造成的影响。 广泛的实验表明，我们的框架（i）在CASME II、SAMM 和 SMIC 评估基准上超越了最先进的MER方法，（ii）对于光流估计和面部标志点检测表现出良好的性能，并且（iii）能够捕捉与微表情相关的局部区域内的细微肌肉动作。代码可在[此处](https://this https URL)获取。
URL
https://arxiv.org/abs/2506.14511
PDF
-
Inference-Time Gaze Refinement for Micro-Expression Recognition: Enhancing Event-Based Eye Tracking with Motion-Aware Post-Processing
2025-06-14 14:48:11Nuwan Bandara, Thivya Kandappu, Archan MisraAbstract
Event-based eye tracking holds significant promise for fine-grained cognitive state inference, offering high temporal resolution and robustness to motion artifacts, critical features for decoding subtle mental states such as attention, confusion, or fatigue. In this work, we introduce a model-agnostic, inference-time refinement framework designed to enhance the output of existing event-based gaze estimation models without modifying their architecture or requiring retraining. Our method comprises two key post-processing modules: (i) Motion-Aware Median Filtering, which suppresses blink-induced spikes while preserving natural gaze dynamics, and (ii) Optical Flow-Based Local Refinement, which aligns gaze predictions with cumulative event motion to reduce spatial jitter and temporal discontinuities. To complement traditional spatial accuracy metrics, we propose a novel Jitter Metric that captures the temporal smoothness of predicted gaze trajectories based on velocity regularity and local signal complexity. Together, these contributions significantly improve the consistency of event-based gaze signals, making them better suited for downstream tasks such as micro-expression analysis and mind-state decoding. Our results demonstrate consistent improvements across multiple baseline models on controlled datasets, laying the groundwork for future integration with multimodal affect recognition systems in real-world environments.
Abstract (translated)
基于事件的眼动追踪在细粒度认知状态推断方面具有巨大的潜力，它提供了高时间分辨率和对运动伪影的鲁棒性，这是解码注意力、困惑或疲劳等微妙心理状态的关键特性。在这项工作中，我们提出了一种模型无关的推理时增强框架，旨在通过不修改现有基于事件的眼动估计模型架构且无需重新训练的方式来提升这些模型的输出效果。我们的方法包括两个关键后处理模块：(i) 运动感知中值滤波器，它抑制眨眼引起的尖峰信号，同时保持自然眼动动态；(ii) 基于光流的地方精化，将眼动预测与累积事件运动对齐，以减少空间抖动和时间不连续性。为了补充传统的空间准确性度量标准，我们提出了一种新颖的抖动度量指标，该指标基于速度规律性和局部信号复杂度捕捉预测的眼动轨迹的时间平滑度。这些贡献共同提高了基于事件眼动信号的一致性，使其更适合微表情分析和心理状态解码等下游任务。我们的结果在多个基准模型上显示出一致的改进效果，并且是在控制数据集上的成果，为未来与多模式情感识别系统在现实环境中的集成奠定了基础。
URL
https://arxiv.org/abs/2506.12524
PDF
-
MambaVSR: Content-Aware Scanning State Space Model for Video Super-Resolution
2025-06-13 13:22:28Linfeng He, Meiqin Liu, Qi Tang, Chao Yao, Yao ZhaoAbstract
Video super-resolution (VSR) faces critical challenges in effectively modeling non-local dependencies across misaligned frames while preserving computational efficiency. Existing VSR methods typically rely on optical flow strategies or transformer architectures, which struggle with large motion displacements and long video sequences. To address this, we propose MambaVSR, the first state-space model framework for VSR that incorporates an innovative content-aware scanning mechanism. Unlike rigid 1D sequential processing in conventional vision Mamba methods, our MambaVSR enables dynamic spatiotemporal interactions through the Shared Compass Construction (SCC) and the Content-Aware Sequentialization (CAS). Specifically, the SCC module constructs intra-frame semantic connectivity graphs via efficient sparse attention and generates adaptive spatial scanning sequences through spectral clustering. Building upon SCC, the CAS module effectively aligns and aggregates non-local similar content across multiple frames by interleaving temporal features along the learned spatial order. To bridge global dependencies with local details, the Global-Local State Space Block (GLSSB) synergistically integrates window self-attention operations with SSM-based feature propagation, enabling high-frequency detail recovery under global dependency guidance. Extensive experiments validate MambaVSR's superiority, outperforming the Transformer-based method by 0.58 dB PSNR on the REDS dataset with 55% fewer parameters.
Abstract (translated)
视频超分辨率（VSR）面临着一个关键挑战，即如何有效地建模不同步帧之间的非局部依赖关系，同时保持计算效率。现有的VSR方法通常依赖于光学流策略或变压器架构，这些方法在处理大运动位移和长视频序列时存在困难。为了解决这些问题，我们提出了一种名为MambaVSR的框架，这是首个基于状态空间模型（SSM）的VSR框架，它集成了创新的内容感知扫描机制。 与传统的视觉Mamba方法中刚性的1D顺序处理不同，我们的MambaVSR通过共享罗盘构造（SCC）和内容感知序列化（CAS）实现了动态时空交互。具体来说，SCC模块通过高效的稀疏注意力构建了帧内语义连接图，并利用谱聚类生成自适应空间扫描序列。在此基础上，CAS模块通过在学习到的空间顺序中交错时间特征来有效地对多个帧中的非局部相似内容进行对齐和聚合。 为了将全局依赖关系与局部细节相结合，我们提出了全局-局部状态空间块（GLSSB），该模块巧妙地结合了窗口自我注意力操作和基于SSM的特性传播，使得在全局依赖性的指导下能够恢复高频细节。 广泛的实验验证了MambaVSR的优势，在RED数据集上超越了基于Transformer的方法0.58 dB PSNR，同时参数减少了55%。
URL
https://arxiv.org/abs/2506.11768
PDF
-
Post-Training Quantization for Video Matting
2025-06-12 15:57:14Tianrui Zhu, Houyuan Chen, Ruihao Gong, Michele Magno, Haotong Qin, Kai ZhangAbstract
Video matting is crucial for applications such as film production and virtual reality, yet deploying its computationally intensive models on resource-constrained devices presents challenges. Quantization is a key technique for model compression and acceleration. As an efficient approach, Post-Training Quantization (PTQ) is still in its nascent stages for video matting, facing significant hurdles in maintaining accuracy and temporal coherence. To address these challenges, this paper proposes a novel and general PTQ framework specifically designed for video matting models, marking, to the best of our knowledge, the first systematic attempt in this domain. Our contributions include: (1) A two-stage PTQ strategy that combines block-reconstruction-based optimization for fast, stable initial quantization and local dependency capture, followed by a global calibration of quantization parameters to minimize accuracy loss. (2) A Statistically-Driven Global Affine Calibration (GAC) method that enables the network to compensate for cumulative statistical distortions arising from factors such as neglected BN layer effects, even reducing the error of existing PTQ methods on video matting tasks up to 20%. (3) An Optical Flow Assistance (OFA) component that leverages temporal and semantic priors from frames to guide the PTQ process, enhancing the model's ability to distinguish moving foregrounds in complex scenes and ultimately achieving near full-precision performance even under ultra-low-bit quantization. Comprehensive quantitative and visual results show that our PTQ4VM achieves the state-of-the-art accuracy performance across different bit-widths compared to the existing quantization methods. We highlight that the 4-bit PTQ4VM even achieves performance close to the full-precision counterpart while enjoying 8x FLOP savings.
Abstract (translated)
视频抠图（Video Matting）在电影制作和虚拟现实等领域中至关重要，但要在计算资源有限的设备上部署其计算密集型模型存在挑战。量化是一种关键的技术，用于模型压缩和加速。作为一种高效的途径，后训练量化（Post-Training Quantization, PTQ）在视频抠图领域尚处于起步阶段，并且面临着保持精度和时间一致性的重要障碍。为了解决这些问题，本文提出了一种专为视频抠图模型设计的新颖而通用的PTQ框架，据我们所知，这是该领域的首次系统性尝试。我们的贡献包括： 1. 一种两阶段PTQ策略，结合基于块重构的优化方法进行快速、稳定的初步量化和局部依赖项捕获，随后通过全局校准量化参数来最小化精度损失。 2. 统计驱动全局仿射校准（GAC）方法，使网络能够补偿由于忽略BN层效应等因素产生的累积统计失真，甚至将现有PTQ方法在视频抠图任务中的误差减少高达20%。 3. 光流辅助（OFA）组件，利用帧间的时间和语义先验引导量化过程，增强模型区分复杂场景中移动前景的能力，并最终实现即使在超低位宽量化下也能达到接近全精度的表现。 综合定量和视觉结果表明，我们的PTQ4VM方法在不同比特宽度下的准确性表现优于现有的量化方法。特别值得注意的是，4位的PTQ4VM甚至实现了与全精度模型相当的性能，同时节省了8倍的计算量（FLOP）。
URL
https://arxiv.org/abs/2506.10840
PDF
-
UFM: A Simple Path towards Unified Dense Correspondence with Flow
2025-06-10 22:32:13Yuchen Zhang, Nikhil Keetha, Chenwei Lyu, Bhuvan Jhamb, Yutian Chen, Yuheng Qiu, Jay Karhade, Shreyas Jha, Yaoyu Hu, Deva Ramanan, Sebastian Scherer, Wenshan WangAbstract
Dense image correspondence is central to many applications, such as visual odometry, 3D reconstruction, object association, and re-identification. Historically, dense correspondence has been tackled separately for wide-baseline scenarios and optical flow estimation, despite the common goal of matching content between two images. In this paper, we develop a Unified Flow & Matching model (UFM), which is trained on unified data for pixels that are co-visible in both source and target images. UFM uses a simple, generic transformer architecture that directly regresses the (u,v) flow. It is easier to train and more accurate for large flows compared to the typical coarse-to-fine cost volumes in prior work. UFM is 28% more accurate than state-of-the-art flow methods (Unimatch), while also having 62% less error and 6.7x faster than dense wide-baseline matchers (RoMa). UFM is the first to demonstrate that unified training can outperform specialized approaches across both domains. This result enables fast, general-purpose correspondence and opens new directions for multi-modal, long-range, and real-time correspondence tasks.
Abstract (translated)
密集图像对应关系在许多应用中至关重要，例如视觉测距、三维重建、物体关联和重新识别。历史上，对于宽基线场景下的稠密对应关系与光流估计问题通常是分开处理的，尽管它们都旨在匹配两张图片之间的内容。在这篇文章中，我们开发了一种统一流动及匹配模型（UFM），该模型在源图像和目标图像中共视像素上进行统一流数据训练。UFM使用了一个简单的、通用的变压器架构，直接回归(u,v)流。相比先前工作中的典型粗到细成本体积方法，UFM更容易训练，并且对于大范围的流动更准确。 与最先进的流方法（Unimatch）相比，UFM精确度高出28%；同时，与稠密宽基线匹配器（RoMa）相比，它的误差减少了62%，运行速度提高了6.7倍。UFM首次证明了统一训练方法可以在两个领域内超越专业化的方法。这一成果使得快速、通用的对应关系成为可能，并为跨模态、长距离和实时对应的任务开辟了新的方向。
URL
https://arxiv.org/abs/2506.09278
PDF
-
Princeton365: A Diverse Dataset with Accurate Camera Pose
2025-06-10 17:57:00Karhan Kayan, Stamatis Alexandropoulos, Rishabh Jain, Yiming Zuo, Erich Liang, Jia DengAbstract
We introduce Princeton365, a large-scale diverse dataset of 365 videos with accurate camera pose. Our dataset bridges the gap between accuracy and data diversity in current SLAM benchmarks by introducing a novel ground truth collection framework that leverages calibration boards and a 360-camera. We collect indoor, outdoor, and object scanning videos with synchronized monocular and stereo RGB video outputs as well as IMU. We further propose a new scene scale-aware evaluation metric for SLAM based on the the optical flow induced by the camera pose estimation error. In contrast to the current metrics, our new metric allows for comparison between the performance of SLAM methods across scenes as opposed to existing metrics such as Average Trajectory Error (ATE), allowing researchers to analyze the failure modes of their methods. We also propose a challenging Novel View Synthesis benchmark that covers cases not covered by current NVS benchmarks, such as fully non-Lambertian scenes with 360-degree camera trajectories. Please visit this https URL for the dataset, code, videos, and submission.
Abstract (translated)
我们介绍了Princeton365，这是一个包含365个视频的大规模多样化数据集，并且每个视频都具有精确的相机姿态。我们的数据集通过引入一个新颖的真实情况收集框架来弥合当前SLAM基准测试中准确性与数据多样性之间的差距，该框架利用校准板和360度摄像头进行工作。我们采集了室内、室外以及物体扫描视频，包括同步的单目和立体RGB视频输出及IMU（惯性测量单元）信息。 此外，我们还提出了一种新的基于场景尺度感知的SLAM评估指标，该指标是根据相机姿态估计误差所引起的光流来衡量。与现有的指标如平均轨迹错误（ATE）相比，我们的新度量标准允许跨不同场景比较SLAM方法的性能，这使得研究者能够分析其方法的失效模式。 我们还提出了一项具有挑战性的新型视图合成基准测试，该测试涵盖了当前NVS（Novel View Synthesis，新颖视角合成）基准所未涉及的情况，例如全非朗伯场景以及360度相机轨迹。请访问此[链接](https://example.com)获取数据集、代码、视频和提交指南等相关信息。
URL
https://arxiv.org/abs/2506.09035
PDF
-
Spatio-Temporal State Space Model For Efficient Event-Based Optical Flow
2025-06-09 15:51:06Muhammad Ahmed Humais, Xiaoqian Huang, Hussain Sajwani, Sajid Javed, Yahya ZweiriAbstract
Event cameras unlock new frontiers that were previously unthinkable with standard frame-based cameras. One notable example is low-latency motion estimation (optical flow), which is critical for many real-time applications. In such applications, the computational efficiency of algorithms is paramount. Although recent deep learning paradigms such as CNN, RNN, or ViT have shown remarkable performance, they often lack the desired computational efficiency. Conversely, asynchronous event-based methods including SNNs and GNNs are computationally efficient; however, these approaches fail to capture sufficient spatio-temporal information, a powerful feature required to achieve better performance for optical flow estimation. In this work, we introduce Spatio-Temporal State Space Model (STSSM) module along with a novel network architecture to develop an extremely efficient solution with competitive performance. Our STSSM module leverages state-space models to effectively capture spatio-temporal correlations in event data, offering higher performance with lower complexity compared to ViT, CNN-based architectures in similar settings. Our model achieves 4.5x faster inference and 8x lower computations compared to TMA and 2x lower computations compared to EV-FlowNet with competitive performance on the DSEC benchmark. Our code will be available at this https URL
Abstract (translated)
事件相机解锁了传统帧基摄像头无法实现的新领域。一个典型的例子是低延迟运动估计（光学流），这对许多实时应用来说至关重要。在这些应用场景中，算法的计算效率尤为重要。尽管最近的深度学习范式如CNN、RNN或ViT展现了卓越的表现力，但它们往往缺乏所需的计算效率。相反，异步事件基方法包括SNN和GNN虽然计算效率高，但却无法捕捉足够的时空信息，这是实现更好的光学流估计性能的关键特征之一。 在这项工作中，我们引入了时空状态空间模型（STSSM）模块，并开发了一种新的网络架构，以构建一个极其高效的解决方案并具有竞争性的性能。我们的STSSM模块利用状态空间模型有效地捕获事件数据中的时空相关性，在相似设置下与ViT和基于CNN的架构相比，实现了更高性能的同时降低了复杂度。我们模型在DSEC基准测试中达到比TMA快4.5倍的推理速度，并且计算量减少了8倍；相较于EV-FlowNet，计算量减少了一半，同时保持了竞争性的性能。 我们的代码将在以下网址提供：[此链接应由原作者补充]
URL
https://arxiv.org/abs/2506.07878
PDF
-
Flow-Anything: Learning Real-World Optical Flow Estimation from Large-Scale Single-view Images
2025-06-09 13:23:44Yingping Liang, Ying Fu, Yutao Hu, Wenqi Shao, Jiaming Liu, Debing ZhangAbstract
Optical flow estimation is a crucial subfield of computer vision, serving as a foundation for video tasks. However, the real-world robustness is limited by animated synthetic datasets for training. This introduces domain gaps when applied to real-world applications and limits the benefits of scaling up datasets. To address these challenges, we propose \textbf{Flow-Anything}, a large-scale data generation framework designed to learn optical flow estimation from any single-view images in the real world. We employ two effective steps to make data scaling-up promising. First, we convert a single-view image into a 3D representation using advanced monocular depth estimation networks. This allows us to render optical flow and novel view images under a virtual camera. Second, we develop an Object-Independent Volume Rendering module and a Depth-Aware Inpainting module to model the dynamic objects in the 3D representation. These two steps allow us to generate realistic datasets for training from large-scale single-view images, namely \textbf{FA-Flow Dataset}. For the first time, we demonstrate the benefits of generating optical flow training data from large-scale real-world images, outperforming the most advanced unsupervised methods and supervised methods on synthetic datasets. Moreover, our models serve as a foundation model and enhance the performance of various downstream video tasks.
Abstract (translated)
光流估计是计算机视觉中的一个重要子领域，为视频任务提供了基础。然而，由于使用动画合成数据集进行训练，其在现实世界中的鲁棒性受到了限制。这导致了应用到实际场景时出现领域差距，并且阻碍了通过扩大数据集来提升性能的效果。为了应对这些挑战，我们提出了**Flow-Anything**，这是一个大型的数据生成框架，旨在从真实世界的单视图图像中学习光流估计。我们采用了两个有效的步骤来使数据量的扩展变得可行。首先，我们利用先进的单目深度估计网络将一幅单视图图像转换为3D表示形式，这让我们能够通过虚拟相机渲染出光流和新视角下的图像。其次，我们开发了独立于物体的体积渲染模块以及基于深度感知的修复模块来模拟3D表示中的动态对象。这两个步骤使我们能够从大规模的单视图图像中生成用于训练的真实数据集，即**FA-Flow 数据集**。 首次展示了从大规模真实世界图像中生成光流训练数据的好处，在无监督方法和合成数据集上的有监督方法方面均超越了最先进的技术水平。此外，我们的模型作为基础模型增强了各种下游视频任务的性能。
URL
https://arxiv.org/abs/2506.07740
PDF
-
Consistent Video Editing as Flow-Driven Image-to-Video Generation
2025-06-09 12:57:30Ge Wang, Songlin Fan, Hangxu Liu, Quanjian Song, Hewei Wang, Jinfeng XuAbstract
With the prosper of video diffusion models, down-stream applications like video editing have been significantly promoted without consuming much computational cost. One particular challenge in this task lies at the motion transfer process from the source video to the edited one, where it requires the consideration of the shape deformation in between, meanwhile maintaining the temporal consistency in the generated video sequence. However, existing methods fail to model complicated motion patterns for video editing, and are fundamentally limited to object replacement, where tasks with non-rigid object motions like multi-object and portrait editing are largely neglected. In this paper, we observe that optical flows offer a promising alternative in complex motion modeling, and present FlowV2V to re-investigate video editing as a task of flow-driven Image-to-Video (I2V) generation. Specifically, FlowV2V decomposes the entire pipeline into first-frame editing and conditional I2V generation, and simulates pseudo flow sequence that aligns with the deformed shape, thus ensuring the consistency during editing. Experimental results on DAVIS-EDIT with improvements of 13.67% and 50.66% on DOVER and warping error illustrate the superior temporal consistency and sample quality of FlowV2V compared to existing state-of-the-art ones. Furthermore, we conduct comprehensive ablation studies to analyze the internal functionalities of the first-frame paradigm and flow alignment in the proposed method.
Abstract (translated)
随着视频扩散模型的发展，下游应用如视频编辑在不消耗过多计算成本的情况下得到了显著推动。然而，在从源视频到编辑后视频的运动转移过程中，一个特别的挑战在于需要考虑形状变形的同时保持生成视频序列的时间一致性。现有的方法无法模拟复杂的运动模式以进行视频编辑，并且基本局限于对象替换任务中，而像多对象和人像编辑这样涉及非刚性物体运动的任务则被很大程度上忽略了。 在本文中，我们观察到光学流为复杂运动建模提供了有希望的替代方案，并提出了FlowV2V来重新审视视频编辑作为由流动驱动的图像至视频（I2V）生成任务。具体而言，FlowV2V将整个流程分解为第一帧编辑和条件I2V生成，并模拟与变形形状对齐的伪流序列，从而确保编辑过程中的时间一致性。在DAVIS-EDIT数据集上的实验结果表明，与现有的最先进方法相比，FlowV2V在DOVER和扭曲误差方面分别提高了13.67%和50.66%，展示了其更优的时间一致性和样本质量。此外，我们进行了全面的消融研究来分析所提出方法中第一帧范式和流对齐的内部功能。
URL
https://arxiv.org/abs/2506.07713
PDF
-
AllTracker: Efficient Dense Point Tracking at High Resolution
2025-06-08 22:55:06Adam W. Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Sheldon Liang, Wen-Hsuan Chu, Achal Dave, Pavel Tokmakov, Suya You, Rares Ambrus, Katerina Fragkiadaki, Leonidas J. GuibasAbstract
We introduce AllTracker: a model that estimates long-range point tracks by way of estimating the flow field between a query frame and every other frame of a video. Unlike existing point tracking methods, our approach delivers high-resolution and dense (all-pixel) correspondence fields, which can be visualized as flow maps. Unlike existing optical flow methods, our approach corresponds one frame to hundreds of subsequent frames, rather than just the next frame. We develop a new architecture for this task, blending techniques from existing work in optical flow and point tracking: the model performs iterative inference on low-resolution grids of correspondence estimates, propagating information spatially via 2D convolution layers, and propagating information temporally via pixel-aligned attention layers. The model is fast and parameter-efficient (16 million parameters), and delivers state-of-the-art point tracking accuracy at high resolution (i.e., tracking 768x1024 pixels, on a 40G GPU). A benefit of our design is that we can train on a wider set of datasets, and we find that doing so is crucial for top performance. We provide an extensive ablation study on our architecture details and training recipe, making it clear which details matter most. Our code and model weights are available at this https URL .
Abstract (translated)
我们介绍 AllTracker：这是一种通过估算查询帧与视频中每一其他帧之间的流场来估计长距离点轨迹的模型。与现有的点跟踪方法不同，我们的方法提供高分辨率和密集（全像素）对应字段，可以可视化为流图。不同于现有光流方法仅将一个帧与其下一个帧进行配准的方式，我们的方法将一个帧与后续的数百个帧进行关联。 为了完成这一任务，我们开发了一种新的架构，结合了现有的光学流技术和点跟踪技术：模型在低分辨率对应估计网格上执行迭代推理，通过二维卷积层传播空间信息，并通过像素对齐注意层传播时间信息。该模型既快速又参数高效（1600万个参数），并在高分辨率下提供最先进的点跟踪精度（即，在40G GPU上追踪768x1024像素）。我们的设计的一个优势是，我们可以使用更广泛的训练数据集进行训练，并且我们发现这样做对于顶级性能至关重要。 我们在架构细节和训练方案方面进行了详尽的消融研究，清晰地指出了哪些细节最重要。我们的代码和模型权重可以在以下链接获取：[此处插入URL] 。
URL
https://arxiv.org/abs/2506.07310
PDF
-
GoTrack: Generic 6DoF Object Pose Refinement and Tracking
2025-06-08 14:01:47Van Nguyen Nguyen, Christian Forster, Sindi Shkodrani, Vincent Lepetit, Bugra Tekin, Cem Keskin, Tomas HodanAbstract
We introduce GoTrack, an efficient and accurate CAD-based method for 6DoF object pose refinement and tracking, which can handle diverse objects without any object-specific training. Unlike existing tracking methods that rely solely on an analysis-by-synthesis approach for model-to-frame registration, GoTrack additionally integrates frame-to-frame registration, which saves compute and stabilizes tracking. Both types of registration are realized by optical flow estimation. The model-to-frame registration is noticeably simpler than in existing methods, relying only on standard neural network blocks (a transformer is trained on top of DINOv2) and producing reliable pose confidence scores without a scoring network. For the frame-to-frame registration, which is an easier problem as consecutive video frames are typically nearly identical, we employ a light off-the-shelf optical flow model. We demonstrate that GoTrack can be seamlessly combined with existing coarse pose estimation methods to create a minimal pipeline that reaches state-of-the-art RGB-only results on standard benchmarks for 6DoF object pose estimation and tracking. Our source code and trained models are publicly available at this https URL
Abstract (translated)
我们介绍了GoTrack，这是一种基于CAD的高效且精确的方法，用于六自由度（6DoF）物体姿态精炼和跟踪。该方法可以处理各种不同的对象，并且无需针对特定对象进行训练。与现有的仅依赖于合成分析法来进行模型到帧注册的追踪方法不同，GoTrack还集成了帧到帧的注册功能，这能够节省计算资源并稳定追踪过程。这两种类型的注册都是通过光流估计来实现的。 在模型到帧的注册中，与现有方法相比，GoTrack的方法明显更加简单，仅依赖于标准神经网络模块（基于DINOv2训练了一个变压器），并且可以在没有评分网络的情况下产生可靠的姿态置信度得分。对于帧到帧的注册问题，由于连续视频帧通常几乎相同，因此这个问题相对容易解决，我们使用了一种轻量级现成的光流模型来处理。 我们展示了GoTrack可以无缝地与现有的粗略的姿态估计方法结合使用，以创建一条达到六自由度物体姿态估计和追踪标准基准测试最佳效果（仅限RGB输入）的最小化管道。我们的源代码和训练好的模型可以在以下链接中公开获取：[提供链接] (请将此占位符替换为实际提供的URL)
URL
https://arxiv.org/abs/2506.07155
PDF
-
EV-LayerSegNet: Self-supervised Motion Segmentation using Event Cameras
2025-06-07 00:07:59Youssef Farah, Federico Paredes-Vall\'es, Guido De Croon, Muhammad Ahmed Humais, Hussain Sajwani, Yahya ZweiriAbstract
Event cameras are novel bio-inspired sensors that capture motion dynamics with much higher temporal resolution than traditional cameras, since pixels react asynchronously to brightness changes. They are therefore better suited for tasks involving motion such as motion segmentation. However, training event-based networks still represents a difficult challenge, as obtaining ground truth is very expensive, error-prone and limited in frequency. In this article, we introduce EV-LayerSegNet, a self-supervised CNN for event-based motion segmentation. Inspired by a layered representation of the scene dynamics, we show that it is possible to learn affine optical flow and segmentation masks separately, and use them to deblur the input events. The deblurring quality is then measured and used as self-supervised learning loss. We train and test the network on a simulated dataset with only affine motion, achieving IoU and detection rate up to 71% and 87% respectively.
Abstract (translated)
事件相机是一种新型的仿生传感器，与传统相机相比，它们能够以更高的时间分辨率捕捉运动动态，因为像素会异步地响应亮度变化。因此，对于涉及运动的任务（如运动分割）而言，事件相机更为适用。然而，训练基于事件的网络仍然是一项具有挑战性的任务，因为获取真实标签非常昂贵、容易出错且频率受限。在本文中，我们介绍了EV-LayerSegNet，这是一种自监督CNN，用于基于事件的运动分割。受场景动态分层表示的启发，我们展示了可以单独学习仿射光流和分割掩码，并使用它们来去除输入事件中的模糊。然后测量去模糊的质量并将其用作自监督学习损失。我们在仅包含仿射运动的模拟数据集上训练和测试网络，分别实现了高达71%的IoU和87%的检测率。
URL
https://arxiv.org/abs/2506.06596
PDF
-
3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model
2025-06-06 16:00:31Hongyan Zhi, Peihao Chen, Siyuan Zhou, Yubo Dong, Quanxi Wu, Lei Han, Mingkui TanAbstract
Manipulation has long been a challenging task for robots, while humans can effortlessly perform complex interactions with objects, such as hanging a cup on the mug rack. A key reason is the lack of a large and uniform dataset for teaching robots manipulation skills. Current robot datasets often record robot action in different action spaces within a simple scene. This hinders the robot to learn a unified and robust action representation for different robots within diverse scenes. Observing how humans understand a manipulation task, we find that understanding how the objects should move in the 3D space is a critical clue for guiding actions. This clue is embodiment-agnostic and suitable for both humans and different robots. Motivated by this, we aim to learn a 3D flow world model from both human and robot manipulation data. This model predicts the future movement of the interacting objects in 3D space, guiding action planning for manipulation. Specifically, we synthesize a large-scale 3D optical flow dataset, named ManiFlow-110k, through a moving object auto-detect pipeline. A video diffusion-based world model then learns manipulation physics from these data, generating 3D optical flow trajectories conditioned on language instructions. With the generated 3D object optical flow, we propose a flow-guided rendering mechanism, which renders the predicted final state and leverages GPT-4o to assess whether the predicted flow aligns with the task description. This equips the robot with a closed-loop planning ability. Finally, we consider the predicted 3D optical flow as constraints for an optimization policy to determine a chunk of robot actions for manipulation. Extensive experiments demonstrate strong generalization across diverse robotic manipulation tasks and reliable cross-embodiment adaptation without hardware-specific training.
Abstract (translated)
机器人操纵长期以来一直是一个具有挑战性的任务，而人类可以轻松地在物体上执行复杂的交互操作，例如将杯子挂在杯架上。这主要是因为缺乏一个大型且统一的数据集来教授机器人的操控技能。目前的机器人数据集通常记录简单场景中不同动作空间内的机器人行为，这阻碍了机器人学习适用于各种环境中不同机器人的统一和稳健的动作表示能力。观察人类如何理解操纵任务时，我们发现了解物体在三维空间中的运动方式是指导行动的关键线索。这一线索不依赖于特定的身体形态，并且适合用于既有人类也包括不同的机器人。受此启发，我们的目标是从人类和机器人的操作数据中学习一个3D流世界模型。该模型预测交互对象在未来三维空间的移动情况，从而为操纵任务中的动作规划提供指导。 具体而言，我们通过一个自动检测移动物体的数据管道合成了一个大规模的3D光流数据集，名为ManiFlow-110k。然后，一种基于视频扩散的世界模型从这些数据中学习操作物理原理，并根据语言指令生成3D光流轨迹。利用生成的3D对象光流，我们提出了一种流引导渲染机制，该机制预测了最终状态并利用GPT-4o评估预测流是否与任务描述一致。这赋予机器人一个闭环规划能力。最后，我们将预测的3D光流视为优化策略中的约束条件以确定一系列机器人的操作动作用于执行操纵任务。 广泛的实验表明，在各种不同的机器人操控任务中具有强大的泛化能力和无需特定硬件训练的可靠跨身体形态适应性。
URL
https://arxiv.org/abs/2506.06199
PDF
-
Dy3DGS-SLAM: Monocular 3D Gaussian Splatting SLAM for Dynamic Environments
2025-06-06 10:43:41Mingrui Li, Yiming Zhou, Hongxing Zhou, Xinggang Hu, Florian Roemer, Hongyu Wang, Ahmad OsmanAbstract
Current Simultaneous Localization and Mapping (SLAM) methods based on Neural Radiance Fields (NeRF) or 3D Gaussian Splatting excel in reconstructing static 3D scenes but struggle with tracking and reconstruction in dynamic environments, such as real-world scenes with moving elements. Existing NeRF-based SLAM approaches addressing dynamic challenges typically rely on RGB-D inputs, with few methods accommodating pure RGB input. To overcome these limitations, we propose Dy3DGS-SLAM, the first 3D Gaussian Splatting (3DGS) SLAM method for dynamic scenes using monocular RGB input. To address dynamic interference, we fuse optical flow masks and depth masks through a probabilistic model to obtain a fused dynamic mask. With only a single network iteration, this can constrain tracking scales and refine rendered geometry. Based on the fused dynamic mask, we designed a novel motion loss to constrain the pose estimation network for tracking. In mapping, we use the rendering loss of dynamic pixels, color, and depth to eliminate transient interference and occlusion caused by dynamic objects. Experimental results demonstrate that Dy3DGS-SLAM achieves state-of-the-art tracking and rendering in dynamic environments, outperforming or matching existing RGB-D methods.
Abstract (translated)
当前基于神经辐射场（NeRF）或三维高斯点阵（3D Gaussian Splatting）的同步定位与建图（SLAM）方法在重建静态三维场景方面表现出色，但在处理包含移动元素的真实世界动态环境时却显得力不从心。现有的针对动态挑战进行优化的基于NeRF的SLAM方法大多依赖于RGB-D输入，而能够适应纯RGB输入的方法则相对较少。为了克服这些限制，我们提出了Dy3DGS-SLAM，这是首个专门用于处理动态场景并采用单目RGB输入的三维高斯点阵（3DGS）SLAM方法。 为了解决动态干扰问题，我们通过概率模型融合光流掩码和深度掩码以获得一个融合后的动态掩码。仅需一次网络迭代即可借助这一融合后的动态掩码限制跟踪尺度并优化渲染几何形状。在设计约束姿态估计网络进行跟踪的新型运动损失时，我们亦基于该融合后的动态掩码进行了创新。 在建图过程中，我们将动像素的渲染损失、颜色和深度相结合以消除由动态对象引起的瞬态干扰与遮挡问题。实验结果表明，在处理动态环境时，Dy3DGS-SLAM不仅能够实现最先进的跟踪与渲染性能，并且超越或匹敌现有的RGB-D方法。
URL
https://arxiv.org/abs/2506.05965
PDF
