Optical_Flow
-
ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors
2026-03-04 17:58:04Zihao Huang, Tianqi Liu, Zhaoxi Chen, Shaocong Xu, Saining Zhang, Lixing Xiao, Zhiguo Cao, Wei Li, Hao Zhao, Ziwei LiuAbstract
Synthesizing physically plausible articulated human-object interactions (HOI) without 3D/4D supervision remains a fundamental challenge. While recent zero-shot approaches leverage video diffusion models to synthesize human-object interactions, they are largely confined to rigid-object manipulation and lack explicit 4D geometric reasoning. To bridge this gap, we formulate articulated HOI synthesis as a 4D reconstruction problem from monocular video priors: given only a video generated by a diffusion model, we reconstruct a full 4D articulated scene without any 3D supervision. This reconstruction-based approach treats the generated 2D video as supervision for an inverse rendering problem, recovering geometrically consistent and physically plausible 4D scenes that naturally respect contact, articulation, and temporal coherence. We introduce ArtHOI, the first zero-shot framework for articulated human-object interaction synthesis via 4D reconstruction from video priors. Our key designs are: 1) Flow-based part segmentation: leveraging optical flow as a geometric cue to disentangle dynamic from static regions in monocular video; 2) Decoupled reconstruction pipeline: joint optimization of human motion and object articulation is unstable under monocular ambiguity, so we first recover object articulation, then synthesize human motion conditioned on the reconstructed object states. ArtHOI bridges video-based generation and geometry-aware reconstruction, producing interactions that are both semantically aligned and physically grounded. Across diverse articulated scenes (e.g., opening fridges, cabinets, microwaves), ArtHOI significantly outperforms prior methods in contact accuracy, penetration reduction, and articulation fidelity, extending zero-shot interaction synthesis beyond rigid manipulation through reconstruction-informed synthesis.
Abstract (translated)
合成物理上合理的人体与物体交互（HOI）场景，而无需3D或4D的监督数据，仍然是一个基本挑战。尽管最近的一些零样本方法利用视频扩散模型来合成人体与物体之间的互动，但这些方法主要局限于对刚性对象的操作，并缺乏显式的四维几何推理能力。为了弥合这一差距，我们把关节式HOI生成定义为一项单目视频先验下的四维重建任务：仅使用由扩散模型生成的视频作为输入，无需任何3D监督数据，即可重构出完整的四维关节场景。基于这种重建的方法将生成的2D视频视为反向渲染问题中的监督信号，在保证几何一致性和物理合理性的前提下尊重接触、关节和时间连续性。 我们引入了ArtHOI——第一个通过从视频先验中进行4D重建来实现零样本关节式人体与物体交互合成的框架。我们的关键设计包括： 1. 基于流的人体部位分割：利用光流作为几何线索，将动态区域从单目视频中的静态区域中分离出来； 2. 解耦重构流水线：在单目模糊性下同时优化人类运动和对象关节是不稳定的，因此我们首先恢复物体的关节状态，然后基于重建的对象状态条件生成人体动作。 ArtHOI通过视频生成与几何感知重建之间的桥梁产生了语义对齐且物理实证的互动场景。在各种复杂的关节式场景（例如打开冰箱、橱柜、微波炉）中，ArtHOI在接触准确性、穿透减少和关节真实度上明显优于先前的方法，在没有监督数据的情况下扩展了零样本交互合成能力，超越了刚性操作范围，并通过重建指导合成来实现这一目标。
URL
https://arxiv.org/abs/2603.04338
PDF
-
Architecture and evaluation protocol for transformer-based visual object tracking in UAV applications
2026-03-04 10:12:12Augustin Borne (ISL, Hochschule Karlsruhe -- Technik und Wirtschaft Karlsruhe University of Applied Sciences, IRIMAS), Pierre Notin (ISL), Christophe Hennequin (ISL), Sebastien Changey (ISL), Stephane Bazeille (IRIMAS), Christophe Cudel (IRIMAS), Franz QuintAbstract
Object tracking from Unmanned Aerial Vehicles (UAVs) is challenged by platform dynamics, camera motion, and limited onboard resources. Existing visual trackers either lack robustness in complex scenarios or are too computationally demanding for real-time embedded use. We propose an Modular Asynchronous Tracking Architecture (MATA) that combines a transformer-based tracker with an Extended Kalman Filter, integrating ego-motion compensation from sparse optical flow and an object trajectory model. We further introduce a hardware-independent, embedded oriented evaluation protocol and a new metric called Normalized time to Failure (NT2F) to quantify how long a tracker can sustain a tracking sequence without external help. Experiments on UAV benchmarks, including an augmented UAV123 dataset with synthetic occlusions, show consistent improvements in Success and NT2F metrics across multiple tracking processing frequency. A ROS 2 implementation on a Nvidia Jetson AGX Orin confirms that the evaluation protocol more closely matches real-time performance on embedded systems.
Abstract (translated)
来自无人飞行器（UAV）的对象跟踪面临平台动态、相机运动和有限的机载资源等挑战。现有的视觉追踪方法要么在复杂场景中不够鲁棒，要么对实时嵌入式应用来说计算需求过高。我们提出了一种模块化异步跟踪架构（MATA），该架构结合了基于变压器的追踪器和扩展卡尔曼滤波器，并集成了稀疏光流带来的自我运动补偿以及对象轨迹模型。此外，我们还引入了一种硬件无关、面向嵌入式设备的评估协议，以及一种新的度量标准——归一化失效时间（NT2F），用以量化追踪器在没有外部帮助的情况下能够维持跟踪序列的时间长度。实验结果表明，在包括增强版UAV123数据集（该数据集中合成了一些遮挡情况）在内的多个无人机基准测试中，MATA方法在成功率和NT2F度量标准上都有显著的改进，并且这些改进在不同的处理频率下是一致的。基于Nvidia Jetson AGX Orin设备上的ROS 2实现表明，这种评估协议更贴近嵌入式系统中的实时性能。 这段文字描述了一种新的对象跟踪架构以及相关的实验结果和评价方法，适用于无人飞行器（UAV）等场景下的视觉追踪任务。
URL
https://arxiv.org/abs/2603.03904
PDF
-
FREE-Edit: Using Editing-aware Injection in Rectified Flow Models for Zero-shot Image-Driven Video Editing
2026-03-01 16:01:44Maomao Li, Yunfei Liu, Yu LiAbstract
Image-driven video editing aims to propagate edit contents from the modified first frame to the rest frames. The existing methods usually invert the source video to noise using a pre-trained image-to-video (I2V) model and then guide the sampling process using the edited first frame. Generally, a popular choice for maintaining motion and layout from the source video is intervening in the denoising process by injecting attention during reconstruction. However, such injection often leads to unsatisfactory results, where excessive injection leads to conflicting semantics from the source video while insufficient injection brings limited source representation. Recognizing this, we propose an Editing-awaRE (REE) injection method to modulate injection intensity of each token. Specifically, we first compute the pixel difference between the source and edited first frame to form a corresponding editing mask. Next, we track the editing area throughout the entire video by using optical flow to warp the first-frame mask. Then, editing-aware feature injection intensity for each token is generated accordingly, where injection is not conducted on editing areas. Building upon REE injection, we further propose a zero-shot image-driven video editing framework with recent-emerging rectified-Flow models, dubbed FREE-Edit. Without fine-tuning or training, our FREE-Edit demonstrates effectiveness in various image-driven video editing scenarios, showing its capability to produce higher-quality outputs compared with existing techniques. Project page: this https URL.
Abstract (translated)
图像驱动的视频编辑旨在将修改后的第一帧上的编辑内容传播到其余所有帧上。现有方法通常会使用预训练的图像到视频（I2V）模型将源视频逆向转换为噪声，然后通过编辑过的首帧来引导采样过程。为了从源视频中保持运动和布局的一致性，一种常见的做法是在去噪过程中注入注意力以进行重建。然而，这种注入方式往往会导致不尽人意的结果：过度注入会使得原始视频中的语义与编辑内容产生冲突，而不足的注入则无法充分表示源视频的内容。 为了应对这一挑战，我们提出了一种名为Editing-awaRE（REE）的注入方法，该方法可以调节每个标记的注入强度。具体来说，首先计算源帧和编辑后的第一帧之间的像素差异，以形成对应的编辑掩码。接着，通过使用光学流技术来变形首帧掩码，我们可以跟踪整个视频中的编辑区域。根据这一过程生成了针对每个标记的、具有编辑意识的功能注入强度，在此过程中不会在编辑区域内进行任何功能注入。 基于REE注入方法，我们进一步提出了一种零样本图像驱动的视频编辑框架——FREE-Edit，并结合了最近出现的矫正流模型（rectified-Flow models）。无需微调或训练，我们的FREE-Edit框架能够在各种图像驱动的视频编辑场景中展现其有效性，并且能够产生比现有技术更高的质量输出。 项目页面：[请在此处替换为实际链接]
URL
https://arxiv.org/abs/2603.01164
PDF
-
Micro-expression Recognition Based on Dual-branch Feature Extraction and Fusion
2026-02-27 11:52:50Mingjie Zhang, Bo Li, Wanting Liu, Hongyan Cui, Yue Li, Qingwen Li, Hong Li, Ge GaoAbstract
Micro-expressions, characterized by transience and subtlety, pose challenges to existing optical flow-based recognition methods. To address this, this paper proposes a dual-branch micro-expression feature extraction network integrated with parallel attention. Key contributions include: 1) a residual network designed to alleviate gradient anishing and network degradation; 2) an Inception network constructed to enhance model representation and suppress interference from irrelevant regions; 3) an adaptive feature fusion module developed to integrate dual-branch features. Experiments on the CASME II dataset demonstrate that the proposed method achieves 74.67% accuracy, outperforming LBP-TOP (by 11.26%), MSMMT (by 3.36%), and other comparative methods.
Abstract (translated)
微表情由于其短暂性和微妙性，对现有的基于光学流的识别方法构成了挑战。为解决这一问题，本文提出了一种集成了并行注意力机制的双分支微表情特征提取网络。主要贡献包括：1）设计了一个残差网络来缓解梯度消失和网络退化的问题；2）构建了一个Inception网络以增强模型表示能力，并抑制无关区域的干扰；3）开发了一个自适应特征融合模块，用于整合双分支特征。在CASME II数据集上的实验表明，所提出的方法实现了74.67%的准确率，优于LBP-TOP（高出11.26%）、MSMMT（高出3.36%）和其他比较方法。
URL
https://arxiv.org/abs/2602.23950
PDF
-
SelfOccFlow: Towards end-to-end self-supervised 3D Occupancy Flow prediction
2026-02-27 10:42:01Xavier Timoneda, Markus Herb, Fabian Duerr, Daniel GoehringAbstract
Estimating 3D occupancy and motion at the vehicle's surroundings is essential for autonomous driving, enabling situational awareness in dynamic environments. Existing approaches jointly learn geometry and motion but rely on expensive 3D occupancy and flow annotations, velocity labels from bounding boxes, or pretrained optical flow models. We propose a self-supervised method for 3D occupancy flow estimation that eliminates the need for human-produced annotations or external flow supervision. Our method disentangles the scene into separate static and dynamic signed distance fields and learns motion implicitly through temporal aggregation. Additionally, we introduce a strong self-supervised flow cue derived from features' cosine similarities. We demonstrate the efficacy of our 3D occupancy flow method on SemanticKITTI, KITTI-MOT, and nuScenes.
Abstract (translated)
在自动驾驶中，估算车辆周围环境的三维占用情况和运动状态对于实现情境感知至关重要，尤其是在动态环境中。现有的方法能够同时学习几何形状和运动，但依赖于昂贵的3D占用标注、来自边界框的速度标签或预训练的光流模型。我们提出了一种无需人工生成注释或外部流动监督的自我监督方法来估算三维占用流。该方法将场景分解为独立的静态和动态有符号距离字段，并通过时间聚合隐式学习运动。此外，我们引入了一个强大的自我监督流动线索，来源于特征余弦相似度。我们在SemanticKITTI、KITTI-MOT和nuScenes数据集上展示了我们的3D占用流估算方法的有效性。
URL
https://arxiv.org/abs/2602.23894
PDF
-
DMAligner: Enhancing Image Alignment via Diffusion Model Based View Synthesis
2026-02-26 14:00:07Xinglong Luo, Ao Luo, Zhengning Wang, Yueqi Yang, Chaoyu Feng, Lei Lei, Bing Zeng, Shuaicheng LiuAbstract
Image alignment is a fundamental task in computer vision with broad applications. Existing methods predominantly employ optical flow-based image warping. However, this technique is susceptible to common challenges such as occlusions and illumination variations, leading to degraded alignment visual quality and compromised accuracy in downstream tasks. In this paper, we present DMAligner, a diffusion-based framework for image alignment through alignment-oriented view synthesis. DMAligner is crafted to tackle the challenges in image alignment from a new perspective, employing a generation-based solution that showcases strong capabilities and avoids the problems associated with flow-based image warping. Specifically, we propose a Dynamics-aware Diffusion Training approach for learning conditional image generation, synthesizing a novel view for image alignment. This incorporates a Dynamics-aware Mask Producing (DMP) module to adaptively distinguish dynamic foreground regions from static backgrounds, enabling the diffusion model to more effectively handle challenges that classical methods struggle to solve. Furthermore, we develop the Dynamic Scene Image Alignment (DSIA) dataset using Blender, which includes 1,033 indoor and outdoor scenes with over 30K image pairs tailored for image alignment. Extensive experimental results demonstrate the superiority of the proposed approach on DSIA benchmarks, as well as on a series of widely-used video datasets for qualitative comparisons. Our code is available at this https URL.
Abstract (translated)
图像对齐是计算机视觉中的基本任务，具有广泛的应用。现有方法主要采用基于光学流的图像扭曲技术。然而，这种方法容易受到遮挡和光照变化等常见问题的影响，导致对齐视觉质量下降，并且在下游任务中准确性降低。本文介绍了DMAligner，这是一种通过面向对齐视角合成的扩散框架进行图像对齐的方法。DMAligner 旨在从一个新的角度解决图像对齐中的挑战，采用基于生成的方法，展示出强大的能力并避免了与流基图像扭曲相关的问题。 具体而言，我们提出了一个感知动态性的扩散训练方法，用于学习条件下的图像生成，为图像对齐合成新视角。这包括了一个感知动态性掩码产生的（DMP）模块，能够自适应地区分动态前景区域和静态背景，使扩散模型更有效地处理传统方法难以解决的挑战。 此外，我们使用Blender开发了Dynamic Scene Image Alignment (DSIA) 数据集，其中包含1,033个室内和室外场景以及超过30K张图像对，专为图像对齐设计。广泛的实验结果展示了该方法在DSIA基准测试中的优越性，并且在一系列广泛使用的视频数据集中进行定性比较时也表现良好。 我们的代码可在[此链接](https://这个URL需要替换为你提供的具体URL)获取。
URL
https://arxiv.org/abs/2602.23022
PDF
-
Velocity and stroke rate reconstruction of canoe sprint team boats based on panned and zoomed video recordings
2026-02-26 12:31:30Julian Ziegler, Daniel Matthes, Finn Gerdts, Patrick Frenzel, Torsten Warnke, Matthias Englert, Tina Koevari, Mirco FuchsAbstract
Pacing strategies, defined by velocity and stroke rate profiles, are essential for peak performance in canoe sprint. While GPS is the gold standard for analysis, its limited availability necessitates automated video-based solutions. This paper presents an extended framework for reconstructing performance metrics from panned and zoomed video recordings across all sprint disciplines (K1-K4, C1-C2) and distances (200m-500m). Our method utilizes YOLOv8 for buoy and athlete detection, leveraging the known buoy grid to estimate homographies. We generalized the estimation of the boat position by means of learning a boat-specific athlete offset using a U-net based boat tip calibration. Further, we implement a robust tracking scheme using optical flow to adapt to multi-athlete boat types. Finally, we introduce methods to extract stroke rate information from either pose estimations or the athlete bounding boxes themselves. Evaluation against GPS data from elite competitions yields a velocity RRMSE of 0.020 +- 0.011 (rho = 0.956) and a stroke rate RRMSE of 0.022 +- 0.024 (rho = 0.932). The methods provide coaches with highly accurate, automated feedback without requiring on-boat sensors or manual annotation.
Abstract (translated)
划船竞速中的速度和划桨频率策略对于达到巅峰表现至关重要。虽然全球定位系统（GPS）是分析的最佳标准，但由于其有限的可用性，需要自动化的视频解决方案。本文提出了一种扩展框架，用于从平移和缩放视频记录中重建所有冲刺项目（K1-K4, C1-C2）及距离（200米至500米）的性能指标。 我们的方法利用YOLOv8检测浮标和运动员，并通过已知的浮标网格来估计同构映射。我们通过学习特定于船的运动员偏移量，采用基于U-net的船尖校准方法，推广了对船位置的估算。此外，我们实施了一种使用光流实现的鲁棒跟踪方案，以适应多运动员类型的船只。 最后，我们引入了几种从姿势估计或运动员边界框自身提取划桨频率信息的方法。与来自精英比赛的GPS数据对比评估显示，速度的相对均方根误差（RRMSE）为0.020 ± 0.011 (ρ = 0.956)，划桨频率的RRMSE为0.022 ± 0.024 (ρ = 0.932)。 这些方法为教练提供了高精度且自动化的反馈，无需船上传感器或手动标注。
URL
https://arxiv.org/abs/2602.22941
PDF
-
GSTurb: Gaussian Splatting for Atmospheric Turbulence Mitigation
2026-02-26 09:37:27Hanliang Du, Zhangji Lu, Zewei Cai, Qijian Tang, Qifeng Yu, Xiaoli LiuAbstract
Atmospheric turbulence causes significant image degradation due to pixel displacement (tilt) and blur, particularly in long-range imaging applications. In this paper, we propose a novel framework for atmospheric turbulence mitigation, GSTurb, which integrates optical flow-guided tilt correction and Gaussian splatting for modeling non-isoplanatic blur. The framework employs Gaussian parameters to represent tilt and blur, and optimizes them across multiple frames to enhance restoration. Experimental results on the ATSyn-static dataset demonstrate the effectiveness of our method, achieving a peak PSNR of 27.67 dB and SSIM of 0.8735. Compared to the state-of-the-art method, GSTurb improves PSNR by 1.3 dB (a 4.5% increase) and SSIM by 0.048 (a 5.8% increase). Additionally, on real datasets, including the TSRWGAN Real-World and CLEAR datasets, GSTurb outperforms existing methods, showing significant improvements in both qualitative and quantitative performance. These results highlight that combining optical flow-guided tilt correction with Gaussian splatting effectively enhances image restoration under both synthetic and real-world turbulence conditions. The code for this method will be available at this https URL.
Abstract (translated)
大气湍流会导致图像退化，主要原因是像素位移（倾斜）和模糊，尤其是在长距离成像应用中。本文提出了一种新的大气湍流缓解框架GSTurb，该框架结合了光学流引导的倾斜校正和高斯喷射来建模非同像平面模糊。此框架使用高斯参数表示倾斜和模糊，并在多帧之间优化这些参数以增强图像恢复效果。 实验结果表明，在ATSyn-static数据集上，我们的方法达到了峰值PSNR为27.67 dB和SSIM为0.8735的性能。相较于最先进的方法，GSTurb将PSNR提高了1.3 dB（增幅4.5%），并将SSIM提高了0.048（增幅5.8%）。此外，在真实数据集TSRWGAN Real-World和CLEAR上，GSTurb也优于现有方法，无论是在定性还是定量性能方面都表现出显著的改进。这些结果表明，结合光学流引导的倾斜校正与高斯喷射可以有效地增强合成湍流和现实世界湍流条件下的图像恢复。 该方法的相关代码将在此网址提供：[此链接](https://this-url.com)（请将此处的链接替换为实际提供的GitHub或其他存储库链接）。
URL
https://arxiv.org/abs/2602.22800
PDF
-
DAGS-SLAM: Dynamic-Aware 3DGS SLAM via Spatiotemporal Motion Probability and Uncertainty-Aware Scheduling
2026-02-25 07:14:07Li Zhang, Yu-An Liu, Xijia Jiang, Conghao Huang, Danyang Li, Yanyong ZhangAbstract
Mobile robots and IoT devices demand real-time localization and dense reconstruction under tight compute and energy budgets. While 3D Gaussian Splatting (3DGS) enables efficient dense SLAM, dynamic objects and occlusions still degrade tracking and mapping. Existing dynamic 3DGS-SLAM often relies on heavy optical flow and per-frame segmentation, which is costly for mobile deployment and brittle under challenging illumination. We present DAGS-SLAM, a dynamic-aware 3DGS-SLAM system that maintains a spatiotemporal motion probability (MP) state per Gaussian and triggers semantics on demand via an uncertainty-aware scheduler. DAGS-SLAM fuses lightweight YOLO instance priors with geometric cues to estimate and temporally update MP, propagates MP to the front-end for dynamic-aware correspondence selection, and suppresses dynamic artifacts in the back-end via MP-guided optimization. Experiments on public dynamic RGB-D benchmarks show improved reconstruction and robust tracking while sustaining real-time throughput on a commodity GPU, demonstrating a practical speed-accuracy tradeoff with reduced semantic invocations toward mobile deployment.
Abstract (translated)
移动机器人和物联网设备在计算资源和能源有限的情况下，需要实时定位和密集重建。虽然3D高斯点阵（3D Gaussian Splatting，简称3DGS）能够实现高效的稠密SLAM（即时定位与地图构建），但是动态物体和遮挡仍然会降低跟踪和建图的质量。现有的动态3DGS-SLAM系统通常依赖于重量级的光流计算和逐帧分割，这在移动部署中成本高昂且在照明条件恶劣的情况下稳定性较差。 我们提出了一个名为DAGS-SLAM（动态感知3DGS-SLAM）的系统，该系统为每个高斯点维护一个时空运动概率（Motion Probability, MP）状态，并通过不确定性感知调度器按需触发语义处理。DAGS-SLAM将轻量级YOLO实例先验与几何线索相结合以估计和临时更新MP，在前端传播MP以便动态感知对应选择，同时在后端利用MP引导优化来抑制动态伪影。 实验显示，DAGS-SLAM在公共的动态RGB-D基准测试中显著提升了重建质量和跟踪鲁棒性，并能在商品级GPU上维持实时吞吐量。这表明通过减少语义调用，DAGS-SLAM实现了移动部署所需的实用速度精度权衡。
URL
https://arxiv.org/abs/2602.21644
PDF
-
SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment
2026-02-18 13:23:28Runlong He, Freweini M. Tesfai, Matthew W. E. Boal, Nazir Sirajudeen, Dimitrios Anastasiou, Jialang Xu, Mobarak I. Hoque, Philip J. Edwards, John D. Kelly, Ashwin Sridhar, Abdolrahim Kadkhodamohammadi, Dhivya Chandrasekaran, Matthew J. Clarkson, Danail Stoyanov, Nader Francis, Evangelos B. MazomenosAbstract
Robotic-assisted surgery (RAS) is established in clinical practice, and automated surgical skill assessment utilizing multimodal data offers transformative potential for surgical analytics and education. However, developing effective multimodal methods remains challenging due to the task complexity, limited annotated datasets and insufficient techniques for cross-modal information fusion. Existing state-of-the-art relies exclusively on RGB video and only applies on dry-lab settings, failing to address the significant domain gap between controlled simulation and real clinical cases, where the surgical environment together with camera and tissue motion introduce substantial complexities. This work introduces SurgFusion-Net and Divergence Regulated Attention (DRA), an innovative fusion strategy for multimodal surgical skill assessment. We contribute two first-of-their-kind clinical datasets: the RAH-skill dataset containing 279,691 RGB frames from 37 videos of Robot-assisted Hysterectomy (RAH), and the RARP-skill dataset containing 70,661 RGB frames from 33 videos of Robot-Assisted Radical Prostatectomy (RARP). Both datasets include M-GEARS skill annotations, corresponding optical flow and tool segmentation masks. DRA incorporates adaptive dual attention and diversity-promoting multi-head attention to fuse multimodal information, from three modalities, based on surgical context, enhancing assessment accuracy and reliability. Validated on the JIGSAWS benchmark, RAH-skill, and RARP-skill datasets, our approach outperforms recent baselines with SCC improvements of 0.02 in LOSO, 0.04 in LOUO across JIGSAWS tasks, and 0.0538 and 0.0493 gains on RAH-skill and RARP-skill, respectively.
Abstract (translated)
机器人辅助手术（RAS）已广泛应用于临床实践中，利用多模态数据进行自动化外科技能评估在手术分析和教育方面具有变革性的潜力。然而，由于任务复杂性、标注数据集有限以及跨模态信息融合技术不足等原因，开发有效的多模态方法仍面临挑战。现有的最先进的技术仅依赖于RGB视频，并且只适用于模拟实验室环境，无法解决受控仿真与实际临床案例之间的显著领域差距问题，在实际手术环境中，摄像机和组织的运动带来了相当大的复杂性。 这项工作引入了SurgFusion-Net以及差异调节注意力（DRA），这是一种创新性的多模态融合策略用于外科技能评估。我们贡献了两个具有开创意义的临床数据集：包含279,691张RGB图像，来自37个机器人辅助子宫切除术(RAH)视频片段的RAH-skill数据集；以及包含70,661张RGB图像，来自33个机器人辅助前列腺根治性切除术(RARP)视频片段的RARP-skill数据集。这两个数据集均包括M-GEARS技能标注、对应的光学流和工具分割掩码。 DRA方法结合了自适应双注意力机制以及多头注意力以促进多样性，并根据手术上下文从三种模态中融合多模态信息，从而提高评估的准确性和可靠性。在JIGSAWS基准测试、RAH-skill数据集及RARP-skill数据集上验证后，我们的方法优于近期的基线，在LOSO（Leave-One-Subject-Out）和LOUO（Leave-One-User-Out）设置下分别取得了0.02和0.04 SCC改进，以及在JIGSAWS任务上的表现有所提高；同时对RAH-skill数据集和RARP-skill数据集的改进幅度分别为0.0538和0.0493。
URL
https://arxiv.org/abs/2603.00108
PDF
-
ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
2026-02-18 12:37:35Daichi Yashima, Shuhei Kurita, Yusuke Oda, Komei SugiuraAbstract
While multimodal large language models (MLLMs) have shown remarkable success across a wide range of tasks, long-form video understanding remains a significant challenge. In this study, we focus on video understanding by MLLMs. This task is challenging because processing a full stream of RGB frames is computationally intractable and highly redundant, as self-attention have quadratic complexity with sequence length. In this paper, we propose ReMoRa, a video MLLM that processes videos by operating directly on their compressed representations. A sparse set of RGB keyframes is retained for appearance, while temporal dynamics are encoded as a motion representation, removing the need for sequential RGB frames. These motion representations act as a compact proxy for optical flow, capturing temporal dynamics without full frame decoding. To refine the noise and low fidelity of block-based motions, we introduce a module to denoise and generate a fine-grained motion representation. Furthermore, our model compresses these features in a way that scales linearly with sequence length. We demonstrate the effectiveness of ReMoRa through extensive experiments across a comprehensive suite of long-video understanding benchmarks. ReMoRa outperformed baseline methods on multiple challenging benchmarks, including LongVideoBench, NExT-QA, and MLVU.
Abstract (translated)
URL
https://arxiv.org/abs/2602.16412
PDF
-
Dual-Signal Adaptive KV-Cache Optimization for Long-Form Video Understanding in Vision-Language Models
2026-02-15 17:06:02Vishnu Sai, Dheeraj Sai, Srinath B, Girish Varma, Priyesh ShuklaAbstract
Vision-Language Models (VLMs) face a critical memory bottleneck when processing long-form video content due to the linear growth of the Key-Value (KV) cache with sequence length. Existing solutions predominantly employ reactive eviction strategies that compute full attention matrices before discarding tokens, resulting in substantial computational waste. We propose Sali-Cache, a novel a priori optimization framework that implements dual-signal adaptive caching through proactive memory management. By integrating a temporal filter based on optical flow analysis for detecting inter-frame redundancy and a spatial filter leveraging saliency detection for identifying visually significant regions, Sali-Cache intelligently manages memory allocation before entering computationally expensive attention operations. Experimental evaluation on the LLaVA 1.6 architecture demonstrates that our method achieves a 2.20x compression ratio in effective memory usage while maintaining 100% accuracy across BLEU, ROUGE-L, and Exact Match metrics. Furthermore, under identical memory budget constraints, Sali-Cache preserves context-rich features over extended temporal durations without degrading model performance, enabling efficient processing of long-form video content on consumer-grade hardware.
Abstract (translated)
URL
https://arxiv.org/abs/2602.14236
PDF
-
UAV-SEAD: State Estimation Anomaly Dataset for UAVs
2026-02-14 21:45:30Aykut Kabaoglu, Sanem SarielAbstract
Accurate state estimation in Unmanned Aerial Vehicles (UAVs) is crucial for ensuring reliable and safe operation, as anomalies occurring during mission execution may induce discrepancies between expected and observed system behaviors, thereby compromising mission success or posing potential safety hazards. It is essential to continuously monitor and detect such conditions in order to ensure a timely response and maintain system reliability. In this work, we focus on UAV state estimation anomalies and provide a large-scale real-world UAV dataset to facilitate research aimed at improving the development of anomaly detection. Unlike existing datasets that primarily rely on injected faults into simulated data, this dataset comprises 1396 real flight logs totaling over 52 hours of flight time, collected across diverse indoor and outdoor environments using a collection of PX4-based UAVs equipped with a variety of sensor configurations. The dataset comprises both normal and anomalous flights without synthetic manipulation, making it uniquely suitable for realistic anomaly detection tasks. A structured classification is proposed that categorizes UAV state estimation anomalies into four classes: mechanical and electrical, external position, global position, and altitude anomalies. These classifications reflect collective, contextual, and outlier anomalies observed in multivariate sensor data streams, including IMU, GPS, barometer, magnetometer, distance sensors, visual odometry, and optical flow, that can be found in the PX4 logging mechanism. It is anticipated that this dataset will play a key role in the development, training, and evaluation of anomaly detection and isolation systems to address the critical gap in UAV reliability research.
Abstract (translated)
URL
https://arxiv.org/abs/2602.13900
PDF
-
Unbiased Gradient Estimation for Event Binning via Functional Backpropagation
2026-02-13 04:05:03Jinze Chen, Wei Zhai, Han Han, Tiankai Ma, Yang Cao, Bin Li, Zheng-Jun ZhaAbstract
Event-based vision encodes dynamic scenes as asynchronous spatio-temporal spikes called events. To leverage conventional image processing pipelines, events are typically binned into frames. However, binning functions are discontinuous, which truncates gradients at the frame level and forces most event-based algorithms to rely solely on frame-based features. Attempts to directly learn from raw events avoid this restriction but instead suffer from biased gradient estimation due to the discontinuities of the binning operation, ultimately limiting their learning efficiency. To address this challenge, we propose a novel framework for unbiased gradient estimation of arbitrary binning functions by synthesizing weak derivatives during backpropagation while keeping the forward output unchanged. The key idea is to exploit integration by parts: lifting the target functions to functionals yields an integral form of the derivative of the binning function during backpropagation, where the cotangent function naturally arises. By reconstructing this cotangent function from the sampled cotangent vector, we compute weak derivatives that provably match long-range finite differences of both smooth and non-smooth targets. Experimentally, our method improves simple optimization-based egomotion estimation with 3.2\% lower RMS error and 1.57$\times$ faster convergence. On complex downstream tasks, we achieve 9.4\% lower EPE in self-supervised optical flow, and 5.1\% lower RMS error in SLAM, demonstrating broad benefits for event-based visual perception. Source code can be found at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.12590
PDF
-
Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultra
2026-02-10 20:40:45Yoichi OchiaiAbstract
While real-time image generation using diffusion models has advanced rapidly on NVIDIA GPUs, systematic optimization research on non-CUDA platforms such as Apple Silicon remains extremely limited. In this study, we conducted comprehensive optimization experiments across 10 phases targeting the Apple M3 Ultra (60-core GPU, 512 GB unified memory) with the goal of achieving real-time camera img2img transformation. We explored a wide range of techniques including CoreML conversion, quantization, Token Merging, Neural Engine utilization, compact model exploration, frame interpolation, kNN search-based synthesis, pix2pix-turbo, optical flow frame skipping, and knowledge distillation, quantitatively evaluating the effectiveness of each approach. Ultimately, by combining CoreML conversion of the distillation-specialized model SDXS-512 with a 3-thread camera pipeline, we achieved real-time camera img2img transformation at 22.7 FPS at 512x512 resolution. The primary contribution of this work is the systematic demonstration that optimization insights established for CUDA are not necessarily effective on Apple Silicon's unified memory architecture. We reveal an optimization landscape fundamentally different from that of NVIDIA GPUs -- including the absence of speedup from quantization, the ineffectiveness of parallel inference, and the unsuitability of the Neural Engine for large-scale models -- and provide practical guidelines for diffusion model inference on Apple Silicon.
Abstract (translated)
URL
https://arxiv.org/abs/2605.16259
PDF
-
RoboSubtaskNet: Temporal Sub-task Segmentation for Human-to-Robot Skill Transfer in Real-World Environments
2026-02-10 17:37:35Dharmendra Sharma, Archit Sharma, John Reberio, Vaibhav Kesharwani, Peeyush Thakur, Narendra Kumar Dhar, Laxmidhar BeheraAbstract
Temporally locating and classifying fine-grained sub-task segments in long, untrimmed videos is crucial to safe human-robot collaboration. Unlike generic activity recognition, collaborative manipulation requires sub-task labels that are directly robot-executable. We present RoboSubtaskNet, a multi-stage human-to-robot sub-task segmentation framework that couples attention-enhanced I3D features (RGB plus optical flow) with a modified MS-TCN employing a Fibonacci dilation schedule to capture better short-horizon transitions such as reach-pick-place. The network is trained with a composite objective comprising cross-entropy and temporal regularizers (truncated MSE and a transition-aware term) to reduce over-segmentation and to encourage valid sub-task progressions. To close the gap between vision benchmarks and control, we introduce RoboSubtask, a dataset of healthcare and industrial demonstrations annotated at the sub-task level and designed for deterministic mapping to manipulator primitives. Empirically, RoboSubtaskNet outperforms MS-TCN and MS-TCN++ on GTEA and our RoboSubtask benchmark (boundary-sensitive and sequence metrics), while remaining competitive on the long-horizon Breakfast benchmark. Specifically, RoboSubtaskNet attains F1 @ 50 = 79.5%, Edit = 88.6%, Acc = 78.9% on GTEA; F1 @ 50 = 30.4%, Edit = 52.0%, Acc = 53.5% on Breakfast; and F1 @ 50 = 94.2%, Edit = 95.6%, Acc = 92.2% on RoboSubtask. We further validate the full perception-to-execution pipeline on a 7-DoF Kinova Gen3 manipulator, achieving reliable end-to-end behavior in physical trials (overall task success approx 91.25%). These results demonstrate a practical path from sub-task level video understanding to deployed robotic manipulation in real-world settings.
Abstract (translated)
在长时间未剪辑的视频中定位和分类精细粒度的任务子段对于安全的人机协作至关重要。与通用活动识别不同，协作操作需要可以直接由机器人执行的任务子标签。我们提出了一种多阶段的人类到机器人的任务分割框架RoboSubtaskNet，该框架结合了增强注意力机制的I3D特征（RGB和光流）以及采用斐波那契膨胀时间表修改后的MS-TCN架构，以更好地捕捉如“伸手-抓取-放置”这类短时域内的转换。网络通过一个包含交叉熵和时间正则化器（截断MSE和过渡感知项）的复合目标进行训练，以减少过度分割并鼓励有效的子任务进展。 为了弥合视觉基准与控制之间的差距，我们引入了RoboSubtask数据集，该数据集包含了医疗保健和工业演示，并在子任务级别进行了注释，旨在确定性地映射到机械臂的基本操作。经验表明，在GTEA和我们的RoboSubtask基准测试（边界敏感性和序列度量）上，RoboSubtaskNet的表现优于MS-TCN和MS-TCN++，同时在长时域的Breakfast数据集上也具有竞争力。具体而言，RoboSubtaskNet在GTEA上的F1@50 = 79.5%，Edit = 88.6%，Acc = 78.9%；在Breakfast上的F1@50 = 30.4%，Edit = 52.0%，Acc = 53.5%；以及在RoboSubtask上的F1@50 = 94.2%，Edit = 95.6%，Acc = 92.2%。我们进一步在一个7自由度的Kinova Gen3机械臂上验证了整个感知到执行的管道，在物理试验中实现了可靠的整体端到端行为（总体任务成功率约为91.25%）。这些结果展示了从子任务级别视频理解到现实世界环境中部署机器人操作的实际路径。 上述内容翻译自原文，详细描述了RoboSubtaskNet框架及其在不同数据集上的性能表现。
URL
https://arxiv.org/abs/2602.10015
PDF
-
SemanticMoments: Training-Free Motion Similarity via Third Moment Features
2026-02-09 19:47:56Saar Huberman, Kfir Goldberg, Or Patashnik, Sagie Benaim, Ron MokadyAbstract
Retrieving videos based on semantic motion is a fundamental, yet unsolved, problem. Existing video representation approaches overly rely on static appearance and scene context rather than motion dynamics, a bias inherited from their training data and objectives. Conversely, traditional motion-centric inputs like optical flow lack the semantic grounding needed to understand high-level motion. To demonstrate this inherent bias, we introduce the SimMotion benchmarks, combining controlled synthetic data with a new human-annotated real-world dataset. We show that existing models perform poorly on these benchmarks, often failing to disentangle motion from appearance. To address this gap, we propose SemanticMoments, a simple, training-free method that computes temporal statistics (specifically, higher-order moments) over features from pre-trained semantic models. Across our benchmarks, SemanticMoments consistently outperforms existing RGB, flow, and text-supervised methods. This demonstrates that temporal statistics in a semantic feature space provide a scalable and perceptually grounded foundation for motion-centric video understanding.
Abstract (translated)
基于语义动作检索视频是一项基本但尚未解决的问题。现有的视频表示方法过于依赖静态外观和场景上下文，而忽视了动态动作，这种偏见源自它们的训练数据和目标设定。相比之下，传统的以动作为中心的输入（如光流）缺乏理解高层次动作所需的语义背景。为了展示这一内在偏见，我们引入了SimMotion基准测试集，它结合了受控合成数据与新的、由人类标注的真实世界数据集。我们发现现有模型在这些基准上表现不佳，经常无法将运动和外观分开。 为了解决这个差距，我们提出了SemanticMoments，这是一种简单且无需训练的方法，通过对预训练语义模型的特征计算时间统计（特别是高阶矩）。在我们的基准测试中，SemanticMoments始终优于现有的RGB、光流以及文本监督方法。这表明，在语义特征空间中的时间统计数据为以动作为中心的视频理解提供了一个可扩展且感知基础良好的框架。
URL
https://arxiv.org/abs/2602.09146
PDF
-
FLAG-4D: Flow-Guided Local-Global Dual-Deformation Model for 4D Reconstruction
2026-02-09 11:55:15Guan Yuan Tan, Ngoc Tuan Vu, Arghya Pal, Sailaja Rajanala, Raphael Phan C. -W., Mettu Srinivas, Chee-Ming TingAbstract
We introduce FLAG-4D, a novel framework for generating novel views of dynamic scenes by reconstructing how 3D Gaussian primitives evolve through space and time. Existing methods typically rely on a single Multilayer Perceptron (MLP) to model temporal deformations, and they often struggle to capture complex point motions and fine-grained dynamic details consistently over time, especially from sparse input views. Our approach, FLAG-4D, overcomes this by employing a dual-deformation network that dynamically warps a canonical set of 3D Gaussians over time into new positions and anisotropic shapes. This dual-deformation network consists of an Instantaneous Deformation Network (IDN) for modeling fine-grained, local deformations and a Global Motion Network (GMN) for capturing long-range dynamics, refined through mutual learning. To ensure these deformations are both accurate and temporally smooth, FLAG-4D incorporates dense motion features from a pretrained optical flow backbone. We fuse these motion cues from adjacent timeframes and use a deformation-guided attention mechanism to align this flow information with the current state of each evolving 3D Gaussian. Extensive experiments demonstrate that FLAG-4D achieves higher-fidelity and more temporally coherent reconstructions with finer detail preservation than state-of-the-art methods.
Abstract (translated)
我们介绍了一种新的框架FLAG-4D，用于通过重建三维高斯原语在空间和时间中的演变来生成动态场景的新视角。现有的方法通常依赖于单个多层感知器（MLP）来建模时序变形，它们往往难以一致地捕捉复杂点的运动以及精细的时间细节，尤其是在从稀疏输入视图中更是如此。我们的方法FLAG-4D通过采用一个双重形变网络解决了这个问题，该网络能够动态地将一组规范化的三维高斯模型随着时间推移转换为新的位置和各向异性形状。这个双重变形网络由即时变形网络（IDN）组成，用于建模细粒度的局部变形，并且还包含全局运动网络（GMN），用于捕捉长距离的动力学变化，通过相互学习进行优化。 为了确保这些形变既准确又在时间上平滑，FLAG-4D从一个预训练的光流主干网中整合了密集的运动特征。我们融合来自相邻时间段的运动线索，并利用一种变形引导注意机制将此流动信息与每个演变中的三维高斯模型当前状态对齐。 通过广泛的实验表明，FLAG-4D在保真度和时间上的一致性方面超越了最先进的方法，且能够更好地保留细节。
URL
https://arxiv.org/abs/2602.08558
PDF
-
RealSynCol: a high-fidelity synthetic colon dataset for 3D reconstruction applications
2026-02-09 08:57:37Chiara Lena, Davide Milesi, Alessandro Casella, Luca Carlini, Joseph C. Norton, James Martin, Bruno Scaglioni, Keith L. Obstein, Roberto De Sire, Marco Spadaccini, Cesare Hassan, Pietro Valdastri, Elena De MomiAbstract
Deep learning has the potential to improve colonoscopy by enabling 3D reconstruction of the colon, providing a comprehensive view of mucosal surfaces and lesions, and facilitating the identification of unexplored areas. However, the development of robust methods is limited by the scarcity of large-scale ground truth data. We propose RealSynCol, a highly realistic synthetic dataset designed to replicate the endoscopic environment. Colon geometries extracted from 10 CT scans were imported into a virtual environment that closely mimics intraoperative conditions and rendered with realistic vascular textures. The resulting dataset comprises 28\,130 frames, paired with ground truth depth maps, optical flow, 3D meshes, and camera trajectories. A benchmark study was conducted to evaluate the available synthetic colon datasets for the tasks of depth and pose estimation. Results demonstrate that the high realism and variability of RealSynCol significantly enhance generalization performance on clinical images, proving it to be a powerful tool for developing deep learning algorithms to support endoscopic diagnosis.
Abstract (translated)
深度学习有可能通过启用结肠的三维重建来改进结肠镜检查，提供粘膜表面和病变的整体视图，并帮助识别未探索的区域。然而，由于缺乏大规模的真实数据集，开发稳健方法的努力受到了限制。我们提出了RealSynCol，这是一个高度逼真的合成数据集，旨在复制内窥镜环境。从10次CT扫描中提取的结肠几何形状被导入到一个虚拟环境中，该环境紧密模拟了术中的条件，并用逼真的血管纹理进行了渲染。生成的数据集中包含28,130帧图像，每张图像都配有一份地面实况深度图、光流数据、三维网格和相机轨迹。 为了评估现有的合成结肠数据集在深度估计和姿态估计任务上的性能，我们进行了一项基准研究。结果显示，RealSynCol的高逼真度和多样性显著提升了在临床图像上的一般化性能，证明它是一种开发支持内窥镜诊断的深度学习算法的强大工具。
URL
https://arxiv.org/abs/2602.08397
PDF
-
POPL-KF: A Pose-Only Geometric Representation-Based Kalman Filter for Point-Line-Based Visual-Inertial Odometry
2026-02-06 06:45:39Aiping Wang, Zhaolong Yang, Shuwen Chen, Hai ZhangAbstract
Mainstream Visual-inertial odometry (VIO) systems rely on point features for motion estimation and localization. However, their performance degrades in challenging scenarios. Moreover, the localization accuracy of multi-state constraint Kalman filter (MSCKF)-based VIO systems suffers from linearization errors associated with feature 3D coordinates and delayed measurement updates. To improve the performance of VIO in challenging scenes, we first propose a pose-only geometric representation for line features. Building on this, we develop POPL-KF, a Kalman filter-based VIO system that employs a pose-only geometric representation for both point and line features. POPL-KF mitigates linearization errors by explicitly eliminating both point and line feature coordinates from the measurement equations, while enabling immediate update of visual measurements. We also design a unified base-frames selection algorithm for both point and line features to ensure optimal constraints on camera poses within the pose-only measurement model. To further improve line feature quality, a line feature filter based on image grid segmentation and bidirectional optical flow consistency is proposed. Our system is evaluated on public datasets and real-world experiments, demonstrating that POPL-KF outperforms the state-of-the-art (SOTA) filter-based methods (OpenVINS, PO-KF) and optimization-based methods (PL-VINS, EPLF-VINS), while maintaining real-time performance.
Abstract (translated)
主流的视觉惯性里程计（VIO）系统依赖于点特征来进行运动估计和定位，但在挑战性的场景中其性能会下降。此外，基于多状态约束卡尔曼滤波器（MSCKF）的VIO系统的定位精度因与特征3D坐标相关的线性化误差以及延迟的测量更新而受到影响。为了提高在困难环境中的VIO性能，我们首先提出了一种仅基于姿态的几何表示方法用于直线特征。在此基础上，我们开发了POPL-KF系统，这是一种卡尔曼滤波器（Kalman filter）基的VIO系统，它为点和直线特征都采用了一种仅考虑姿态的几何表示方法。通过从测量方程中显式地消除点和线特征坐标，POPL-KF减少了线性化误差，并同时实现了视觉测量的即时更新功能。我们还设计了一个统一的基础帧选择算法，该算法适用于点和线特征，以确保在仅基于姿态的测量模型下对相机姿态施加最优约束条件。为了进一步提高直线特征的质量，提出了一种基于图像网格分割和双向光流一致性的直线特征过滤器。 我们的系统已在公开数据集及真实世界实验中进行了评估，结果表明POPL-KF优于现有的滤波方法（如OpenVINS、PO-KF）以及优化方法（如PL-VINS、EPLF-VINS），同时保持了实时性能。
URL
https://arxiv.org/abs/2602.06425
PDF
