Optical_Flow
-
TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation
2026-03-24 17:53:41Jini Yang, Eunbeen Hong, Soowon Son, Hyunkoo Lee, Sunghwan Hong, Sunok Kim, Seungryong KimAbstract
Event cameras capture per-pixel brightness changes with microsecond resolution, offering continuous motion information lost between RGB frames. However, existing event-based motion estimators depend on large-scale synthetic data that often suffers from a significant sim-to-real gap. We propose TETO (Tracking Events with Teacher Observation), a teacher-student framework that learns event motion estimation from only $\sim$25 minutes of unannotated real-world recordings through knowledge distillation from a pretrained RGB tracker. Our motion-aware data curation and query sampling strategy maximizes learning from limited data by disentangling object motion from dominant ego-motion. The resulting estimator jointly predicts point trajectories and dense optical flow, which we leverage as explicit motion priors to condition a pretrained video diffusion transformer for frame interpolation. We achieve state-of-the-art point tracking on EVIMO2 and optical flow on DSEC using orders of magnitude less training data, and demonstrate that accurate motion estimation translates directly to superior frame interpolation quality on BS-ERGB and HQ-EVFI.
Abstract (translated)
事件相机以微秒级分辨率捕捉像素级亮度变化，提供RGB帧间丢失的连续运动信息。然而，现有基于事件的运动估计器依赖于大规模合成数据，常受显著的仿真到现实差距影响。我们提出TETO（通过教师观测追踪事件），一个师生框架，仅通过从预训练RGB追踪器进行知识蒸馏，就能从约25分钟未标注的真实世界记录中学习事件运动估计。我们提出的运动感知数据整理与查询采样策略，通过将物体运动与主导的自我运动解耦，最大化有限数据的学习效果。所得估计器联合预测点轨迹和密集光流，我们将其作为显式运动先验，用于调节预训练的视频扩散Transformer以实现帧插值。我们在EVIMO2上实现最先进的点追踪，在DSEC上实现最先进的光流，训练数据量减少数个数量级，并证明准确运动估计可直接提升BS-ERGB和HQ-EVFI上的帧插值质量。
URL
https://arxiv.org/abs/2603.23487
PDF
-
GenOpticalFlow: A Generative Approach to Unsupervised Optical Flow Learning
2026-03-23 17:55:52Yixuan Luo, Feng Qiao, Zhexiao Xiong, Yanjing Li, Nathan JacobsAbstract
Optical flow estimation is a fundamental problem in computer vision, yet the reliance on expensive ground-truth annotations limits the scalability of supervised approaches. Although unsupervised and semi-supervised methods alleviate this issue, they often suffer from unreliable supervision signals based on brightness constancy and smoothness assumptions, leading to inaccurate motion estimation in complex real-world scenarios. To overcome these limitations, we introduce \textbf{\modelname}, a novel framework that synthesizes large-scale, perfectly aligned frame--flow data pairs for supervised optical flow training without human annotations. Specifically, our method leverages a pre-trained depth estimation network to generate pseudo optical flows, which serve as conditioning inputs for a next-frame generation model trained to produce high-fidelity, pixel-aligned subsequent frames. This process enables the creation of abundant, high-quality synthetic data with precise motion correspondence. Furthermore, we propose an \textit{inconsistent pixel filtering} strategy that identifies and removes unreliable pixels in generated frames, effectively enhancing fine-tuning performance on real-world datasets. Extensive experiments on KITTI2012, KITTI2015, and Sintel demonstrate that \textbf{\modelname} achieves competitive or superior results compared to existing unsupervised and semi-supervised approaches, highlighting its potential as a scalable and annotation-free solution for optical flow learning. We will release our code upon acceptance.
Abstract (translated)
光流估计是计算机视觉中的基础问题，但依赖昂贵的真值标注限制了监督方法的可扩展性。尽管无监督与半监督方法缓解了这一问题，它们常因基于亮度恒定和平滑性假设的不可靠监督信号，在复杂真实场景中产生不准确的运动估计。为克服这些限制，我们提出\textbf{\modelname}，一种无需人工标注即可为监督光流训练合成大规模、精确对齐的帧-光流数据对的新框架。具体而言，我们的方法利用预训练深度估计网络生成伪光流，作为下一帧生成模型的输入条件，该模型经训练可生成高保真、像素对齐的后续帧。此过程能创建具有精确运动对应关系的丰富高质量合成数据。此外，我们提出\textit{不一致像素过滤}策略，识别并移除生成帧中的不可靠像素，有效提升在真实数据集上的微调性能。在KITTI2012、KITTI2015和Sintel上的大量实验表明，\textbf{\modelname}相比现有无监督与半监督方法取得了具有竞争力或更优的结果，凸显其作为可扩展、无标注解决方案的光流学习潜力。代码将在论文接收后公开。
URL
https://arxiv.org/abs/2603.22270
PDF
-
$x^2$-Fusion: Cross-Modality and Cross-Dimension Flow Estimation in Event Edge Space
2026-03-17 15:36:57Ruishan Guo, Ciyu Ruan, Haoyang Wang, Zihang Gong, Jingao Xu, Xinlei ChenAbstract
Estimating dense 2D optical flow and 3D scene flow is essential for dynamic scene understanding. Recent work combines images, LiDAR, and event data to jointly predict 2D and 3D motion, yet most approaches operate in separate heterogeneous feature spaces. Without a shared latent space that all modalities can align to, these systems rely on multiple modality-specific blocks, leaving cross-sensor mismatches unresolved and making fusion unnecessarily this http URL cameras naturally provide a spatiotemporal edge signal, which we can treat as an intrinsic edge field to anchor a unified latent representation, termed the Event Edge Space. Building on this idea, we introduce $x^2$-Fusion, which reframes multimodal fusion as representation unification: event-derived spatiotemporal edges define an edge-centric homogeneous space, and image and LiDAR features are explicitly aligned in this shared this http URL this space, we perform reliability-aware adaptive fusion to estimate modality reliability and emphasize stable cues under degradation. We further employ cross-dimension contrast learning to tightly couple 2D optical flow with 3D scene flow. Extensive experiments on both synthetic and real benchmarks show that $x^2$-Fusion achieves state-of-the-art accuracy under standard conditions and delivers substantial improvements in challenging scenarios.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16671
PDF
-
LenghuSky-8: An 8-Year All-Sky Cloud Dataset with Star-Aware Masks and Alt-Az Calibration for Segmentation and Nowcasting
2026-03-17 12:05:21Yicheng Rui, Xiao-Wei Duan, Licai Deng, Fan Yang, Zhengming Dang, Zhengjun Du, Junhao Peng, Wenhao Chu, Umut Mahmut, Kexin Li, Yiyun Wu, Fabo FengAbstract
Ground-based time-domain observatories require minute-by-minute, site-scale awareness of cloud cover, yet existing all-sky datasets are short, daylight-biased, or lack astrometric calibration. We present LenghuSky-8, an eight-year (2018-2025) all-sky imaging dataset from a premier astronomical site, comprising 429,620 $512 \times 512$ frames with 81.2% night-time coverage, star-aware cloud masks, background masks, and per-pixel altitude-azimuth (Alt-Az) calibration. For robust cloud segmentation across day, night, and lunar phases, we train a linear probe on DINOv3 local features and obtain 93.3% $\pm$ 1.1% overall accuracy on a balanced, manually labeled set of 1,111 images. Using stellar astrometry, we map each pixel to local alt-az coordinates and measure calibration uncertainties of approximately 0.37 deg at zenith and approximately 1.34 deg at 30 deg altitude, sufficient for integration with telescope schedulers. Beyond segmentation, we introduce a short-horizon nowcasting benchmark over per-pixel three-class logits (sky/cloud/contamination) with four baselines: persistence (copying the last frame), optical flow, ConvLSTM, and VideoGPT. ConvLSTM performs best but yields only limited gains over persistence, underscoring the difficulty of near-term cloud evolution. We release the dataset, calibrations, and an open-source toolkit for loading, evaluation, and scheduler-ready alt-az maps to boost research in segmentation, nowcasting, and autonomous observatory operations.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16429
PDF
-
Towards Generalizable Robotic Manipulation in Dynamic Environments
2026-03-16 17:59:57Heng Fang, Shangru Li, Shuhan Wang, Xuanyang Xi, Dingkang Liang, Xiang BaiAbstract
Vision-Language-Action (VLA) models excel in static manipulation but struggle in dynamic environments with moving targets. This performance gap primarily stems from a scarcity of dynamic manipulation datasets and the reliance of mainstream VLAs on single-frame observations, restricting their spatiotemporal reasoning capabilities. To address this, we introduce DOMINO, a large-scale dataset and benchmark for generalizable dynamic manipulation, featuring 35 tasks with hierarchical complexities, over 110K expert trajectories, and a multi-dimensional evaluation suite. Through comprehensive experiments, we systematically evaluate existing VLAs on dynamic tasks, explore effective training strategies for dynamic awareness, and validate the generalizability of dynamic data. Furthermore, we propose PUMA, a dynamics-aware VLA architecture. By integrating scene-centric historical optical flow and specialized world queries to implicitly forecast object-centric future states, PUMA couples history-aware perception with short-horizon prediction. Results demonstrate that PUMA achieves state-of-the-art performance, yielding a 6.3% absolute improvement in success rate over baselines. Moreover, we show that training on dynamic data fosters robust spatiotemporal representations that transfer to static tasks. All code and data are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15620
PDF
-
$text{F}^2text{HDR}$: Two-Stage HDR Video Reconstruction via Flow Adapter and Physical Motion Modeling
2026-03-16 07:27:17Huanjing Yue, Dawei Li, Shaoxiong Tu, Jingyu YangAbstract
Reconstructing High Dynamic Range (HDR) videos from sequences of alternating-exposure Low Dynamic Range (LDR) frames remains highly challenging, especially under dynamic scenes where cross-exposure inconsistencies and complex motion make inter-frame alignment difficult, leading to ghosting and detail loss. Existing methods often suffer from inaccurate alignment, suboptimal feature aggregation, and degraded reconstruction quality in motion-dominated regions. To address these challenges, we propose $\text{F}^2\text{HDR}$, a two-stage HDR video reconstruction framework that robustly perceives inter-frame motion and restores fine details in complex dynamic scenarios. The proposed framework integrates a flow adapter that adapts generic optical flow for robust cross-exposure alignment, a physical motion modeling to identify salient motion regions, and a motion-aware refinement network that aggregates complementary information while removing ghosting and noise. Extensive experiments demonstrate that $\text{F}^2\text{HDR}$ achieves state-of-the-art performance on real-world HDR video benchmarks, producing ghost-free and high-fidelity results under large motion and exposure variations.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14920
PDF
-
Zero-Shot Reconstruction of Animatable 3D Avatars with Cloth Dynamics from a Single Image
2026-03-16 03:12:27Joohyun Kwon, Geonhee Sim, Gyeongsik MoonAbstract
Existing single-image 3D human avatar methods primarily rely on rigid joint transformations, limiting their ability to model realistic cloth dynamics. We present DynaAvatar, a zero-shot framework that reconstructs animatable 3D human avatars with motion-dependent cloth dynamics from a single image. Trained on large-scale multi-person motion datasets, DynaAvatar employs a Transformer-based feed-forward architecture that directly predicts dynamic 3D Gaussian deformations without subject-specific optimization. To overcome the scarcity of dynamic captures, we introduce a static-to-dynamic knowledge transfer strategy: a Transformer pretrained on large-scale static captures provides strong geometric and appearance priors, which are efficiently adapted to motion-dependent deformations through lightweight LoRA fine-tuning on dynamic captures. We further propose the DynaFlow loss, an optical flow-guided objective that provides reliable motion-direction geometric cues for cloth dynamics in rendered space. Finally, we reannotate the missing or noisy SMPL-X fittings in existing dynamic capture datasets, as most public dynamic capture datasets contain incomplete or unreliable fittings that are unsuitable for training high-quality 3D avatar reconstruction models. Experiments demonstrate that DynaAvatar produces visually rich and generalizable animations, outperforming prior methods.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14772
PDF
-
RetimeGS: Continuous-Time Reconstruction of 4D Gaussian Splatting
2026-03-14 06:26:27Xuezhen Wang, Li Ma, Yulin Shen, Zeyu Wang, Pedro V. SanderAbstract
Temporal retiming, the ability to reconstruct and render dynamic scenes at arbitrary timestamps, is crucial for applications such as slow-motion playback, temporal editing, and post-production. However, most existing 4D Gaussian Splatting (4DGS) methods overfit at discrete frame indices but struggle to represent continuous-time frames, leading to ghosting artifacts when interpolating between timestamps. We identify this limitation as a form of temporal aliasing and propose RetimeGS, a simple yet effective 4DGS representation that explicitly defines the temporal behavior of the 3D Gaussian and mitigates temporal aliasing. To achieve smooth and consistent interpolation, we incorporate optical flow-guided initialization and supervision, triple-rendering supervision, and other targeted strategies. Together, these components enable ghost-free, temporally coherent rendering even under large motions. Experiments on datasets featuring fast motion, non-rigid deformation, and severe occlusions demonstrate that RetimeGS achieves superior quality and coherence over state-of-the-art methods.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13783
PDF
-
Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos
2026-03-12 15:31:32Shuo Sun, Unal Artan, Malcolm Mielle, Achim J. Lilienthaland, Martin MagnussonAbstract
We address the challenging problem of dense dynamic scene reconstruction and camera pose estimation from multiple freely moving cameras -- a setting that arises naturally when multiple observers capture a shared event. Prior approaches either handle only single-camera input or require rigidly mounted, pre-calibrated camera rigs, limiting their practical applicability. We propose a two-stage optimization framework that decouples the task into robust camera tracking and dense depth refinement. In the first stage, we extend single-camera visual SLAM to the multi-camera setting by constructing a spatiotemporal connection graph that exploits both intra-camera temporal continuity and inter-camera spatial overlap, enabling consistent scale and robust tracking. To ensure robustness under limited overlap, we introduce a wide-baseline initialization strategy using feed-forward reconstruction models. In the second stage, we refine depth and camera poses by optimizing dense inter- and intra-camera consistency using wide-baseline optical flow. Additionally, we introduce MultiCamRobolab, a new real-world dataset with ground-truth poses from a motion capture system. Finally, we demonstrate that our method significantly outperforms state-of-the-art feed-forward models on both synthetic and real-world benchmarks, while requiring less memory.
Abstract (translated)
我们解决了从多个自由移动的摄像机重建密集动态场景和估计相机姿态这一难题——这种情况自然地出现在多名观察者捕捉共享事件时。之前的解决方案要么只能处理单一摄像头输入，要么需要固定安装且预先校准好的多摄系统，这限制了它们的实际应用性。为此，我们提出了一种两阶段优化框架，将任务分解为稳健的摄像机跟踪和密集深度细化两个部分。 在第一阶段，我们将单摄视觉SLAM（同步定位与地图构建）扩展到多摄设置中，通过构建一个时空连接图来利用每个摄像头内的时间连续性和跨摄像头的空间重叠，从而使尺度一致且跟踪更加稳定。为了确保在有限重叠的情况下也能保持稳健性，我们引入了一种使用前馈重建模型的宽基线初始化策略。 第二阶段，我们通过对多摄和单摄内部的一致性进行优化来细化深度信息和摄像机姿态，具体而言就是通过宽基线光学流来实现密集的跨摄一致性优化。此外，为了验证我们的方法有效性，我们还引入了MultiCamRobolab，这是一个新的真实世界数据集，包含来自动作捕捉系统的地面真值位置。 最后，我们在合成和实际世界的基准测试中展示了我们的方法显著优于最新的前馈模型，并且在内存需求上也更为节省。
URL
https://arxiv.org/abs/2603.12064
PDF
-
HiSync: Spatio-Temporally Aligning Hand Motion from Wearable IMU and On-Robot Camera for Command Source Identification in Long-Range HRI
2026-03-12 11:10:54Chengwen Zhang, Chun Yu, Borong Zhuang, Haopeng Jin, Qingyang Wan, Zhuojun Li, Zhe He, Zhoutong Ye, Yu Mei, Chang Liu, Weinan Shi, Yuanchun ShiAbstract
Long-range Human-Robot Interaction (HRI) remains underexplored. Within it, Command Source Identification (CSI) - determining who issued a command - is especially challenging due to multi-user and distance-induced sensor ambiguity. We introduce HiSync, an optical-inertial fusion framework that treats hand motion as binding cues by aligning robot-mounted camera optical flow with hand-worn IMU signals. We first elicit a user-defined (N=12) gesture set and collect a multimodal command gesture dataset (N=38) in long-range multi-user HRI scenarios. Next, HiSync extracts frequency-domain hand motion features from both camera and IMU data, and a learned CSINet denoises IMU readings, temporally aligns modalities, and performs distance-aware multi-window fusion to compute cross-modal similarity of subtle, natural gestures, enabling robust CSI. In three-person scenes up to 34m, HiSync achieves 92.32% CSI accuracy, outperforming the prior SOTA by 48.44%. HiSync is also validated on real-robot deployment. By making CSI reliable and natural, HiSync provides a practical primitive and design guidance for public-space HRI.
Abstract (translated)
长距离人机交互（HRI）领域尚待探索。其中，命令来源识别（CSI）——即确定哪位用户发出的指令——由于多用户环境和因距离导致的传感器模糊性而尤为具有挑战性。我们提出了一种光学-惯性融合框架HiSync，该框架通过将机器人安装摄像头捕捉到的手部运动光流与佩戴在手上的IMU信号对齐来处理手势绑定线索。 首先，我们从一组12名用户中定义并征集了一系列自定义手势，并收集了长距离多用户HRI场景下的跨模态命令手势数据集（N=38）。接下来，HiSync框架会分别从摄像头和IMU数据中提取频域下手部运动特征；通过CSINet算法对IMU读数进行去噪处理并完成不同传感器的时间同步工作，并采用具有距离感知能力的多窗口融合技术计算细微自然手势间的跨模态相似度，从而实现可靠的CSI。 在三人场景（最远34米）中，HiSync框架实现了92.32%的CSI准确性，比之前的最佳解决方案高出48.44%。此外，HiSync还在实际机器人部署中进行了验证。通过确保CSI既可靠又自然，HiSync为公共场所的人机交互提供了一种实用的基础技术，并对未来的系统设计提出了指导性建议。
URL
https://arxiv.org/abs/2603.11809
PDF
-
Edge-Assisted Multi-Robot Visual-Inertial SLAM with Efficient Communication
2026-03-11 03:35:28Xin Liu, Shuhuan Wen, Jing Zhao, Tony Z. Qiu, Hong ZhangAbstract
The integration of cloud computing and edge computing is an effective way to achieve global consistent and real-time multi-robot Simultaneous Localization and Mapping (SLAM). Cloud computing effectively solves the problem of limited computing, communication and storage capacity of terminal equipment. However, limited bandwidth and extremely long communication links between terminal devices and the cloud result in serious performance degradation of multi-robot SLAM systems. To reduce the computational cost of feature tracking and improve the real-time performance of the robot, a lightweight SLAM method of optical flow tracking based on pyramid IMU prediction is proposed. On this basis, a centralized multi-robot SLAM system based on a robot-edge-cloud layered architecture is proposed to realize real-time collaborative SLAM. It avoids the problems of limited on-board computing resources and low execution efficiency of single robot. In this framework, only the feature points and keyframe descriptors are transmitted and lossless encoding and compression are carried out to realize real-time remote information transmission with limited bandwidth resources. This design reduces the actual bandwidth occupied in the process of data transmission, and does not cause the loss of SLAM accuracy caused by data compression. Through experimental verification on the EuRoC dataset, compared with the current most advanced local feature compression method, our method can achieve lower data volume feature transmission, and compared with the current advanced centralized multi-robot SLAM scheme, it can achieve the same or better positioning accuracy under low computational load.
Abstract (translated)
云计算与边缘计算的融合是实现全球一致性和实时多机器人同时定位与地图构建（SLAM）的有效途径。云计算有效地解决了终端设备在计算、通信和存储能力方面的局限性问题。然而，终端设备与云端之间的有限带宽以及极长的通信链路导致了多机器人SLAM系统性能严重下降。为了降低特征跟踪的计算成本并提高机器人的实时性能，提出了一种基于金字塔惯性测量单元（IMU）预测的光流跟踪轻量级SLAM方法。在此基础上，提出了一个基于机器人-边缘-云分层架构的集中式多机器人SLAM系统，以实现实时协作SLAM。这种方法避免了车载计算资源有限和单个机器人执行效率低的问题。在该框架中，仅传输特征点和关键帧描述符，并进行无损编码压缩，从而实现在有限带宽资源下的实时远程信息传输。这一设计减少了数据传输过程中的实际带宽占用，并且不会因数据压缩而导致SLAM精度的损失。通过在EuRoC数据集上的实验验证表明，与目前最先进的局部特征压缩方法相比，我们的方法可以实现更小的数据量特征传输；与当前先进的集中式多机器人SLAM方案相比，在低计算负载下可以达到相同或更好的定位精度。
URL
https://arxiv.org/abs/2603.11085
PDF
-
Robotic Ultrasound Makes CBCT Alive
2026-03-10 20:37:24Feng Li, Ziyuan Li, Zhongliang Jiang, Nassir Navab, Yuan BiAbstract
Intraoperative Cone Beam Computed Tomography (CBCT) provides a reliable 3D anatomical context essential for interventional planning. However, its static nature fails to provide continuous monitoring of soft-tissue deformations induced by respiration, probe pressure, and surgical manipulation, leading to navigation discrepancies. We propose a deformation-aware CBCT updating framework that leverages robotic ultrasound as a dynamic proxy to infer tissue motion and update static CBCT slices in real time. Starting from calibration-initialized alignment with linear correlation of linear combination (LC2)-based rigid refinement, our method establishes accurate multimodal correspondence. To capture intraoperative dynamics, we introduce the ultrasound correlation UNet (USCorUNet), a lightweight network trained with optical flow-guided supervision to learn deformation-aware correlation representations, enabling accurate, real-time dense deformation field estimation from ultrasound streams. The inferred deformation is spatially regularized and transferred to the CBCT reference to produce deformation-consistent visualizations without repeated radiation exposure. We validate the proposed approach through deformation estimation and ultrasound-guided CBCT updating experiments. Results demonstrate real-time end-to-end CBCT slice updating and physically plausible deformation estimation, enabling dynamic refinement of static CBCT guidance during robotic ultrasound-assisted interventions. The source code is publicly available at this https URL.
Abstract (translated)
术中锥形束计算机断层摄影（CBCT）提供了一种可靠的三维解剖学背景，这对于介入规划至关重要。然而，由于其静态特性，无法持续监测由呼吸、探针压力和手术操作引起的软组织变形，导致导航误差。我们提出了一种基于机器人超声的变形感知CBCT更新框架，利用动态代理来推断组织运动，并实时更新静态CBCT切片。从线性组合（LC2）基础刚体细化的校准初始化对齐开始，我们的方法建立了准确的多模式对应关系。为了捕捉术中动力学变化，我们引入了超声相关UNet (USCorUNet)，这是一个轻量级网络，在光流引导监督下训练，以学习感知变形的相关表示，能够从超声流中实现精确、实时密集变形场估计。推断出的变形经过空间正则化后被转移到CBCT参考上，从而生成不需重复辐射暴露的变形一致可视化图。我们通过变形估计和超声引导CBCT更新实验验证了该方法的有效性。结果表明，实现了端到端的实时CBCT切片更新，并且可以进行物理上合理的变形估算，使静态CBCT指导在机器人超声辅助介入过程中能够动态精炼化。源代码可在以下链接获取：[此URL]
URL
https://arxiv.org/abs/2603.10220
PDF
-
Streaming Autoregressive Video Generation via Diagonal Distillation
2026-03-10 10:45:24Jinxiu Liu, Xuanming Liu, Kangfu Mei, Yandong Wen, Ming-HsuanYang, Weiyang LiuAbstract
Large pretrained diffusion models have significantly enhanced the quality of generated videos, and yet their use in real-time streaming remains limited. Autoregressive models offer a natural framework for sequential frame synthesis but require heavy computation to achieve high fidelity. Diffusion distillation can compress these models into efficient few-step variants, but existing video distillation approaches largely adapt image-specific methods that neglect temporal dependencies. These techniques often excel in image generation but underperform in video synthesis, exhibiting reduced motion coherence, error accumulation over long sequences, and a latency-quality trade-off. We identify two factors that result in these limitations: insufficient utilization of temporal context during step reduction and implicit prediction of subsequent noise levels in next-chunk prediction (i.e., exposure bias). To address these issues, we propose Diagonal Distillation, which operates orthogonally to existing approaches and better exploits temporal information across both video chunks and denoising steps. Central to our approach is an asymmetric generation strategy: more steps early, fewer steps later. This design allows later chunks to inherit rich appearance information from thoroughly processed early chunks, while using partially denoised chunks as conditional inputs for subsequent synthesis. By aligning the implicit prediction of subsequent noise levels during chunk generation with the actual inference conditions, our approach mitigates error propagation and reduces oversaturation in long-range sequences. We further incorporate implicit optical flow modeling to preserve motion quality under strict step constraints. Our method generates a 5-second video in 2.61 seconds (up to 31 FPS), achieving a 277.3x speedup over the undistilled model.
Abstract (translated)
大型预训练扩散模型在生成视频的质量上有了显著的提升，然而它们在实时流媒体中的应用仍然受到限制。自回归模型为序列帧合成提供了一个自然框架，但要达到高保真度则需要大量计算资源。扩散蒸馏可以将这些大型模型压缩成高效的几步变体版本，但是现有的视频蒸馏方法大多借鉴了特定于图像的方法，忽视了时间依赖性。虽然这些技术在图像生成中表现出色，但在视频合成中的表现较差，表现为运动连贯性降低、长时间序列中的误差积累以及延迟与质量之间的权衡问题。 我们发现导致这些问题的两个关键因素是：减少步骤过程中对时间上下文利用不足和下一片段预测（即曝光偏差）中隐式预测随后噪声水平的问题。为了解决这些问题，我们提出了对角蒸馏方法，这种方法与现有的方法形成了补充，并能更好地利用视频片段以及去噪步之间的时空信息。我们的方法的核心是对称生成策略：早期步骤多、后期步骤少。这种设计使后续片段能够从先前充分处理的片段中继承丰富的视觉信息，同时将部分去噪后的片段作为条件输入用于后续合成。通过在片段生成期间与实际推断条件对齐隐式预测随后噪声水平，我们的方法减轻了误差传播，并减少了长时间序列中的过度饱和问题。 此外，我们还结合了隐式光流建模以在严格的步骤限制下保持运动质量。我们的方法可以在2.61秒内生成5秒钟的视频（最高可达31帧每秒），比未蒸馏模型的速度提高了277.3倍。
URL
https://arxiv.org/abs/2603.09488
PDF
-
HDR-NSFF: High Dynamic Range Neural Scene Flow Fields
2026-03-09 12:36:46Shin Dong-Yeon, Kim Jun-Seong, Kwon Byung-Ki, Tae-Hyun OhAbstract
Radiance of real-world scenes typically spans a much wider dynamic range than what standard cameras can capture. While conventional HDR methods merge alternating-exposure frames, these approaches are inherently constrained to 2D pixel-level alignment, often leading to ghosting artifacts and temporal inconsistency in dynamic scenes. To address these limitations, we present HDR-NSFF, a paradigm shift from 2D-based merging to 4D spatio-temporal modeling. Our framework reconstructs dynamic HDR radiance fields from alternating-exposure monocular videos by representing the scene as a continuous function of space and time, and is compatible with both neural radiance field and 4D Gaussian Splatting (4DGS) based dynamic representations. This unified end-to-end pipeline explicitly models HDR radiance, 3D scene flow, geometry, and tone-mapping, ensuring physical plausibility and global coherence. We further enhance robustness by (i) extending semantic-based optical flow with DINO features to achieve exposure-invariant motion estimation, and (ii) incorporating a generative prior as a regularizer to compensate for limited observation in monocular captures and saturation-induced information loss. To evaluate HDR space-time view synthesis, we present the first real-world HDR-GoPro dataset specifically designed for dynamic HDR scenes. Experiments demonstrate that HDR-NSFF recovers fine radiance details and coherent dynamics even under challenging exposure variations, thereby achieving state-of-the-art performance in novel space-time view synthesis. Project page: this https URL
Abstract (translated)
真实世界的场景亮度范围通常远超出标准相机能够捕捉的范围。虽然传统的高动态范围（HDR）方法通过合并不同曝光度的照片来解决这个问题，但这些方法本质上局限于二维像素级别的对齐，这往往会导致动态场景中的鬼影效应和时间上的一致性问题。为了克服这些限制，我们提出了HDR-NSFF，这是一种从基于2D融合转向4D时空建模的范式转变。 我们的框架通过将场景表示为时间和空间上的连续函数来从交替曝光的单目视频中重建动态HDR辐射场，并且与神经辐射场和4DGaussian Splatting（4DGS）基的动态表示兼容。这一统一的端到端流水线显式地建模了HDR辐射、3D场景流、几何结构及色调映射，确保物理上的可信度和全局一致性。 为了增强鲁棒性，我们通过(i) 使用DINO特征扩展基于语义的光流以实现曝光不变性的运动估计；(ii) 作为正则化器纳入生成先验来弥补单目捕捉中的有限观测以及饱和导致的信息损失。 为评估HDR时空视图合成的效果，我们首次提出了针对动态HDR场景的真实世界HDR-GoPro数据集。实验表明，HDR-NSFF即使在挑战性的曝光变化下也能恢复精细的辐射细节和连贯的动力学，并因此达到了新型时空视图合成中的最先进性能。 项目页面：[此URL](this%20https%20URL)
URL
https://arxiv.org/abs/2603.08313
PDF
-
Event-based Motion & Appearance Fusion for 6D Object Pose Tracking
2026-03-09 11:34:50Zhichao Li, Chiara Bartolozzi, Lorenzo Natale, Arren GloverAbstract
Object pose tracking is a fundamental and essential task for robotics to perform tasks in the home and industrial settings. The most commonly used sensors to do so are RGB-D cameras, which can hit limitations in highly dynamic environments due to motion blur and frame-rate constraints. Event cameras have remarkable features such as high temporal resolution and low latency, which make them a potentially ideal vision sensors for object pose tracking at high speed. Even so, there are still only few works on 6D pose tracking with event cameras. In this work, we take advantage of the high temporal resolution and propose a method that uses both a propagation step fused with a pose correction strategy. Specifically, we use 6D object velocity obtained from event-based optical flow for pose propagation, after which, a template-based local pose correction module is utilized for pose correction. Our learning-free method has comparable performance to the state-of-the-art algorithms, and in some cases out performs them for fast-moving objects. The results indicate the potential for using event cameras in highly-dynamic scenarios where the use of deep network approaches are limited by low update rates.
Abstract (translated)
物体姿态跟踪是机器人在家庭和工业环境中执行任务的基础且必不可少的任务。最常用的传感器是RGB-D相机，但在动态强烈的环境下，这些设备可能会因运动模糊和帧率限制而受到限制。事件相机具有高时间分辨率和低延迟等显著特性，使其成为高速物体姿态跟踪的理想视觉传感器。尽管如此，目前利用事件相机进行6D姿态追踪的研究仍然很少。 在这项工作中，我们充分利用了事件相机的高时间分辨率，并提出了一种结合传播步骤与姿态修正策略的方法。具体来说，我们使用从基于事件的光流中获得的6D物体速度来执行姿态传播，在此之后，采用基于模板的局部姿态校正模块来进行姿态修正。我们的无学习方法在性能上可媲美当前最先进的算法，并且对于快速移动的对象，在某些情况下甚至超越了它们的表现。这些结果表明，事件相机在更新率较低的传统深度网络受限的应用场景中具有巨大的潜力。 通过这种方法，我们不仅展示了如何克服传统RGB-D传感器的限制，还为高速度和动态环境中的物体姿态跟踪开辟了一条新的途径。这为进一步探索事件相机在机器人任务中的应用奠定了基础，并可能开启一系列新的应用场景和技术突破。
URL
https://arxiv.org/abs/2603.08264
PDF
-
Fine-Grained 3D Facial Reconstruction for Micro-Expressions
2026-03-07 05:34:57Che Sun, Xinjie Zhang, Rui Gao, Xu Chen, Yuwei Wu, Yunde JiaAbstract
Recent advances in 3D facial expression reconstruction have demonstrated remarkable performance in capturing macro-expressions, yet the reconstruction of micro-expressions remains unexplored. This novel task is particularly challenging due to the subtle, transient, and low-intensity nature of micro-expressions, which complicate the extraction of stable and discriminative features essential for accurate reconstruction. In this paper, we propose a fine-grained micro-expression reconstruction method that integrates a global dynamic feature capturing stable facial motion patterns with a locally-enriched feature incorporating multiple informative cues from 2D motions, facial priors and 3D facial geometry. Specifically, we devise a plug-and-play dynamic-encoded module to extract micro-expression feature for global facial action, allowing it to leverage prior knowledge from abundant macro-expression data to mitigate the scarcity of micro-expression data. Subsequently, a dynamic-guided mesh deformation module is designed for extracting aggregated local features from dense optical flow, sparse landmark cues and facial mesh geometry, which adaptively refines fine-grained facial micro-expression without compromising global 3D geometry. Extensive experiments on micro-expression datasets demonstrate that our method consistently outperforms state-of-the-art methods in both geometric accuracy and perceptual detail.
Abstract (translated)
最近在3D面部表情重建方面取得的进展展示了捕捉宏观表情方面的出色性能，然而，对微观表情（micro-expressions）的重建研究尚处于探索阶段。这一新颖的任务尤其具有挑战性，因为微表情通常非常微妙、短暂且强度低，这使得提取稳定而有区分度的特征变得复杂，这些特征对于准确地进行微表情重建至关重要。 在本文中，我们提出了一种细粒度的微表情重构方法，该方法结合了用于捕获面部整体运动模式的全局动态特性以及通过2D运动、面部先验知识和3D面部几何结构获取的局部丰富特征。具体而言，我们设计了一个即插即用的动态编码模块来提取面向全球面部动作的微表情特征，并利用宏表情数据中的先前知识以缓解微表情数据稀缺的问题。随后，我们还设计了动态引导网格变形模块，用于从密集光流、稀疏地标线索和面部网格几何结构中提取聚合局部特性，从而在不损害全局3D几何的情况下适应性地细化细微的面部微表情。 通过在多个微表情数据集上进行广泛的实验，我们的方法在几何精度和感知细节方面始终优于现有最先进的技术。
URL
https://arxiv.org/abs/2603.07043
PDF
-
SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics
2026-03-06 22:38:45Haoying Zhou, Chang Liu, Yimeng Wu, Junlin Wu, Zijian Wu, Yu Chung Lee, Sara Martuscelli, Spetimiu E. Salcudean, Gregory S. Fischer, Peter KazanzidesAbstract
Most existing robotic surgery systems adopt a human-in-the-loop paradigm, often with the surgeon directly teleoperating the robotic system. Adding intelligence to these robots would enable higher-level control, such as supervised autonomy or even full autonomy. However, artificial intelligence (AI) requires large amounts of training data, which is currently lacking. This work proposes SurgSync, a multi-modal data collection framework with offline and online synchronization to support training and real-time inference, respectively. The framework is implemented on a da Vinci Research Kit (dVRK) and introduces (1) dual-mode (online/offline-matching) synchronized recorders, (2) a modern stereo endoscope to achieve image quality on par with clinical systems, and (3) additional sensors such as a side-view camera and a novel capacitive contact sensor to provide ground truth contact data. The framework also incorporates a post-processing toolbox for tasks such as depth estimation, optical flow, and a practical kinematic reprojection method using Gaussian heatmap. User studies with participants of varying skill levels are performed with ex-vivo tissue to provide clinically realistic data, and a network for surgical skill assessment is employed to demonstrate utilization of the collected data. Through the user study experiments, we obtained a dataset of 214 validated instances across multiple canonical training tasks. All software and data are available at this http URL.
Abstract (translated)
大多数现有的机器人手术系统采用人在环路（human-in-the-loop）的模式，通常是由外科医生直接操控机器人系统。向这些机器人添加智能功能将能够实现更高层次的控制，比如监督自主性或完全自主性。然而，人工智能（AI）需要大量的训练数据，而目前这方面的数据严重不足。为此，本研究提出了一种名为SurgSync的多模态数据采集框架，该框架包括离线和在线同步两个部分，分别用于支持训练和实时推断。该框架在达芬奇机器人手术系统的研发套件（da Vinci Research Kit, dVRK）上实现，并引入了以下三个关键组件： 1. 双模式（在线/离线匹配）的同步记录器； 2. 一种现代立体内窥镜，能够在图像质量方面与临床系统相媲美； 3. 额外传感器，如侧视摄像头和创新的电容接触传感器，提供真实的接触数据。 此外，该框架还集成了一个后期处理工具包，用于执行深度估计、光学流等任务，并采用高斯热图（Gaussian heatmap）来实现实际运动学重投影方法。通过使用离体组织进行技能水平各异参与者的用户研究，获取了临床真实的数据；同时部署了一个手术技能评估网络以展示收集数据的利用情况。通过这些用户实验研究，我们获得了涵盖多个典型训练任务的214个验证实例的数据集。所有软件和数据均可在提供的网址上获得。
URL
https://arxiv.org/abs/2603.06919
PDF
-
An Extended Topological Model For High-Contrast Optical Flow
2026-03-06 20:18:01Brad Turow, Jose A. PereaAbstract
In this paper, we identify low-dimensional models for dense core subsets in the space of $3\times 3$ high-contrast optical flow patches sampled from the Sintel dataset. In particular, we leverage the theory of approximate and discrete circle bundles to identify a 3-manifold whose boundary is a previously proposed optical flow torus, together with disjoint circles corresponding to pairs of binary step-edge range image patches. The 3-manifold model we introduce provides an explanation for why the previously-proposed torus model could not be verified with direct methods (e.g., a straightforward persistent homology computation). We also demonstrate that nearly all optical flow patches in the top 1 percent by contrast norm are found near the family of binary step-edge circles described above, rather than the optical flow torus, and that these frequently occurring patches are concentrated near motion boundaries (which are of particular importance for computer vision tasks such as object segmentation and tracking). Our findings offer insights on the subtle interplay between topology and geometry in inference for visual data.
Abstract (translated)
在这篇论文中，我们为Sintel数据集中采样的高对比度$3\times 3$光学流补丁的空间中的密集核心子集识别出了低维模型。具体来说，我们利用近似和离散圆束理论来识别出一个三维流形，其边界是一个先前提出的光学流环面，并且与二进制阶跃边缘范围图像补丁对相对应的分离圆圈相交。 我们引入的三维流形模型解释了为什么之前的环面模型不能通过直接方法（例如，简单的持久同调计算）来验证。此外，我们也展示了几乎所有按对比度范数排名前1%的光学流补丁都靠近上述二进制阶跃边缘圆家族，而非光学流环面上。并且，这些频繁出现的补丁集中在运动边界附近（这对于如目标分割和跟踪等计算机视觉任务特别重要）。 我们的发现提供了关于在视觉数据推断中拓扑与几何之间微妙相互作用的理解。
URL
https://arxiv.org/abs/2603.06853
PDF
-
RealWonder: Real-Time Physical Action-Conditioned Video Generation
2026-03-05 18:22:54Wei Liu, Ziyu Chen, Zizhang Li, Yue Wang, Hong-Xing Yu, Jiajun WuAbstract
Current video generation models cannot simulate physical consequences of 3D actions like forces and robotic manipulations, as they lack structural understanding of how actions affect 3D scenes. We present RealWonder, the first real-time system for action-conditioned video generation from a single image. Our key insight is using physics simulation as an intermediate bridge: instead of directly encoding continuous actions, we translate them through physics simulation into visual representations (optical flow and RGB) that video models can process. RealWonder integrates three components: 3D reconstruction from single images, physics simulation, and a distilled video generator requiring only 4 diffusion steps. Our system achieves 13.2 FPS at 480x832 resolution, enabling interactive exploration of forces, robot actions, and camera controls on rigid objects, deformable bodies, fluids, and granular materials. We envision RealWonder opens new opportunities to apply video models in immersive experiences, AR/VR, and robot learning. Our code and model weights are publicly available in our project website: this https URL
Abstract (translated)
当前的视频生成模型无法模拟3D动作（如力和机器人操作）对物理世界的连续影响，因为它们缺乏对动作如何影响3D场景的结构理解。我们提出了一种名为RealWonder的新系统，这是首个基于单张图像进行动作条件下的实时视频生成系统的解决方案。 我们的关键洞察在于使用物理学仿真作为中间桥梁：不是直接编码连续的动作，而是通过物理学仿真将其转换为视觉表示（光流和RGB），这些是视频模型能够处理的。 RealWonder集成三大组件：从单一图片重建3D场景、物理仿真以及一个仅需4次扩散步骤的精简视频生成器。 该系统在480x832分辨率下可达到13.2帧每秒的速度，支持刚体对象、可变形物体、流体和颗粒材料上的力、机器人动作及相机控制交互式的探索。我们设想RealWonder将为视频模型在沉浸式体验、AR/VR以及机器人学习中的应用开辟新的机会。 我们的代码和模型权重可在项目网站上公开获取：[此URL](this https URL)
URL
https://arxiv.org/abs/2603.05449
PDF
-
FC-VFI: Faithful and Consistent Video Frame Interpolation for High-FPS Slow Motion Video Generation
2026-03-05 07:41:34Ganggui Ding, Hao Chen, Xiaogang XuAbstract
Large pre-trained video diffusion models excel in video frame interpolation but struggle to generate high fidelity frames due to reliance on intrinsic generative priors, limiting detail preservation from start and end frames. Existing methods often depend on motion control for temporal consistency, yet dense optical flow is error-prone, and sparse points lack structural context. In this paper, we propose FC-VFI for faithful and consistent video frame interpolation, supporting \(4\times\)x and \(8\times\) interpolation, boosting frame rates from 30 FPS to 120 and 240 FPS at \(2560\times 1440\)resolution while preserving visual fidelity and motion consistency. We introduce a temporal modeling strategy on the latent sequences to inherit fidelity cues from start and end frames and leverage semantic matching lines for structure-aware motion guidance, improving motion consistency. Furthermore, we propose a temporal difference loss to mitigate temporal inconsistencies. Extensive experiments show FC-VFI achieves high performance and structural integrity across diverse scenarios.
Abstract (translated)
大型预训练视频扩散模型在视频帧插值方面表现出色，但在生成高保真度的帧时却由于依赖于内在的生成先验而难以保留细节。现有的方法通常依靠运动控制来保持时间一致性，然而密集光流容易出错，稀疏点则缺乏结构上下文信息。在这篇论文中，我们提出了FC-VFI（Faithful and Consistent Video Frame Interpolation）模型，旨在实现忠实且一致的视频帧插值，并支持4倍和8倍的插值率，在2560×1440分辨率下将帧率从30 FPS提升至120 FPS及240 FPS的同时保持视觉保真度与运动一致性。我们引入了一种时序建模策略，用于在潜在序列中继承起始帧和结束帧的保真度线索，并利用语义匹配线来为结构感知运动引导提供支持，从而提高运动一致性。此外，我们还提出了一种时间差异损失函数以减少时间上的不一致现象。通过广泛的实验验证，FC-VFI模型在各种场景下均展现出高水准的表现及结构完整性。
URL
https://arxiv.org/abs/2603.04899
PDF
