Optical_Flow
-
Relative Localization System Design for SnailBot: A Modular Self-reconfigurable Robot
Abstract
This paper presents the design and implementation of a relative localization system for SnailBot, a modular self reconfigurable robot. The system integrates ArUco marker recognition, optical flow analysis, and IMU data processing into a unified fusion framework, enabling robust and accurate relative positioning for collaborative robotic tasks. Experimental validation demonstrates the effectiveness of the system in realtime operation, with a rule based fusion strategy ensuring reliability across dynamic scenarios. The results highlight the potential for scalable deployment in modular robotic systems.
Abstract (translated)
本文介绍了SnailBot（一种模块化自重构机器人）相对定位系统的设计与实现。该系统将ArUco标记识别、光学流分析和惯性测量单元(IMU)数据处理整合到一个统一的融合框架中，从而实现了协作机器人任务中的稳健且精确的相对定位。实验验证表明，该系统在实时操作中表现有效，并通过基于规则的融合策略确保了其在动态场景下的可靠性。研究结果突显了该技术在模块化机器人系统中可扩展部署的潜力。
URL
https://arxiv.org/abs/2512.21226
PDF
-
Optical Flow-Guided 6DoF Object Pose Tracking with an Event Camera
2025-12-24 08:40:57Zibin Liu, Banglei Guan, Yang Shang, Shunkun Liang, Zhenbao Yu, Qifeng YuAbstract
Object pose tracking is one of the pivotal technologies in multimedia, attracting ever-growing attention in recent years. Existing methods employing traditional cameras encounter numerous challenges such as motion blur, sensor noise, partial occlusion, and changing lighting conditions. The emerging bio-inspired sensors, particularly event cameras, possess advantages such as high dynamic range and low latency, which hold the potential to address the aforementioned challenges. In this work, we present an optical flow-guided 6DoF object pose tracking method with an event camera. A 2D-3D hybrid feature extraction strategy is firstly utilized to detect corners and edges from events and object models, which characterizes object motion precisely. Then, we search for the optical flow of corners by maximizing the event-associated probability within a spatio-temporal window, and establish the correlation between corners and edges guided by optical flow. Furthermore, by minimizing the distances between corners and edges, the 6DoF object pose is iteratively optimized to achieve continuous pose tracking. Experimental results of both simulated and real events demonstrate that our methods outperform event-based state-of-the-art methods in terms of both accuracy and robustness.
Abstract (translated)
物体姿态跟踪是多媒体领域中的关键技术之一，近年来备受关注。现有的使用传统摄像头的方法面临诸如运动模糊、传感器噪声、部分遮挡和光照变化等众多挑战。新兴的仿生传感设备，特别是事件相机（event camera），具有高动态范围和低延迟的优点，有望解决上述问题。在本工作中，我们提出了一种基于事件相机的光流引导6自由度物体姿态跟踪方法。首先采用2D-3D混合特征提取策略，从事件和物体模型中检测出角点和边缘，精确表征了物体的运动。接着，在时空窗口内通过最大化与事件关联的概率来搜索角点的光流，并基于光流建立了角点和边缘之间的关联性。此外，通过最小化角点和边缘之间的距离，迭代优化6自由度物体姿态，实现连续的姿态跟踪。模拟数据和真实事件实验结果表明，我们的方法在精度和鲁棒性方面都优于现有的最先进的事件驱动方法。
URL
https://arxiv.org/abs/2512.21053
PDF
-
Decoding Predictive Inference in Visual Language Processing via Spatiotemporal Neural Coherence
2025-12-24 04:19:20Sean C. Borneman, Julia Krebs, Ronnie B. Wilbur, Evie A. MalaiaAbstract
Human language processing relies on the brain's capacity for predictive inference. We present a machine learning framework for decoding neural (EEG) responses to dynamic visual language stimuli in Deaf signers. Using coherence between neural signals and optical flow-derived motion features, we construct spatiotemporal representations of predictive neural dynamics. Through entropy-based feature selection, we identify frequency-specific neural signatures that differentiate interpretable linguistic input from linguistically disrupted (time-reversed) stimuli. Our results reveal distributed left-hemispheric and frontal low-frequency coherence as key features in language comprehension, with experience-dependent neural signatures correlating with age. This work demonstrates a novel multimodal approach for probing experience-driven generative models of perception in the brain.
Abstract (translated)
人类语言处理依赖于大脑进行预测性推理的能力。我们提出了一种机器学习框架，用于解码聋人手语使用者对动态视觉语言刺激的神经（EEG）反应。通过将神经信号与基于光流的运动特征之间的相干性结合，我们构建了预测性神经动力学的空间时间表示。利用熵为基础的特征选择方法，我们识别出频率特异性的神经标志物，这些标志物能够区分可理解的语言输入和语言被打乱（时间倒置）的刺激。我们的研究结果揭示了左半球广泛分布以及额叶低频相干是语言理解的关键特征，并且随着经验积累而变化的神经标志物与年龄相关联。这项工作展示了探索大脑中由经验驱动的感知生成模型的新颖多模态方法。
URL
https://arxiv.org/abs/2512.20929
PDF
-
Learning Skills from Action-Free Videos
2025-12-23 05:03:33Hung-Chieh Fang, Kuo-Han Hung, Chu-Rong Chen, Po-Jung Chou, Chun-Kai Yang, Po-Chen Ko, Yu-Chiang Wang, Yueh-Hua Wu, Min-Hung Chen, Shao-Hua SunAbstract
Learning from videos offers a promising path toward generalist robots by providing rich visual and temporal priors beyond what real robot datasets contain. While existing video generative models produce impressive visual predictions, they are difficult to translate into low-level actions. Conversely, latent-action models better align videos with actions, but they typically operate at the single-step level and lack high-level planning capabilities. We bridge this gap by introducing Skill Abstraction from Optical Flow (SOF), a framework that learns latent skills from large collections of action-free videos. Our key idea is to learn a latent skill space through an intermediate representation based on optical flow that captures motion information aligned with both video dynamics and robot actions. By learning skills in this flow-based latent space, SOF enables high-level planning over video-derived skills and allows for easier translation of these skills into actions. Experiments show that our approach consistently improves performance in both multitask and long-horizon settings, demonstrating the ability to acquire and compose skills directly from raw visual data.
Abstract (translated)
从视频中学习为通用机器人的开发提供了一条充满希望的道路，因为它提供了超出真实机器人数据集的丰富视觉和时间先验。尽管现有的视频生成模型能够产生令人印象深刻的视觉预测，但它们很难转化为低级别的行动指令。相反，基于潜在动作的模型更好地将视频与具体操作对齐，不过通常这些模型只在单步水平上运行，并且缺乏高级规划能力。为了弥合这一差距，我们引入了Skill Abstraction from Optical Flow (SOF)框架，该框架能够从大量的无动作视频集合中学习潜在技能。我们的核心理念是通过基于光学流的中间表示来学习潜在的技能空间，这种表示方法能捕捉与视频动态和机器人操作都对齐的运动信息。通过对这种基于流动的潜在空间中的技能进行学习，SOF能够在基于视频获取的技能上实现高级规划，并使这些技能更容易转化为具体行动。实验表明，我们的方法在多任务和长时展望设置中均表现出持续改进性能的能力，展示了直接从原始视觉数据中获得和组合技能的能力。
URL
https://arxiv.org/abs/2512.20052
PDF
-
SynergyWarpNet: Attention-Guided Cooperative Warping for Neural Portrait Animation
2025-12-19 08:21:23Shihang Li, Zhiqiang Gong, Minming Ye, Yue Gao, Wen YaoAbstract
Recent advances in neural portrait animation have demonstrated remarked potential for applications in virtual avatars, telepresence, and digital content creation. However, traditional explicit warping approaches often struggle with accurate motion transfer or recovering missing regions, while recent attention-based warping methods, though effective, frequently suffer from high complexity and weak geometric grounding. To address these issues, we propose SynergyWarpNet, an attention-guided cooperative warping framework designed for high-fidelity talking head synthesis. Given a source portrait, a driving image, and a set of reference images, our model progressively refines the animation in three stages. First, an explicit warping module performs coarse spatial alignment between the source and driving image using 3D dense optical flow. Next, a reference-augmented correction module leverages cross-attention across 3D keypoints and texture features from multiple reference images to semantically complete occluded or distorted regions. Finally, a confidence-guided fusion module integrates the warped outputs with spatially-adaptive fusing, using a learned confidence map to balance structural alignment and visual consistency. Comprehensive evaluations on benchmark datasets demonstrate state-of-the-art performance.
Abstract (translated)
最近在神经肖像动画领域的进展展示了其在虚拟化身、远程呈现和数字内容创作方面的重要潜力。然而，传统的显式扭曲方法通常难以实现精确的动作转移或恢复缺失区域，而基于注意力的扭曲方法虽然有效，但常常面临高复杂度和弱几何基础的问题。为了解决这些问题，我们提出了SynergyWarpNet，这是一种以注意力引导的合作扭曲框架，专门用于高质量的话筒头合成。 给定一个源肖像、一张驱动图像以及一系列参考图像，我们的模型通过三个阶段逐步细化动画效果： 1. **显式扭曲模块**：使用3D密集光流在源图像和驱动图像之间执行粗略的空间对齐。 2. **参考增强校正模块**：利用跨注意力机制，在多个参考图像的3D关键点和纹理特征之间进行交互，从而语义化地完成遮挡或变形区域。 3. **信心引导融合模块**：通过空间自适应融合方法整合扭曲后的输出，并使用学习到的信心图来平衡结构对齐与视觉一致性。 在基准数据集上的全面评估表明了该模型的最先进的性能。
URL
https://arxiv.org/abs/2512.17331
PDF
-
DESSERT: Diffusion-based Event-driven Single-frame Synthesis via Residual Training
2025-12-19 08:12:20Jiyun Kong, Jun-Hyuk Kim, Jong-Seok LeeAbstract
Video frame prediction extrapolates future frames from previous frames, but suffers from prediction errors in dynamic scenes due to the lack of information about the next frame. Event cameras address this limitation by capturing per-pixel brightness changes asynchronously with high temporal resolution. Prior research on event-based video frame prediction has leveraged motion information from event data, often by predicting event-based optical flow and reconstructing frames via pixel warping. However, such approaches introduce holes and blurring when pixel displacement is inaccurate. To overcome this limitation, we propose DESSERT, a diffusion-based event-driven single-frame synthesis framework via residual training. Leveraging a pre-trained Stable Diffusion model, our method is trained on inter-frame residuals to ensure temporal consistency. The training pipeline consists of two stages: (1) an Event-to-Residual Alignment Variational Autoencoder (ER-VAE) that aligns the event frame between anchor and target frames with the corresponding residual, and (2) a diffusion model that denoises the residual latent conditioned on event data. Furthermore, we introduce Diverse-Length Temporal (DLT) augmentation, which improves robustness by training on frame segments of varying temporal lengths. Experimental results demonstrate that our method outperforms existing event-based reconstruction, image-based video frame prediction, event-based video frame prediction, and one-sided event-based video frame interpolation methods, producing sharper and more temporally consistent frame synthesis.
Abstract (translated)
视频帧预测通过从前一帧中推断未来帧来进行，但在动态场景中会因为缺乏下一帧的信息而导致预测误差。事件相机（event cameras）通过异步捕捉每个像素的亮度变化来解决这一问题，从而提供高时间分辨率的数据。先前基于事件的数据进行视频帧预测的研究通常利用事件数据中的运动信息，例如通过预测基于事件的光流并使用像素扭曲重建帧的方式实现。然而，此类方法在像素位移不准确时会产生孔洞和模糊。 为了克服这些限制，我们提出了DESSERT框架，这是一个基于扩散模型（diffusion-based）并通过残差训练进行单帧合成的事件驱动框架。该方法利用预训练的Stable Diffusion模型，在帧间差异的基础上进行训练以确保时间一致性。训练流程分为两个阶段： 1. 事件到残差对齐变分自编码器（ER-VAE），将锚定帧和目标帧之间的事件帧与对应的残差点对齐。 2. 扩散模型，基于事件数据清理残差隐变量的噪声。 此外，我们还引入了多样时间长度增强技术（DLT augmentation），通过在不同时间长度的帧段上进行训练来提高鲁棒性。实验结果显示，我们的方法超过了现有的事件基重构、图像基视频帧预测、事件基视频帧预测以及单向事件基视频帧插值的方法，在生成更清晰且时间一致性更好的合成帧方面表现出色。
URL
https://arxiv.org/abs/2512.17323
PDF
-
A2VISR: An Active and Adaptive Ground-Aerial Localization System Using Visual Inertial and Single-Range Fusion
Abstract
It's a practical approach using the ground-aerial collaborative system to enhance the localization robustness of flying robots in cluttered environments, especially when visual sensors degrade. Conventional approaches estimate the flying robot's position using fixed cameras observing pre-attached markers, which could be constrained by limited distance and susceptible to capture failure. To address this issue, we improve the ground-aerial localization framework in a more comprehensive manner, which integrates active vision, single-ranging, inertial odometry, and optical flow. First, the designed active vision subsystem mounted on the ground vehicle can be dynamically rotated to detect and track infrared markers on the aerial robot, improving the field of view and the target recognition with a single camera. Meanwhile, the incorporation of single-ranging extends the feasible distance and enhances re-capture capability under visual degradation. During estimation, a dimension-reduced estimator fuses multi-source measurements based on polynomial approximation with an extended sliding window, balancing computational efficiency and redundancy. Considering different sensor fidelities, an adaptive sliding confidence evaluation algorithm is implemented to assess measurement quality and dynamically adjust the weighting parameters based on moving variance. Finally, extensive experiments under conditions such as smoke interference, illumination variation, obstacle occlusion, prolonged visual loss, and extended operating range demonstrate that the proposed approach achieves robust online localization, with an average root mean square error of approximately 0.09 m, while maintaining resilience to capture loss and sensor failures.
Abstract (translated)
这是一种实用的方法，利用地面-空中协作系统来增强飞行机器人在复杂环境中定位的鲁棒性，特别是在视觉传感器性能下降的情况下。传统方法通常使用固定摄像头观测预先附着的标记来估计飞行机器人的位置，这种方法受限于观察距离，并且容易因捕捉失败而失效。为了解决这个问题，我们从更全面的角度改进了地面-空中定位框架，该框架整合了主动视觉、单程测距、惯性里程计和光学流。 首先，安装在地面上的车辆上的设计好的主动视觉子系统可以动态旋转以检测和跟踪飞行机器人上红外标记的位置，从而提高视野范围并用单一摄像头实现目标识别。与此同时，引入单程测距技术扩展了可行距离，并且增强了在视觉传感器性能下降时重新捕捉的能力。 在估计过程中，采用一种基于多项式逼近的缩减维数估计算法，在一个延伸的时间窗口内融合多源测量值，从而平衡计算效率和冗余度。考虑到不同传感器的数据精确性差异，我们实施了一种自适应滑动置信评估算法来评估测量质量，并根据移动方差动态调整权重参数。 最后，在烟雾干扰、光照变化、障碍物遮挡、长时间视觉丢失以及扩展操作范围等条件下进行了广泛的实验，结果显示所提出的方案实现了鲁棒的在线定位，平均均方根误差约为0.09米，同时保持了对捕捉失败和传感器故障的抗性。
URL
https://arxiv.org/abs/2512.16367
PDF
-
Charge: A Comprehensive Novel View Synthesis Benchmark and Dataset to Bind Them All
2025-12-15 18:33:08Michal Nazarczuk, Thomas Tanay, Arthur Moreau, Zhensong Zhang, Eduardo P\'erez-PelliteroAbstract
This paper presents a new dataset for Novel View Synthesis, generated from a high-quality, animated film with stunning realism and intricate detail. Our dataset captures a variety of dynamic scenes, complete with detailed textures, lighting, and motion, making it ideal for training and evaluating cutting-edge 4D scene reconstruction and novel view generation models. In addition to high-fidelity RGB images, we provide multiple complementary modalities, including depth, surface normals, object segmentation and optical flow, enabling a deeper understanding of scene geometry and motion. The dataset is organised into three distinct benchmarking scenarios: a dense multi-view camera setup, a sparse camera arrangement, and monocular video sequences, enabling a wide range of experimentation and comparison across varying levels of data sparsity. With its combination of visual richness, high-quality annotations, and diverse experimental setups, this dataset offers a unique resource for pushing the boundaries of view synthesis and 3D vision.
Abstract (translated)
本文介绍了一个新的用于新颖视图合成的数据集，该数据集源自一部具有惊人真实感和复杂细节的高质量动画电影。我们的数据集捕捉了各种动态场景，包括详细的纹理、光照和运动，非常适合训练和评估前沿的4D场景重建和新颖视图生成模型。除了高保真的RGB图像外，我们还提供了多种互补模式，如深度信息、表面法线、对象分割及光流，这有助于更深入地理解场景几何结构与运动。 数据集按三个不同的基准测试情景组织：密集多视角相机设置、稀疏相机排列以及单目视频序列。这使得在不同程度的数据稀疏性上进行广泛实验和对比成为可能。凭借其视觉丰富度、高质量注释及多样化的实验设定，该数据集为推进视图合成与3D视觉领域的边界提供了独特资源。
URL
https://arxiv.org/abs/2512.13639
PDF
-
Motus: A Unified Latent Action World Model
2025-12-15 06:58:40Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, Hongyan Zhao, Hanyu Liu, Zhizhong Su, Lei Ma, Hang Su, Jun ZhuAbstract
While a general embodied agent must function as a unified system, current methods are built on isolated models for understanding, world modeling, and control. This fragmentation prevents unifying multimodal generative capabilities and hinders learning from large-scale, heterogeneous data. In this paper, we propose Motus, a unified latent action world model that leverages existing general pretrained models and rich, sharable motion information. Motus introduces a Mixture-of-Transformer (MoT) architecture to integrate three experts (i.e., understanding, video generation, and action) and adopts a UniDiffuser-style scheduler to enable flexible switching between different modeling modes (i.e., world models, vision-language-action models, inverse dynamics models, video generation models, and video-action joint prediction models). Motus further leverages the optical flow to learn latent actions and adopts a recipe with three-phase training pipeline and six-layer data pyramid, thereby extracting pixel-level "delta action" and enabling large-scale action pretraining. Experiments show that Motus achieves superior performance against state-of-the-art methods in both simulation (a +15% improvement over X-VLA and a +45% improvement over Pi0.5) and real-world scenarios(improved by +11~48%), demonstrating unified modeling of all functionalities and priors significantly benefits downstream robotic tasks.
Abstract (translated)
虽然通用具身代理必须作为一个统一系统运行，但目前的方法是基于用于理解、世界建模和控制的孤立模型构建的。这种碎片化阻碍了多模态生成能力的一体化，并且妨碍从大规模异构数据中学习。在本文中，我们提出了Motus，这是一种统一的动作潜变量世界模型，它利用现有的通用预训练模型和丰富的可共享运动信息。Motus引入了一种Transformer混合架构（MoT），以整合三个专家领域（即理解、视频生成和动作）并采用类似UniDiffuser的调度器来实现在不同建模模式之间的灵活切换（例如：世界模型、视觉-语言-行动模型、逆动力学模型、视频生成模型以及视频-行动联合预测模型）。此外，Motus利用光学流学习潜变量动作，并采用了包含三个阶段训练流程和六层数据金字塔的配方方法，从而提取像素级别的“delta动作”并实现大规模的动作预训练。实验表明，在模拟环境（相对于X-VLA性能提高15%，相较于Pi0.5则提高了45%）以及真实世界场景中（性能提升范围为+11至+48%），Motus的表现优于最先进的方法，证明了一体化模型能够统一所有功能和先验知识，显著地有利于下游的机器人任务。
URL
https://arxiv.org/abs/2512.13030
PDF
-
SMRABooth: Subject and Motion Representation Alignment for Customized Video Generation
2025-12-13 05:44:20Xuancheng Xu, Yaning Li, Sisi You, Bing-Kun BaoAbstract
Customized video generation aims to produce videos that faithfully preserve the subject's appearance from reference images while maintaining temporally consistent motion from reference videos. Existing methods struggle to ensure both subject appearance similarity and motion pattern consistency due to the lack of object-level guidance for subject and motion. To address this, we propose SMRABooth, which leverages the self-supervised encoder and optical flow encoder to provide object-level subject and motion representations. These representations are aligned with the model during the LoRA fine-tuning process. Our approach is structured in three core stages: (1) We exploit subject representations via a self-supervised encoder to guide subject alignment, enabling the model to capture overall structure of subject and enhance high-level semantic consistency. (2) We utilize motion representations from an optical flow encoder to capture structurally coherent and object-level motion trajectories independent of appearance. (3) We propose a subject-motion association decoupling strategy that applies sparse LoRAs injection across both locations and timing, effectively reducing interference between subject and motion LoRAs. Extensive experiments show that SMRABooth excels in subject and motion customization, maintaining consistent subject appearance and motion patterns, proving its effectiveness in controllable text-to-video generation.
Abstract (translated)
定制化的视频生成旨在根据参考图像忠实保留主体外观的同时，依据参考视频保持时间上的一致性动作。现有方法由于缺乏对主体和运动的对象级指导，在确保相似的主体外观与一致的动作模式方面遇到了挑战。为了解决这一问题，我们提出了SMRABooth方案，该方案利用自监督编码器和光学流编码器来提供对象级别的主题和运动表示，并在LoRA微调过程中对这些表示进行校准。我们的方法由三个核心阶段组成： 1. 我们通过一个自监督编码器来使用主体表示，以引导模型捕捉整个主体结构并增强高层次的语义一致性。 2. 采用光学流编码器提供的运动表示，独立于外观特征地捕获结构一致且对象级别的运动轨迹。 3. 提出了一种解耦主题-运动关联策略，在位置和时间上稀疏注入LoRA，有效减少了主体与运动LoRA之间的干扰。 广泛实验表明，SMRABooth在定制化的主题和动作方面表现优异，能够保持一致的主题外观及动作模式，证明了其在可控文本到视频生成中的有效性。
URL
https://arxiv.org/abs/2512.12193
PDF
-
Structure From Tracking: Distilling Structure-Preserving Motion for Video Generation
2025-12-12 18:56:35Yang Fei, George Stoica, Jingyuan Liu, Qifeng Chen, Ranjay Krishna, Xiaojuan Wang, Benlin LiuAbstract
Reality is a dance between rigid constraints and deformable structures. For video models, that means generating motion that preserves fidelity as well as structure. Despite progress in diffusion models, producing realistic structure-preserving motion remains challenging, especially for articulated and deformable objects such as humans and animals. Scaling training data alone, so far, has failed to resolve physically implausible transitions. Existing approaches rely on conditioning with noisy motion representations, such as optical flow or skeletons extracted using an external imperfect model. To address these challenges, we introduce an algorithm to distill structure-preserving motion priors from an autoregressive video tracking model (SAM2) into a bidirectional video diffusion model (CogVideoX). With our method, we train SAM2VideoX, which contains two innovations: (1) a bidirectional feature fusion module that extracts global structure-preserving motion priors from a recurrent model like SAM2; (2) a Local Gram Flow loss that aligns how local features move together. Experiments on VBench and in human studies show that SAM2VideoX delivers consistent gains (+2.60\% on VBench, 21-22\% lower FVD, and 71.4\% human preference) over prior baselines. Specifically, on VBench, we achieve 95.51\%, surpassing REPA (92.91\%) by 2.60\%, and reduce FVD to 360.57, a 21.20\% and 22.46\% improvement over REPA- and LoRA-finetuning, respectively. The project website can be found at this https URL .
Abstract (translated)
现实是在刚性约束和可变形结构之间的舞蹈。对于视频模型而言，这意味着生成既保持保真度又保留结构的运动。尽管在扩散模型方面取得了进展，但产生真实且结构保守的动作仍然具有挑战性，特别是对于像人类和动物这样的连杆和可变形对象来说。仅靠扩大训练数据尚未解决物理上不可能的过渡问题。现有的方法依赖于使用嘈杂的动作表示进行条件设置，例如通过外部不完美的模型提取的光学流或骨架。 为了解决这些挑战，我们引入了一种算法，将结构保守运动先验从自回归视频跟踪模型（SAM2）提炼到双向视频扩散模型（CogVideoX）中。利用我们的方法，我们训练了名为SAM2VideoX的新模型，该模型包含两项创新： 1. 一个双向特征融合模块，可以从像SAM2这样的递归模型中提取全局结构保守运动先验。 2. 局部Gram Flow损失，用于对齐局部特征如何共同移动。 在VBench和人类研究中的实验表明，SAM2VideoX相对于之前的基准线提供了持续的改进（VBench上的+2.60%，FVD降低了21-22%，并且在71.4%的人类偏好度上表现出优势）。具体来说，在VBench上我们达到了95.51%，超越了REPA的92.91%，将FVD降低到了360.57，分别比REPA和LoRA微调改进了21.20% 和 22.46%。 项目的网站可以在[这里](https://this-should-be-the-url-of-the-project)找到。请注意，链接处应填写实际的项目网址。
URL
https://arxiv.org/abs/2512.11792
PDF
-
Surveillance Video-Based Traffic Accident Detection Using Transformer Architecture
2025-12-12 07:57:36Tanu Singh, Pranamesh Chakraborty, Long T. TruongAbstract
Road traffic accidents represent a leading cause of mortality globally, with incidence rates rising due to increasing population, urbanization, and motorization. Rising accident rates raise concerns about traffic surveillance effectiveness. Traditional computer vision methods for accident detection struggle with limited spatiotemporal understanding and poor cross-domain generalization. Recent advances in transformer architectures excel at modeling global spatial-temporal dependencies and parallel computation. However, applying these models to automated traffic accident detection is limited by small, non-diverse datasets, hindering the development of robust, generalizable systems. To address this gap, we curated a comprehensive and balanced dataset that captures a wide spectrum of traffic environments, accident types, and contextual variations. Utilizing the curated dataset, we propose an accident detection model based on a transformer architecture using pre-extracted spatial video features. The architecture employs convolutional layers to extract local correlations across diverse patterns within a frame, while leveraging transformers to capture sequential-temporal dependencies among the retrieved features. Moreover, most existing studies neglect the integration of motion cues, which are essential for understanding dynamic scenes, especially during accidents. These approaches typically rely on static features or coarse temporal information. In this study, multiple methods for incorporating motion cues were evaluated to identify the most effective strategy. Among the tested input approaches, concatenating RGB features with optical flow achieved the highest accuracy at 88.3%. The results were further compared with vision language models (VLM) such as GPT, Gemini, and LLaVA-NeXT-Video to assess the effectiveness of the proposed method.
Abstract (translated)
道路交通事故是全球死亡的主要原因之一，由于人口增长、城市化和机动化的加剧，事故发生率正在上升。这一趋势引发了人们对交通监控有效性的担忧。传统的计算机视觉方法在事故检测方面面临着时空理解有限以及跨领域泛化能力差的问题。最近，基于变换器架构的进展在建模全局空间-时间依赖性和并行计算方面表现出色。然而，在自动交通事故检测中应用这些模型受到小规模、非多样化的数据集的限制，这阻碍了开发稳健且通用系统的进程。 为了解决这一问题，我们整理了一个全面而平衡的数据集，该数据集涵盖了广泛的交通环境、事故类型和情境变化。利用这个精心策划的数据集，我们提出了一种基于变换器架构并使用预提取空间视频特征的事故检测模型。这种架构通过卷积层从帧内各种模式中抽取局部相关性，并利用变换器捕捉所检索到的特性之间的序列-时间依赖关系。 此外，大多数现有研究忽略了运动线索（对理解动态场景尤其是事故发生时至关重要的信息）的整合。这些方法通常依赖于静态特征或粗略的时间信息。在本研究中，我们评估了多种包含运动线索的方法以确定最有效的策略。在测试的输入方式中，并联RGB特性与光流获得最高的准确性，达到了88.3%。 我们的结果还与其他视觉语言模型（如GPT、Gemini和LLaVA-NeXT-Video）进行了对比，以评估所提出方法的有效性。
URL
https://arxiv.org/abs/2512.11350
PDF
-
Physics-Informed Video Flare Synthesis and Removal Leveraging Motion Independence between Flare and Scene
2025-12-12 07:01:44Junqiao Wang, Yuanfei Huang, Hua HuangAbstract
Lens flare is a degradation phenomenon caused by strong light sources. Existing researches on flare removal have mainly focused on images, while the spatiotemporal characteristics of video flare remain largely unexplored. Video flare synthesis and removal pose significantly greater challenges than in image, owing to the complex and mutually independent motion of flare, light sources, and scene content. This motion independence further affects restoration performance, often resulting in flicker and artifacts. To address this issue, we propose a physics-informed dynamic flare synthesis pipeline, which simulates light source motion using optical flow and models the temporal behaviors of both scattering and reflective flares. Meanwhile, we design a video flare removal network that employs an attention module to spatially suppress flare regions and incorporates a Mamba-based temporal modeling component to capture long range spatio-temporal dependencies. This motion-independent spatiotemporal representation effectively eliminates the need for multi-frame alignment, alleviating temporal aliasing between flares and scene content and thereby improving video flare removal performance. Building upon this, we construct the first video flare dataset to comprehensively evaluate our method, which includes a large set of synthetic paired videos and additional real-world videos collected from the Internet to assess generalization capability. Extensive experiments demonstrate that our method consistently outperforms existing video-based restoration and image-based flare removal methods on both real and synthetic videos, effectively removing dynamic flares while preserving light source integrity and maintaining spatiotemporal consistency of scene.
Abstract (translated)
镜头光晕是一种由强光源引起的退化现象。现有的关于去除光晕的研究主要集中在图像上，而视频中的光晕的空间和时间特性则鲜有研究。与图像相比，视频光晕的合成和去除面临更大的挑战，原因在于光晕、光源以及场景内容之间的复杂且相互独立的运动模式进一步影响了修复效果，通常会导致闪烁和伪影。为了解决这个问题，我们提出了一种基于物理信息的动态光晕合成管道，该管道利用光学流模拟光源运动，并对散射和反射性光晕的时间行为进行建模。与此同时，我们设计了一个视频光晕去除网络，其中运用了注意力模块来抑制空间上的光晕区域，并整合了一个基于Mamba的时间模型组件以捕捉长程时空依赖关系。这种独立于运动的时空表示有效地消除了多帧对齐的需求，减轻了光晕与场景内容之间的时间混叠问题，从而提升了视频光晕去除的效果。 在此基础上，我们构建了首个用于全面评估我们方法的视频光晕数据集，其中包括大量的合成配对视频以及从互联网收集的真实世界视频以测试泛化能力。广泛的实验表明，我们的方法在真实和合成视频上始终优于现有的基于视频恢复和图像去光晕的方法，能够有效地去除动态光晕的同时保持光源的完整性，并维持场景时空一致性。
URL
https://arxiv.org/abs/2512.11327
PDF
-
Error-Propagation-Free Learned Video Compression With Dual-Domain Progressive Temporal Alignment
2025-12-11 09:14:51Han Li, Shaohui Li, Wenrui Dai, Chenglin Li, Xinlong Pan, Haipeng Wang, Junni Zou, Hongkai XiongAbstract
Existing frameworks for learned video compression suffer from a dilemma between inaccurate temporal alignment and error propagation for motion estimation and compensation (ME/MC). The separate-transform framework employs distinct transforms for intra-frame and inter-frame compression to yield impressive rate-distortion (R-D) performance but causes evident error propagation, while the unified-transform framework eliminates error propagation via shared transforms but is inferior in ME/MC in shared latent domains. To address this limitation, in this paper, we propose a novel unifiedtransform framework with dual-domain progressive temporal alignment and quality-conditioned mixture-of-expert (QCMoE) to enable quality-consistent and error-propagation-free streaming for learned video compression. Specifically, we propose dualdomain progressive temporal alignment for ME/MC that leverages coarse pixel-domain alignment and refined latent-domain alignment to significantly enhance temporal context modeling in a coarse-to-fine fashion. The coarse pixel-domain alignment efficiently handles simple motion patterns with optical flow estimated from a single reference frame, while the refined latent-domain alignment develops a Flow-Guided Deformable Transformer (FGDT) over latents from multiple reference frames to achieve long-term motion refinement (LTMR) for complex motion patterns. Furthermore, we design a QCMoE module for continuous bit-rate adaptation that dynamically assigns different experts to adjust quantization steps per pixel based on target quality and content rather than relies on a single quantization step. QCMoE allows continuous and consistent rate control with appealing R-D performance. Experimental results show that the proposed method achieves competitive R-D performance compared with the state-of-the-arts, while successfully eliminating error propagation.
Abstract (translated)
现有的基于学习的视频压缩框架在时间对齐的准确性与运动估计和补偿（ME/MC）中的误差传播之间存在着一个难题。分离变换框架通过为帧内和帧间压缩使用不同的变换来获得出色的率失真（R-D）性能，但会导致明显的误差传播问题；统一变换框架则通过共享变换消除了误差传播的影响，但在共享潜在领域中对ME/MC的处理效果较差。 为了克服这一限制，在本文中我们提出了一种新的统一变换框架，该框架结合了双域渐进式时间对齐和基于质量条件的专家混合（QCMoE），从而能够在无误差传播的情况下实现学习视频压缩的质量一致性流媒体传输。具体来说，我们提出了针对ME/MC的双域渐进式时间对齐方法，这种方法利用粗略像素域对齐与精细潜在域对齐相结合的方式，以一种从粗到细的方法显著增强了时态上下文建模。 粗略像素域对齐能够高效地处理基于单一参考帧估算出光流的简单运动模式。而精细化潜在域对齐则通过在多参考帧的潜在值上发展流动引导可变形变换器（FGDT），实现了复杂运动模式下的长期运动细化（LTMR）。 此外，我们设计了一个QCMoE模块来实现连续比特率自适应调整，该模块根据目标质量和内容动态地分配不同的专家以调节每个像素的量化步骤，而不是依赖单一的量化步长。QCMoE模块允许进行连续且一致的速率控制，并取得令人满意的R-D性能。 实验结果表明，所提出的方法在与现有最佳方法相比时能够达到竞争性的率失真性能，同时成功地消除了误差传播问题。
URL
https://arxiv.org/abs/2512.10450
PDF
-
VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densification
2025-12-10 13:40:24Wanyue Zhang, Lin Geng Foo, Thabo Beeler, Rishabh Dabral, Christian TheobaltAbstract
Synthesizing realistic human-object interactions (HOI) in video is challenging due to the complex, instance-specific interaction dynamics of both humans and objects. Incorporating controllability in video generation further adds to the complexity. Existing controllable video generation approaches face a trade-off: sparse controls like keypoint trajectories are easy to specify but lack instance-awareness, while dense signals such as optical flow, depths or 3D meshes are informative but costly to obtain. We propose VHOI, a two-stage framework that first densifies sparse trajectories into HOI mask sequences, and then fine-tunes a video diffusion model conditioned on these dense masks. We introduce a novel HOI-aware motion representation that uses color encodings to distinguish not only human and object motion, but also body-part-specific dynamics. This design incorporates a human prior into the conditioning signal and strengthens the model's ability to understand and generate realistic HOI dynamics. Experiments demonstrate state-of-the-art results in controllable HOI video generation. VHOI is not limited to interaction-only scenarios and can also generate full human navigation leading up to object interactions in an end-to-end manner. Project page: this https URL.
Abstract (translated)
在视频中合成逼真的人与物体交互（HOI）是具有挑战性的，因为人类和物体之间复杂的、特定实例的交互动态。将可控性加入到视频生成过程中进一步增加了复杂度。现有的可控制视频生成方法面临一个权衡：稀疏控制信号如关键点轨迹易于指定但缺乏具体实例的认识；而密集信号（例如光流、深度或3D网格）虽然信息量大，却获取成本高。 为此，我们提出了VHOI，这是一种两阶段框架，首先将稀疏的轨迹转化为HOI掩码序列，然后根据这些密集掩码对视频扩散模型进行微调。我们引入了一种新颖的人体交互感知运动表示方法，这种方法使用颜色编码来区分不仅人类和物体的移动性，还包含身体部位特定的动力学特征。这种设计把人的先验知识融入到了条件信号中，并增强了模型理解与生成逼真的HOI动态的能力。 实验表明，在可控人机互动视频生成方面VHOI达到了最先进的成果。值得注意的是，VHOI不仅仅局限于交互场景，还能以端到端的方式生成完整的从人体导航直至物体交互的全过程。项目的详细信息可以在这里找到：[项目页面](https://this/url)（请将URL替换为实际链接）。
URL
https://arxiv.org/abs/2512.09646
PDF
-
Label-free Motion-Conditioned Diffusion Model for Cardiac Ultrasound Synthesis
2025-12-10 08:32:34Zhe Li, Hadrien Reynaud, Johanna P M\"uller, Bernhard KainzAbstract
Ultrasound echocardiography is essential for the non-invasive, real-time assessment of cardiac function, but the scarcity of labelled data, driven by privacy restrictions and the complexity of expert annotation, remains a major obstacle for deep learning methods. We propose the Motion Conditioned Diffusion Model (MCDM), a label-free latent diffusion framework that synthesises realistic echocardiography videos conditioned on self-supervised motion features. To extract these features, we design the Motion and Appearance Feature Extractor (MAFE), which disentangles motion and appearance representations from videos. Feature learning is further enhanced by two auxiliary objectives: a re-identification loss guided by pseudo appearance features and an optical flow loss guided by pseudo flow fields. Evaluated on the EchoNet-Dynamic dataset, MCDM achieves competitive video generation performance, producing temporally coherent and clinically realistic sequences without reliance on manual labels. These results demonstrate the potential of self-supervised conditioning for scalable echocardiography synthesis. Our code is available at this https URL.
Abstract (translated)
超声心动图是评估心脏功能的非侵入性和实时方法，但受隐私限制和专家标注复杂性影响的数据标记不足，仍然是深度学习方法面临的主要障碍。我们提出了一种无标签的潜在扩散框架——运动条件扩散模型（MCDM），该模型可以在自监督运动特征条件下合成逼真的超声心动图视频。为了提取这些特征，我们设计了运动与外观特征提取器（MAFE），它可以将运动和外观表示从视频中分离出来。通过两个辅助目标进一步增强了特性学习：一个由伪外观特征引导的重新识别损失，以及一个由伪流场引导的光流损失。 在EchoNet-Dynamic数据集上进行评估时，MCDM达到了竞争性的视频生成性能，能够产生时间一致性高且临床真实的序列，并且无需依赖手动标签。这些结果展示了自监督条件对于可扩展超声心动图合成的潜力。我们的代码可在[此处](https://this https URL)获取。
URL
https://arxiv.org/abs/2512.09418
PDF
-
GeoDiffMM: Geometry-Guided Conditional Diffusion for Motion Magnification
2025-12-09 07:40:51Xuedeng Liu, Jiabao Guo, Zheng Zhang, Fei Wang, Zhi Liu, Dan GuoAbstract
Video Motion Magnification (VMM) amplifies subtle macroscopic motions to a perceptible level. Recently, existing mainstream Eulerian approaches address amplification-induced noise via decoupling representation learning such as texture, shape and frequancey schemes, but they still struggle to separate photon noise from true micro-motion when motion displacements are very small. We propose GeoDiffMM, a novel diffusion-based Lagrangian VMM framework conditioned on optical flow as a geometric cue, enabling structurally consistent motion magnification. Specifically, we design a Noise-free Optical Flow Augmentation strategy that synthesizes diverse nonrigid motion fields without photon noise as supervision, helping the model learn more accurate geometry-aware optial flow and generalize better. Next, we develop a Diffusion Motion Magnifier that conditions the denoising process on (i) optical flow as a geometry prior and (ii) a learnable magnification factor controlling magnitude, thereby selectively amplifying motion components consistent with scene semantics and structure while suppressing content-irrelevant perturbations. Finally, we perform Flow-based Video Synthesis to map the amplified motion back to the image domain with high fidelity. Extensive experiments on real and synthetic datasets show that GeoDiffMM outperforms state-of-the-art methods and significantly improves motion magnification.
Abstract (translated)
视频运动放大（VMM）技术能够将微小的宏观运动放大到可以被感知的程度。近年来，现有的主流欧拉方法通过分离纹理、形状和频率等表示学习来解决由放大引起的噪声问题，但当运动位移非常小时，它们仍然难以区分光子噪声与真实微观运动。 我们提出了一种新的基于扩散的方法——GeoDiffMM（几何扩散微动放大框架），该框架以光学流作为几何线索进行条件化处理，从而实现结构一致的运动放大。具体而言，我们设计了无光子噪声的光学流增强策略，通过合成各种非刚性运动场来为模型提供监督信息，这些运动场上没有光子噪声，这有助于模型学习更准确的几何感知光学流，并提高其泛化能力。 接下来，我们开发了一个扩散运动放大器，该放大器在去噪过程中基于（i）作为几何先验条件的光学流和（ii）一个可学习的放大因子控制幅度。这个放大部分能够选择性地放大与场景语义和结构一致的运动成分，并抑制无关的内容扰动。 最后，我们通过基于流的视频合成将放大后的运动映射回图像域，以实现高保真度。 在真实数据集和合成数据集上的广泛实验表明，GeoDiffMM优于现有最先进的方法，并显著提高了运动放大的性能。
URL
https://arxiv.org/abs/2512.08325
PDF
-
UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation
2025-12-08 18:59:01Jiehui Huang, Yuechen Zhang, Xu He, Yuan Gao, Zhi Cen, Bin Xia, Yan Zhou, Xin Tao, Pengfei Wan, Jiaya JiaAbstract
Recent video generation models demonstrate impressive synthesis capabilities but remain limited by single-modality conditioning, constraining their holistic world understanding. This stems from insufficient cross-modal interaction and limited modal diversity for comprehensive world knowledge representation. To address these limitations, we introduce UnityVideo, a unified framework for world-aware video generation that jointly learns across multiple modalities (segmentation masks, human skeletons, DensePose, optical flow, and depth maps) and training paradigms. Our approach features two core components: (1) dynamic noising to unify heterogeneous training paradigms, and (2) a modality switcher with an in-context learner that enables unified processing via modular parameters and contextual learning. We contribute a large-scale unified dataset with 1.3M samples. Through joint optimization, UnityVideo accelerates convergence and significantly enhances zero-shot generalization to unseen data. We demonstrate that UnityVideo achieves superior video quality, consistency, and improved alignment with physical world constraints. Code and data can be found at: this https URL
Abstract (translated)
最近的视频生成模型展示了令人印象深刻的合成能力，但仍然受限于单一模态的条件设定，这限制了它们对世界整体理解的能力。这种局限性源于跨模态互动不足和用于综合世界知识表示的模态多样性有限。为了解决这些问题，我们引入了一个名为UnityVideo的统一框架，该框架旨在通过多模态（包括分割掩码、人体骨架、密集姿态图、光流以及深度图）进行学习，并采用多种训练范式。我们的方法包含两个核心组成部分：(1) 动态噪声处理，用于统一分异的训练范式；(2) 带有上下文学习器的模态切换器，通过模块化参数和上下文学习实现统一处理。 我们贡献了一个大规模的多模态数据集，其中包含超过130万个样本。通过联合优化，UnityVideo加速了收敛速度，并显著提升了对未见过的数据的零样本泛化能力。实验表明，UnityVideo实现了视频质量、一致性和与物理世界约束更好的对齐方面的改进。 代码和数据可在以下链接获取：[此URL](this https URL)
URL
https://arxiv.org/abs/2512.07831
PDF
-
UltrasODM: A Dual Stream Optical Flow Mamba Network for 3D Freehand Ultrasound Reconstruction
2025-12-08 17:39:34Mayank Anand, Ujair Alam, Surya Prakash, Priya Shukla, Gora Chand Nandi, Domenec PuigAbstract
Clinical ultrasound acquisition is highly operator-dependent, where rapid probe motion and brightness fluctuations often lead to reconstruction errors that reduce trust and clinical utility. We present UltrasODM, a dual-stream framework that assists sonographers during acquisition through calibrated per-frame uncertainty, saliency-based diagnostics, and actionable prompts. UltrasODM integrates (i) a contrastive ranking module that groups frames by motion similarity, (ii) an optical-flow stream fused with Dual-Mamba temporal modules for robust 6-DoF pose estimation, and (iii) a Human-in-the-Loop (HITL) layer combining Bayesian uncertainty, clinician-calibrated thresholds, and saliency maps highlighting regions of low confidence. When uncertainty exceeds the threshold, the system issues unobtrusive alerts suggesting corrective actions such as re-scanning highlighted regions or slowing the sweep. Evaluated on a clinical freehand ultrasound dataset, UltrasODM reduces drift by 15.2%, distance error by 12.1%, and Hausdorff distance by 10.1% relative to UltrasOM, while producing per-frame uncertainty and saliency outputs. By emphasizing transparency and clinician feedback, UltrasODM improves reconstruction reliability and supports safer, more trustworthy clinical workflows. Our code is publicly available at this https URL.
Abstract (translated)
临床超声检查高度依赖于操作员，快速探头移动和亮度波动常常导致重建错误，从而降低信任度和临床实用性。我们提出了一种名为UltrasODM的双流框架，该框架通过每帧校准不确定性和基于显著性图的诊断以及可执行提示来帮助超声技术人员进行检查。UltrasODM集成了(i)一个对比排名模块，按照运动相似性分组图像帧；(ii)一个融合了Dual-Mamba时间模块的光流流，用于稳健的6自由度姿态估计；(iii)一个人机交互（Human-in-the-Loop）层，结合了贝叶斯不确定性、临床医生校准的阈值和突出显示低置信区域的显著性图。当不确定性超过阈值时，系统会发出不显眼的警报，建议采取纠正措施，如重新扫描被标记的区域或减缓扫查速度。 在一项使用临床自由手超声数据集进行评估的研究中，UltrasODM比UltrasOM减少了15.2%的姿态漂移、12.1%的距离误差以及10.1%的豪斯多夫距离，并且还生成了每帧不确定性及显著性输出。通过强调透明度和临床医生反馈，UltrasODM提高了重建可靠性并支持更安全、更可信的临床工作流程。 我们的代码在以下链接公开提供：[此处应填写具体的URL]。
URL
https://arxiv.org/abs/2512.07756
PDF
-
Overcoming Small Data Limitations in Video-Based Infant Respiration Estimation
2025-12-07 15:25:17Liyang Song, Hardik Bishnoi, Sai Kumar Reddy Manne, Sarah Ostadabbas, Briana J. Taylor, Michael WanAbstract
The development of contactless respiration monitoring for infants could enable advances in the early detection and treatment of breathing irregularities, which are associated with neurodevelopmental impairments and conditions like sudden infant death syndrome (SIDS). But while respiration estimation for adults is supported by a robust ecosystem of computer vision algorithms and video datasets, only one small public video dataset with annotated respiration data for infant subjects exists, and there are no reproducible algorithms which are effective for infants. We introduce the annotated infant respiration dataset of 400 videos (AIR-400), contributing 275 new, carefully annotated videos from 10 recruited subjects to the public corpus. We develop the first reproducible pipelines for infant respiration estimation, based on infant-specific region-of-interest detection and spatiotemporal neural processing enhanced by optical flow inputs. We establish, through comprehensive experiments, the first reproducible benchmarks for the state-of-the-art in vision-based infant respiration estimation. We make our dataset, code repository, and trained models available for public use.
Abstract (translated)
针对婴儿的非接触式呼吸监测的发展可能有助于早期发现和治疗与神经发育障碍及婴儿猝死综合症(SIDS)相关的呼吸不规律。然而，尽管成人呼吸估计有强大的计算机视觉算法和支持视频数据集生态系统，但仅存在一个包含少量带有标注呼吸数据的小型公共视频数据集用于婴儿，并且没有适用于婴儿的有效可重复的算法。我们介绍了注释婴儿呼吸数据集（AIR-400），其中包括从10名招募来的受试者中收集到的275个新的、精心标记的视频，为公众贡献了400个视频的数据集合。我们开发了首个基于特定于婴儿的兴趣区域检测和通过光学流输入增强的空间时间神经处理的第一批可重复管道用于婴儿呼吸估计。通过全面实验，我们建立了第一个基于视觉技术的婴儿呼吸估计的先进状态下的可重复基准。我们将数据集、代码库以及训练模型向公众开放使用。
URL
https://arxiv.org/abs/2512.06888
PDF
