Optical_Flow
-
STG-Avatar: Animatable Human Avatars via Spacetime Gaussian
2025-10-25 03:23:38Guangan Jiang, Tianzi Zhang, Dong Li, Zhenjun Zhao, Haoang Li, Mingrui Li, Hongyu WangAbstract
Realistic animatable human avatars from monocular videos are crucial for advancing human-robot interaction and enhancing immersive virtual experiences. While recent research on 3DGS-based human avatars has made progress, it still struggles with accurately representing detailed features of non-rigid objects (e.g., clothing deformations) and dynamic regions (e.g., rapidly moving limbs). To address these challenges, we present STG-Avatar, a 3DGS-based framework for high-fidelity animatable human avatar reconstruction. Specifically, our framework introduces a rigid-nonrigid coupled deformation framework that synergistically integrates Spacetime Gaussians (STG) with linear blend skinning (LBS). In this hybrid design, LBS enables real-time skeletal control by driving global pose transformations, while STG complements it through spacetime adaptive optimization of 3D Gaussians. Furthermore, we employ optical flow to identify high-dynamic regions and guide the adaptive densification of 3D Gaussians in these regions. Experimental results demonstrate that our method consistently outperforms state-of-the-art baselines in both reconstruction quality and operational efficiency, achieving superior quantitative metrics while retaining real-time rendering capabilities. Our code is available at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2510.22140
PDF
-
RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scaling
2025-10-23 04:45:09Bingjie Gao, Qianli Ma, Xiaoxue Wu, Shuai Yang, Guanzhou Lan, Haonan Zhao, Jiaxuan Chen, Qingyang Liu, Yu Qiao, Xinyuan Chen, Yaohui Wang, Li NiuAbstract
Prompt design plays a crucial role in text-to-video (T2V) generation, yet user-provided prompts are often short, unstructured, and misaligned with training data, limiting the generative potential of diffusion-based T2V models. We present \textbf{RAPO++}, a cross-stage prompt optimization framework that unifies training-data--aligned refinement, test-time iterative scaling, and large language model (LLM) fine-tuning to substantially improve T2V generation without modifying the underlying generative backbone. In \textbf{Stage 1}, Retrieval-Augmented Prompt Optimization (RAPO) enriches user prompts with semantically relevant modifiers retrieved from a relation graph and refactors them to match training distributions, enhancing compositionality and multi-object fidelity. \textbf{Stage 2} introduces Sample-Specific Prompt Optimization (SSPO), a closed-loop mechanism that iteratively refines prompts using multi-source feedback -- including semantic alignment, spatial fidelity, temporal coherence, and task-specific signals such as optical flow -- yielding progressively improved video generation quality. \textbf{Stage 3} leverages optimized prompt pairs from SSPO to fine-tune the rewriter LLM, internalizing task-specific optimization patterns and enabling efficient, high-quality prompt generation even before inference. Extensive experiments across five state-of-the-art T2V models and five benchmarks demonstrate that RAPO++ achieves significant gains in semantic alignment, compositional reasoning, temporal stability, and physical plausibility, outperforming existing methods by large margins. Our results highlight RAPO++ as a model-agnostic, cost-efficient, and scalable solution that sets a new standard for prompt optimization in T2V generation. The code is available at this https URL.
Abstract (translated)
提示设计在文本到视频（T2V）生成中扮演着关键角色，但用户提供的提示往往简短、结构松散，并且与训练数据不匹配，这限制了基于扩散模型的T2V生成能力。我们提出了\textbf{RAPO++}，这是一个跨阶段提示优化框架，它统一了与训练数据对齐的细化过程、测试时间迭代扩展以及大型语言模型（LLM）微调，以在无需修改底层生成架构的情况下显著提升T2V生成质量。 **第一阶段**：检索增强提示优化（RAPO），通过从关系图中检索语义相关的修饰符来丰富用户提示，并将这些提示重构为与训练分布匹配的形式。这一过程增强了提示的组合性和多对象保真度。 **第二阶段**引入了特定样本提示优化（SSPO），这是一种闭环机制，它迭代地使用包括语义对齐、空间保真度、时间连贯性以及如光流这样的任务特有信号在内的多源反馈来改进提示。这一过程逐步提高了生成视频的质量。 **第三阶段**利用来自SSPO的优化提示对重写LLM进行微调，将特定任务的优化模式内化为模型的一部分，并且即使在推理之前也能高效地生成高质量的提示。 通过五个最先进的T2V模型和五个基准测试所进行的广泛实验表明，RAPO++在语义一致性、组合性推理、时间稳定性以及物理合理性方面取得了显著提升，大幅超越了现有方法。我们的结果强调了RAPO++作为一个与模型无关、成本效益高且可扩展的解决方案的价值，在T2V生成提示优化领域设立了新的标准。 相关代码可在以下链接获取：[https URL]（请将方括号中的内容替换为实际的URL）。
URL
https://arxiv.org/abs/2510.20206
PDF
-
MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models
2025-10-21 19:05:23Aritra Bhowmik, Denis Korzhenkov, Cees G. M. Snoek, Amirhossein Habibian, Mohsen GhafoorianAbstract
Text-to-video diffusion models have enabled high-quality video synthesis, yet often fail to generate temporally coherent and physically plausible motion. A key reason is the models' insufficient understanding of complex motions that natural videos often entail. Recent works tackle this problem by aligning diffusion model features with those from pretrained video encoders. However, these encoders mix video appearance and dynamics into entangled features, limiting the benefit of such alignment. In this paper, we propose a motion-centric alignment framework that learns a disentangled motion subspace from a pretrained video encoder. This subspace is optimized to predict ground-truth optical flow, ensuring it captures true motion dynamics. We then align the latent features of a text-to-video diffusion model to this new subspace, enabling the generative model to internalize motion knowledge and generate more plausible videos. Our method improves the physical commonsense in a state-of-the-art video diffusion model, while preserving adherence to textual prompts, as evidenced by empirical evaluations on VideoPhy, VideoPhy2, VBench, and VBench-2.0, along with a user study.
Abstract (translated)
文本到视频的扩散模型已经实现了高质量的视频合成，然而它们往往难以生成时间上连贯且物理上合理的运动。主要原因在于这些模型缺乏对自然视频中常见复杂运动的理解。最近的一些研究通过将扩散模型特征与预训练的视频编码器特征对齐来解决这一问题。然而，这种方法会混合视频外观和动态信息，导致提取出的特征纠缠在一起，从而限制了这种对齐方法的效果。 在本文中，我们提出了一种以运动为中心的对齐框架，该框架从预训练的视频编码器中学得一个解耦的运动子空间。这个子空间经过优化可以预测真实的光流，确保它捕捉到了真正的运动动态。然后我们将文本到视频扩散模型中的潜在特征与新的子空间进行对齐，使生成模型能够内化有关运动的知识，并产生更加合理的视频。 我们的方法在不牺牲对文本提示的遵从性的前提下，改进了最先进的视频扩散模型中物理常识的表现力。这一点通过在VideoPhy、VideoPhy2、VBench和VBench-2.0上的实证评估以及用户研究得到了验证。
URL
https://arxiv.org/abs/2510.19022
PDF
-
VelocityNet: Real-Time Crowd Anomaly Detection via Person-Specific Velocity Analysis
2025-10-21 00:26:54Fatima AlGhamdi, Omar Alharbi, Abdullah Aldwyish, Raied Aljadaany, Muhammad Kamran J Khan, Huda AlamriAbstract
Detecting anomalies in crowded scenes is challenging due to severe inter-person occlusions and highly dynamic, context-dependent motion patterns. Existing approaches often struggle to adapt to varying crowd densities and lack interpretable anomaly indicators. To address these limitations, we introduce VelocityNet, a dual-pipeline framework that combines head detection and dense optical flow to extract person-specific velocities. Hierarchical clustering categorizes these velocities into semantic motion classes (halt, slow, normal, and fast), and a percentile-based anomaly scoring system measures deviations from learned normal patterns. Experiments demonstrate the effectiveness of our framework in real-time detection of diverse anomalous motion patterns within densely crowded environments.
Abstract (translated)
在拥挤场景中检测异常行为具有挑战性，因为存在严重的人员遮挡以及高度动态且依赖于上下文的运动模式。现有的方法往往难以适应不同的人群密度，并缺乏可解释的异常指标。为了解决这些限制，我们引入了VelocityNet，这是一种双管道框架，结合头部检测和密集光流来提取个人特定的速度信息。通过层次聚类将这些速度分类为语义运动类别（停止、慢速、正常和快速），并使用基于百分位数的方法来衡量与学习到的正常模式之间的偏差。实验表明，我们的框架能够在高密度拥挤环境中实时检测出各种异常运动模式，证明了其有效性。
URL
https://arxiv.org/abs/2510.18187
PDF
-
DGME-T: Directional Grid Motion Encoding for Transformer-Based Historical Camera Movement Classification
2025-10-17 15:14:11Tingyu Lin, Armin Dadras, Florian Kleber, Robert SablatnigAbstract
Camera movement classification (CMC) models trained on contemporary, high-quality footage often degrade when applied to archival film, where noise, missing frames, and low contrast obscure motion cues. We bridge this gap by assembling a unified benchmark that consolidates two modern corpora into four canonical classes and restructures the HISTORIAN collection into five balanced categories. Building on this benchmark, we introduce DGME-T, a lightweight extension to the Video Swin Transformer that injects directional grid motion encoding, derived from optical flow, via a learnable and normalised late-fusion layer. DGME-T raises the backbone's top-1 accuracy from 81.78% to 86.14% and its macro F1 from 82.08% to 87.81% on modern clips, while still improving the demanding World-War-II footage from 83.43% to 84.62% accuracy and from 81.72% to 82.63% macro F1. A cross-domain study further shows that an intermediate fine-tuning stage on modern data increases historical performance by more than five percentage points. These results demonstrate that structured motion priors and transformer representations are complementary and that even a small, carefully calibrated motion head can substantially enhance robustness in degraded film analysis. Related resources are available at this https URL.
Abstract (translated)
基于当代高质量视频训练的相机运动分类（CMC）模型在应用于存档影片时通常会出现性能下降，因为噪声、缺失帧和低对比度会遮蔽运动线索。为解决这一问题，我们建立了一个统一基准测试平台，该平台整合了两个现代数据集，并将其重组为四种典型类别，同时将HISTORIAN集合重新组织成五个平衡的类别。基于此基准，我们推出了DGME-T，这是Video Swin Transformer的一个轻量级扩展版本，通过一个可学习和规范化的晚期融合层注入来自光流的方向网格运动编码（DGME）。DGME-T使骨干网络在现代视频片段上的顶级准确率从81.78%提升至86.14%，宏平均F1分数从82.08%提高到87.81%。同时，它还能将对二战时期影片的准确性从83.43%提升至84.62%，宏平均F1分数则从81.72%增加到82.63%。跨域研究进一步显示，在现代数据上进行中间微调阶段可以使历史性能提高超过五个百分点。这些结果表明，结构化的运动先验和变换表示是互补的，并且即使是一个小而精心校准的运动头部也能显著提升在降级影片分析中的鲁棒性。 相关资源可在以下链接获取：[此链接]（请将"[此链接]"替换为实际提供的URL）。
URL
https://arxiv.org/abs/2510.15725
PDF
-
Iterative Motion Compensation for Canonical 3D Reconstruction from UAV Plant Images Captured in Windy Conditions
2025-10-17 10:04:08Andre Rochow, Jonas Marcic, Svetlana Seliunina, Sven BehnkeAbstract
3D phenotyping of plants plays a crucial role for understanding plant growth, yield prediction, and disease control. We present a pipeline capable of generating high-quality 3D reconstructions of individual agricultural plants. To acquire data, a small commercially available UAV captures images of a selected plant. Apart from placing ArUco markers, the entire image acquisition process is fully autonomous, controlled by a self-developed Android application running on the drone's controller. The reconstruction task is particularly challenging due to environmental wind and downwash of the UAV. Our proposed pipeline supports the integration of arbitrary state-of-the-art 3D reconstruction methods. To mitigate errors caused by leaf motion during image capture, we use an iterative method that gradually adjusts the input images through deformation. Motion is estimated using optical flow between the original input images and intermediate 3D reconstructions rendered from the corresponding viewpoints. This alignment gradually reduces scene motion, resulting in a canonical representation. After a few iterations, our pipeline improves the reconstruction of state-of-the-art methods and enables the extraction of high-resolution 3D meshes. We will publicly release the source code of our reconstruction pipeline. Additionally, we provide a dataset consisting of multiple plants from various crops, captured across different points in time.
Abstract (translated)
植物的三维表型分析在理解植物生长、预测产量和控制疾病方面发挥着关键作用。我们提出了一条能够生成单个农业植物高质量三维重建的流水线。为了获取数据，一个小型商用无人机拍摄选定植物的图像。除了放置ArUco标记外，整个图像采集过程完全自主化，由运行在无人机控制器上的自开发Android应用程序控制。由于环境风和无人机下洗的影响，重建任务尤其具有挑战性。我们提出的流水线支持集成任意先进的三维重建方法。为了减轻叶片移动期间拍摄图像时产生的错误，我们采用了一种迭代方法，逐步通过变形调整输入图像。运动是利用原始输入图像与从相应视点渲染的中间三维重建之间的光流进行估计的。这种对齐逐渐减少场景中的运动，从而形成规范表示。经过几次迭代后，我们的流水线提升了现有方法的重建效果，并能够提取高分辨率的3D网格。我们将公开发布我们重建流水线的源代码。此外，我们还提供了一个包含来自不同作物的多株植物的数据集，这些植物在不同的时间点被捕获。
URL
https://arxiv.org/abs/2510.15491
PDF
-
A Novel Combined Optical Flow Approach for Comprehensive Micro-Expression Recognition
2025-10-17 09:29:17Vu Tram Anh Khuong, Thi Bich Phuong Man, Luu Tu Nguyen, Thanh Ha Le, Thi Duyen NgoAbstract
Facial micro-expressions are brief, involuntary facial movements that reveal hidden emotions. Most Micro-Expression Recognition (MER) methods that rely on optical flow typically focus on the onset-to-apex phase, neglecting the apex-to-offset phase, which holds key temporal dynamics. This study introduces a Combined Optical Flow (COF), integrating both phases to enhance feature representation. COF provides a more comprehensive motion analysis, improving MER performance. Experimental results on CASMEII and SAMM datasets show that COF outperforms single optical flow-based methods, demonstrating its effectiveness in capturing micro-expression dynamics.
Abstract (translated)
面部微表情是短暂的、非自愿的面部动作，能揭示隐藏的情绪。大多数依赖于光学流的微表情识别（MER）方法通常只关注从开始到峰值的阶段，而忽视了从峰值到结束的关键时间动态变化。这项研究引入了一种结合型光学流（COF），整合这两个阶段来增强特征表示。COF提供了更全面的运动分析，从而提高了微表情识别性能。在CASMEII和SAMM数据集上的实验结果显示，COF优于单一基于光学流的方法，证明了其捕捉微表情动态的有效性。
URL
https://arxiv.org/abs/2510.15471
PDF
-
MAVR-Net: Robust Multi-View Learning for MAV Action Recognition with Cross-View Attention
2025-10-17 09:04:51Nengbo Zhang, Hann Woei HoAbstract
Recognizing the motion of Micro Aerial Vehicles (MAVs) is crucial for enabling cooperative perception and control in autonomous aerial swarms. Yet, vision-based recognition models relying only on RGB data often fail to capture the complex spatial temporal characteristics of MAV motion, which limits their ability to distinguish different actions. To overcome this problem, this paper presents MAVR-Net, a multi-view learning-based MAV action recognition framework. Unlike traditional single-view methods, the proposed approach combines three complementary types of data, including raw RGB frames, optical flow, and segmentation masks, to improve the robustness and accuracy of MAV motion recognition. Specifically, ResNet-based encoders are used to extract discriminative features from each view, and a multi-scale feature pyramid is adopted to preserve the spatiotemporal details of MAV motion patterns. To enhance the interaction between different views, a cross-view attention module is introduced to model the dependencies among various modalities and feature scales. In addition, a multi-view alignment loss is designed to ensure semantic consistency and strengthen cross-view feature representations. Experimental results on benchmark MAV action datasets show that our method clearly outperforms existing approaches, achieving 97.8\%, 96.5\%, and 92.8\% accuracy on the Short MAV, Medium MAV, and Long MAV datasets, respectively.
Abstract (translated)
微小飞行器（MAVs）的运动识别对于实现自主空中集群中的协作感知和控制至关重要。然而，仅依赖RGB数据的视觉模型常常无法捕捉到MAV运动复杂的时空特性，这限制了其区分不同动作的能力。为了解决这个问题，本文提出了基于多视图学习的MAV动作识别框架——MAVR-Net。不同于传统的单视图方法，该方法结合了三种互补类型的数据：原始RGB帧、光流和分割掩码，以提高MAV运动识别的鲁棒性和准确性。 具体而言，使用ResNet基元编码器从每个视角提取判别性特征，并采用多尺度特征金字塔来保留MAV运动模式的空间时间细节。为了增强不同视图之间的交互，引入了跨视图注意力模块，用于建模各种模式和特征尺度间的依赖关系。此外，设计了一种多视图对齐损失，以确保语义一致性并强化跨视图特征表示。 在基准MAV动作数据集上的实验结果表明，该方法明显优于现有方法，在Short MAV、Medium MAV和Long MAV数据集上分别实现了97.8%、96.5%和92.8%的准确率。
URL
https://arxiv.org/abs/2510.15448
PDF
-
C4D: 4D Made from 3D through Dual Correspondences
2025-10-16 17:59:06Shizun Wang, Zhenxiang Jiang, Xingyi Yang, Xinchao WangAbstract
Recovering 4D from monocular video, which jointly estimates dynamic geometry and camera poses, is an inevitably challenging problem. While recent pointmap-based 3D reconstruction methods (e.g., DUSt3R) have made great progress in reconstructing static scenes, directly applying them to dynamic scenes leads to inaccurate results. This discrepancy arises because moving objects violate multi-view geometric constraints, disrupting the reconstruction. To address this, we introduce C4D, a framework that leverages temporal Correspondences to extend existing 3D reconstruction formulation to 4D. Specifically, apart from predicting pointmaps, C4D captures two types of correspondences: short-term optical flow and long-term point tracking. We train a dynamic-aware point tracker that provides additional mobility information, facilitating the estimation of motion masks to separate moving elements from the static background, thus offering more reliable guidance for dynamic scenes. Furthermore, we introduce a set of dynamic scene optimization objectives to recover per-frame 3D geometry and camera parameters. Simultaneously, the correspondences lift 2D trajectories into smooth 3D trajectories, enabling fully integrated 4D reconstruction. Experiments show that our framework achieves complete 4D recovery and demonstrates strong performance across multiple downstream tasks, including depth estimation, camera pose estimation, and point tracking. Project Page: this https URL
Abstract (translated)
从单目视频中恢复四维（4D）信息，即同时估计动态几何和相机姿态，是一项不可避免的挑战。尽管最近基于点图的三维重建方法（如DUSt3R）在静态场景重建方面取得了巨大进展，但直接将其应用于动态场景会导致不准确的结果。这种差异出现的原因是因为移动物体违反了多视角几何约束，破坏了重建过程。为了解决这个问题，我们引入了一个名为C4D的新框架，该框架利用时间对应关系来扩展现有的三维重建方法以实现四维重建。 具体来说，在预测点图之外，C4D捕捉到了两种类型的对应关系：短期光流和长期点跟踪。我们训练了一种动态感知的点追踪器，提供额外的移动信息，便于估算运动掩模，从而将移动元素与静态背景分离，为动态场景提供了更可靠的指导。此外，我们引入了一系列针对动态场景优化的目标函数，以恢复每帧的三维几何结构和相机参数。 同时，这些对应关系将二维轨迹提升为平滑的三维轨迹，使完全集成的四维重建成为可能。实验表明，我们的框架实现了完整的4D恢复，并在包括深度估计、相机姿态估计和点跟踪在内的多个下游任务中表现出强大的性能。项目主页：[此链接](https://this-url.com/)（请将"this https URL"替换为实际项目页面链接）。
URL
https://arxiv.org/abs/2510.14960
PDF
-
E-MoFlow: Learning Egomotion and Optical Flow from Event Data via Implicit Regularization
2025-10-14 17:33:44Wenpu Li, Bangyan Liao, Yi Zhou, Qi Xu, Pian Wan, Peidong LiuAbstract
The estimation of optical flow and 6-DoF ego-motion, two fundamental tasks in 3D vision, has typically been addressed independently. For neuromorphic vision (e.g., event cameras), however, the lack of robust data association makes solving the two problems separately an ill-posed challenge, especially in the absence of supervision via ground truth. Existing works mitigate this ill-posedness by either enforcing the smoothness of the flow field via an explicit variational regularizer or leveraging explicit structure-and-motion priors in the parametrization to improve event alignment. The former notably introduces bias in results and computational overhead, while the latter, which parametrizes the optical flow in terms of the scene depth and the camera motion, often converges to suboptimal local minima. To address these issues, we propose an unsupervised framework that jointly optimizes egomotion and optical flow via implicit spatial-temporal and geometric regularization. First, by modeling camera's egomotion as a continuous spline and optical flow as an implicit neural representation, our method inherently embeds spatial-temporal coherence through inductive biases. Second, we incorporate structure-and-motion priors through differential geometric constraints, bypassing explicit depth estimation while maintaining rigorous geometric consistency. As a result, our framework (called E-MoFlow) unifies egomotion and optical flow estimation via implicit regularization under a fully unsupervised paradigm. Experiments demonstrate its versatility to general 6-DoF motion scenarios, achieving state-of-the-art performance among unsupervised methods and competitive even with supervised approaches.
Abstract (translated)
在三维视觉中，光流（optical flow）和六自由度自我运动估计（6-DoF ego-motion）是两个基本任务。通常这两个任务会分别独立处理。但对于仿生视觉系统（例如事件相机），由于缺乏稳健的数据关联性，在没有通过地面实况进行监督的情况下单独解决这两个问题变得非常困难。现有的研究通过在光流场中强制平滑度来减少这种病态性，或者通过参数化中的结构和运动先验信息提高事件对齐的准确性。前者通常会在结果中引入偏差并增加计算负担，而后者，即通过对场景深度及相机运动进行参数化，往往只能达到次优局部最小值。 为了解决这些问题，我们提出了一种无监督框架，该框架通过隐式的时空和几何正则化同时优化自我运动和光流。首先，我们的方法通过将摄像机的自我运动建模为连续样条曲线，并且把光流表示为一个隐式神经网络模型，在此基础上，本方法内在地通过归纳偏差嵌入了时空一致性。其次，我们通过微分几何约束引入结构与运动先验知识，这避免了显式的深度估计同时保持严格的几何一致性。 因此，我们的框架（称为E-MoFlow）在完全无监督的范式下，通过隐式正则化统一自我运动和光流的估计。实验表明，该方法能够处理各种6-DoF运动场景，并在无监督的方法中表现出最先进的性能，在与有监督方法的竞争中也具有竞争力。
URL
https://arxiv.org/abs/2510.12753
PDF
-
SPORTS: Simultaneous Panoptic Odometry, Rendering, Tracking and Segmentation for Urban Scenes Understanding
2025-10-14 17:28:19Zhiliu Yang, Jinyu Dai, Jianyuan Zhang, Zhu YangAbstract
The scene perception, understanding, and simulation are fundamental techniques for embodied-AI agents, while existing solutions are still prone to segmentation deficiency, dynamic objects' interference, sensor data sparsity, and view-limitation problems. This paper proposes a novel framework, named SPORTS, for holistic scene understanding via tightly integrating Video Panoptic Segmentation (VPS), Visual Odometry (VO), and Scene Rendering (SR) tasks into an iterative and unified perspective. Firstly, VPS designs an adaptive attention-based geometric fusion mechanism to align cross-frame features via enrolling the pose, depth, and optical flow modality, which automatically adjust feature maps for different decoding stages. And a post-matching strategy is integrated to improve identities tracking. In VO, panoptic segmentation results from VPS are combined with the optical flow map to improve the confidence estimation of dynamic objects, which enhances the accuracy of the camera pose estimation and completeness of the depth map generation via the learning-based paradigm. Furthermore, the point-based rendering of SR is beneficial from VO, transforming sparse point clouds into neural fields to synthesize high-fidelity RGB views and twin panoptic views. Extensive experiments on three public datasets demonstrate that our attention-based feature fusion outperforms most existing state-of-the-art methods on the odometry, tracking, segmentation, and novel view synthesis tasks.
Abstract (translated)
场景感知、理解和模拟是具身人工智能代理的基本技术，而现有解决方案仍然容易出现分割不足、动态物体干扰、传感器数据稀疏和视点限制等问题。本文提出了一种新颖的框架——SPORTS（通过紧密整合视频全景分割(VPS)、视觉测距(VO) 和场景渲染(SR) 任务，在迭代和统一的角度下进行整体场景理解）。首先，VPS 设计了一个基于自适应注意力的几何融合机制，该机制通过引入姿态、深度和光流模态来对齐跨帧特征，并自动调整不同解码阶段的特征图。此外，还集成了一个后匹配策略以改进身份追踪。 在VO任务中，将从VPS得到的全景分割结果与光流图结合使用，提高了动态物体置信度估计，增强了基于学习范式的相机姿态估计精度和深度图生成的完整性。另外，SR中的点云渲染得益于VO模块，通过转换稀疏点云为神经场来合成高保真RGB视图和孪生全景视图。 在三个公开数据集上的广泛实验表明，我们提出的注意力特征融合方法在测距、追踪、分割以及新颖视图合成任务上均优于大多数现有的最先进的方法。
URL
https://arxiv.org/abs/2510.12749
PDF
-
CurriFlow: Curriculum-Guided Depth Fusion with Optical Flow-Based Temporal Alignment for 3D Semantic Scene Completion
2025-10-14 10:25:26Jinzhou Lin, Jie Zhou, Wenhao Xu, Rongtao Xu, Changwei Wang, Shunpeng Chen, Kexue Fu, Yihua Shao, Li Guo, Shibiao XuAbstract
Semantic Scene Completion (SSC) aims to infer complete 3D geometry and semantics from monocular images, serving as a crucial capability for camera-based perception in autonomous driving. However, existing SSC methods relying on temporal stacking or depth projection often lack explicit motion reasoning and struggle with occlusions and noisy depth supervision. We propose CurriFlow, a novel semantic occupancy prediction framework that integrates optical flow-based temporal alignment with curriculum-guided depth fusion. CurriFlow employs a multi-level fusion strategy to align segmentation, visual, and depth features across frames using pre-trained optical flow, thereby improving temporal consistency and dynamic object understanding. To enhance geometric robustness, a curriculum learning mechanism progressively transitions from sparse yet accurate LiDAR depth to dense but noisy stereo depth during training, ensuring stable optimization and seamless adaptation to real-world deployment. Furthermore, semantic priors from the Segment Anything Model (SAM) provide category-agnostic supervision, strengthening voxel-level semantic learning and spatial consistency. Experiments on the SemanticKITTI benchmark demonstrate that CurriFlow achieves state-of-the-art performance with a mean IoU of 16.9, validating the effectiveness of our motion-guided and curriculum-aware design for camera-based 3D semantic scene completion.
Abstract (translated)
语义场景补全（SSC）旨在从单目图像中推断出完整的三维几何和语义信息，这对于基于摄像头的自动驾驶感知能力至关重要。然而，现有的SSC方法依赖于时间堆叠或深度投影时往往缺乏显式的运动推理，并且在处理遮挡以及噪声深度监督方面存在困难。我们提出了一种新颖的语义占用预测框架CurriFlow，该框架结合了基于光学流的时间对齐与引导式课程学习的深度融合。 CurriFlow采用多级融合策略，利用预训练的光学流，在帧间对齐分割、视觉和深度特征，从而提高时间一致性并增强动态对象的理解。为了提升几何鲁棒性，引入了一种课程学习机制，在训练过程中逐步从稀疏但准确的激光雷达深度过渡到密集却带有噪声的立体深度，确保了稳定的优化及在实际部署中的平滑适应。 此外，来自Segment Anything Model（SAM）的语义先验提供了类别无关的监督信息，加强了体素级语义学习和空间一致性。在SemanticKITTI基准测试上的实验表明，CurriFlow达到了最先进的性能表现，平均IoU为16.9%，验证了我们基于运动引导及课程感知的设计对于基于摄像头的三维语义场景补全的有效性。
URL
https://arxiv.org/abs/2510.12362
PDF
-
Injecting Frame-Event Complementary Fusion into Diffusion for Optical Flow in Challenging Scenes
2025-10-12 12:52:31Haonan Wang, Hanyu Zhou, Haoyue Liu, Luxin YanAbstract
Optical flow estimation has achieved promising results in conventional scenes but faces challenges in high-speed and low-light scenes, which suffer from motion blur and insufficient illumination. These conditions lead to weakened texture and amplified noise and deteriorate the appearance saturation and boundary completeness of frame cameras, which are necessary for motion feature matching. In degraded scenes, the frame camera provides dense appearance saturation but sparse boundary completeness due to its long imaging time and low dynamic range. In contrast, the event camera offers sparse appearance saturation, while its short imaging time and high dynamic range gives rise to dense boundary completeness. Traditionally, existing methods utilize feature fusion or domain adaptation to introduce event to improve boundary completeness. However, the appearance features are still deteriorated, which severely affects the mostly adopted discriminative models that learn the mapping from visual features to motion fields and generative models that generate motion fields based on given visual features. So we introduce diffusion models that learn the mapping from noising flow to clear flow, which is not affected by the deteriorated visual features. Therefore, we propose a novel optical flow estimation framework Diff-ABFlow based on diffusion models with frame-event appearance-boundary fusion.
Abstract (translated)
光流估计在传统场景中取得了显著成果，但在高速和低光照条件下则面临挑战。这些条件会导致运动模糊和照明不足，进而削弱纹理、放大噪声，并损害帧相机的外观饱和度和边界完整性，而这些都是进行运动特征匹配所必需的。在这种退化的环境中，由于较长的成像时间和较低的动态范围，帧相机虽然提供密集的外观饱和度，但其边界完整性稀疏。相比之下，事件相机提供了稀疏的外观饱和度，但由于其较短的成像时间和较高的动态范围，它可以产生密集的边界完整性。 传统方法通常采用特征融合或领域适应技术，将事件引入以提高边界的完整性。然而，这些方法并未改善外观特征的质量，这严重阻碍了大多数使用的判别模型（基于视觉特征学习运动场映射）和生成模型（根据给定的视觉特征生成运动场）。因此，我们提出了一个全新的框架——Diff-ABFlow，该框架基于扩散模型，并结合帧事件在外观边界上的融合技术。这种新方法不依赖于退化的视觉特征，而是从模糊光流中学习清晰光流的映射，从而提升在困难条件下的性能。
URL
https://arxiv.org/abs/2510.10577
PDF
-
Ortho-Fuse: Orthomosaic Generation for Sparse High-Resolution Crop Health Maps Through Intermediate Optical Flow Estimation
2025-10-11 22:33:34Rugved Katole, Christopher StewartAbstract
AI-driven crop health mapping systems offer substantial advantages over conventional monitoring approaches through accelerated data acquisition and cost reduction. However, widespread farmer adoption remains constrained by technical limitations in orthomosaic generation from sparse aerial imagery datasets. Traditional photogrammetric reconstruction requires 70-80\% inter-image overlap to establish sufficient feature correspondences for accurate geometric registration. AI-driven systems operating under resource-constrained conditions cannot consistently achieve these overlap thresholds, resulting in degraded reconstruction quality that undermines user confidence in autonomous monitoring technologies. In this paper, we present Ortho-Fuse, an optical flow-based framework that enables the generation of a reliable orthomosaic with reduced overlap requirements. Our approach employs intermediate flow estimation to synthesize transitional imagery between consecutive aerial frames, artificially augmenting feature correspondences for improved geometric reconstruction. Experimental validation demonstrates a 20\% reduction in minimum overlap requirements. We further analyze adoption barriers in precision agriculture to identify pathways for enhanced integration of AI-driven monitoring systems.
Abstract (translated)
AI驱动的农作物健康地图系统通过加速数据获取和降低成本，相对于传统监测方法提供了显著的优势。然而，由于从稀疏的航空影像数据集中生成正射镶嵌图的技术限制，这些系统的广泛农民采用受到了阻碍。传统的摄影测量重建需要70-80%的照片间重叠度来建立足够的特征对应关系以实现准确的几何配准。在资源受限条件下运行的AI系统无法持续达到这种重叠阈值，导致重建质量下降，损害了用户对自主监测技术的信心。 本文提出了Ortho-Fuse框架，这是一个基于光学流的方法，可以在减少重叠要求的情况下生成可靠的正射镶嵌图。我们的方法利用中间流量估计来合成连续航拍帧之间的过渡影像，从而人工增加特征对应关系以改善几何重建。实验验证表明，这可以将最低重叠度需求降低20%。 我们进一步分析了精准农业中的采用障碍，并确定了AI驱动监测系统集成的改进路径。
URL
https://arxiv.org/abs/2510.10360
PDF
-
Dynamic Weight-based Temporal Aggregation for Low-light Video Enhancement
2025-10-10 15:00:31Ruirui Lin, Guoxi Huang, Nantheera AnantrasirichaiAbstract
Low-light video enhancement (LLVE) is challenging due to noise, low contrast, and color degradations. Learning-based approaches offer fast inference but still struggle with heavy noise in real low-light scenes, primarily due to limitations in effectively leveraging temporal information. In this paper, we address this issue with DWTA-Net, a novel two-stage framework that jointly exploits short- and long-term temporal cues. Stage I employs Visual State-Space blocks for multi-frame alignment, recovering brightness, color, and structure with local consistency. Stage II introduces a recurrent refinement module with dynamic weight-based temporal aggregation guided by optical flow, adaptively balancing static and dynamic regions. A texture-adaptive loss further preserves fine details while promoting smoothness in flat areas. Experiments on real-world low-light videos show that DWTA-Net effectively suppresses noise and artifacts, delivering superior visual quality compared with state-of-the-art methods.
Abstract (translated)
低光视频增强（LLVE）由于噪声、对比度低和颜色退化的问题而具有挑战性。基于学习的方法提供了快速的推理，但仍然难以处理真实低光场景中的严重噪声问题，主要原因是无法有效利用时间信息。在本文中，我们通过引入DWTA-Net框架来解决这一问题，这是一个新颖的两阶段框架，能够同时利用短时和长时的时间线索。 第一阶段使用视觉状态空间块进行多帧对齐，恢复亮度、颜色和结构，并保持局部一致性。 第二阶段则引入了一个基于光学流引导动态权重的时间聚合递归细化模块，根据场景自适应地平衡静态区域和平移区域的处理。此外，一个纹理自适应损失被用于保留细节的同时在平坦区域促进平滑度。 实验结果表明，在真实世界的低光视频上，DWTA-Net能够有效地抑制噪声和伪影，并且视觉质量明显优于最先进的方法。
URL
https://arxiv.org/abs/2510.09450
PDF
-
Re-Identifying K={a}k={a} with AI-Automated Video Key Frame Extraction
2025-10-09 19:46:46Paula Maddigan, Andrew Lensen, Rachael C. ShawAbstract
Accurate recognition and re-identification of individual animals is essential for successful wildlife population monitoring. Traditional methods, such as leg banding of birds, are time consuming and invasive. Recent progress in artificial intelligence, particularly computer vision, offers encouraging solutions for smart conservation and efficient automation. This study presents a unique pipeline for extracting high-quality key frames from videos of kÄkÄ (Nestor meridionalis), a threatened forest-dwelling parrot in New Zealand. Key frame extraction is well-studied in person re-identification, however, its application to wildlife is limited. Using video recordings at a custom-built feeder, we extract key frames and evaluate the re-identification performance of our pipeline. Our unsupervised methodology combines object detection using YOLO and Grounding DINO, optical flow blur detection, image encoding with DINOv2, and clustering methods to identify representative key frames. The results indicate that our proposed key frame selection methods yield image collections which achieve high accuracy in kÄkÄ re-identification, providing a foundation for future research using media collected in more diverse and challenging environments. Through the use of artificial intelligence and computer vision, our non-invasive and efficient approach provides a valuable alternative to traditional physical tagging methods for recognising kÄkÄ individuals and therefore improving the monitoring of populations. This research contributes to developing fresh approaches in wildlife monitoring, with applications in ecology and conservation biology.
Abstract (translated)
准确识别和重新识别野生动物个体对于成功监测野生动植物种群至关重要。传统方法，例如给鸟类佩戴脚环，既耗时又具有侵扰性。近年来，人工智能（特别是计算机视觉）的进步为智能保护和高效自动化提供了令人鼓舞的解决方案。本研究提出了一套独特的管道系统，用于从新西兰特有濒危森林鹦鹉——kākā (Nestor meridionalis) 的视频中提取高质量的关键帧。关键帧提取在人员重识别领域已得到充分研究，但在野生动物应用中的使用还相对有限。 我们通过定制设计的喂食器记录下的视频来提取这些关键帧，并评估该管道系统的重新识别性能。我们的无监督方法结合了YOLO和Grounding DINO进行物体检测、基于光流模糊检测的方法以及利用DINOv2进行图像编码，最后采用聚类技术以确定代表性的关键帧。 实验结果显示，我们提出的选帧方法能够生成高精度的用于kākā重新识别的图像集，在更多样化且更具挑战性的环境中采集媒体的基础上奠定了未来研究的基础。通过使用人工智能和计算机视觉，我们的非侵入性和高效的方法为传统的物理标记方法提供了一个有价值的替代方案，以更有效地识别个体 kākā 并改进种群监测。 这项研究有助于开发新的野生动物监测方法，在生态学和保护生物学领域具有广泛的应用价值。
URL
https://arxiv.org/abs/2510.08775
PDF
-
FMANet: A Novel Dual-Phase Optical Flow Approach with Fusion Motion Attention Network for Robust Micro-expression Recognition
2025-10-09 05:36:40Luu Tu Nguyen, Vu Tram Anh Khuong, Thi Bich Phuong Man, Thi Duyen Ngo, Thanh Ha LeAbstract
Facial micro-expressions, characterized by their subtle and brief nature, are valuable indicators of genuine emotions. Despite their significance in psychology, security, and behavioral analysis, micro-expression recognition remains challenging due to the difficulty of capturing subtle facial movements. Optical flow has been widely employed as an input modality for this task due to its effectiveness. However, most existing methods compute optical flow only between the onset and apex frames, thereby overlooking essential motion information in the apex-to-offset phase. To address this limitation, we first introduce a comprehensive motion representation, termed Magnitude-Modulated Combined Optical Flow (MM-COF), which integrates motion dynamics from both micro-expression phases into a unified descriptor suitable for direct use in recognition networks. Building upon this principle, we then propose FMANet, a novel end-to-end neural network architecture that internalizes the dual-phase analysis and magnitude modulation into learnable modules. This allows the network to adaptively fuse motion cues and focus on salient facial regions for classification. Experimental evaluations on the MMEW, SMIC, CASME-II, and SAMM datasets, widely recognized as standard benchmarks, demonstrate that our proposed MM-COF representation and FMANet outperforms existing methods, underscoring the potential of a learnable, dual-phase framework in advancing micro-expression recognition.
Abstract (translated)
面部微表情由于其微妙和短暂的特性，是真实情感的重要指标。尽管在心理学、安全和行为分析领域具有重要意义，但由于捕捉细微面部动作的困难性，微表情识别仍然极具挑战性。光学流因其有效性而广泛用于此类任务作为输入模式。然而，大多数现有的方法仅计算从起始帧到峰值帧的光学流，从而忽略了顶峰至结束阶段的重要运动信息。 为了解决这一局限性，我们首先引入了一种全面的运动表示方式——调制幅度组合光学流（Magnitude-Modulated Combined Optical Flow, MM-COF），它将微表情两个相位中的运动动态集成到一个统一描述符中，可以直接用于识别网络。在此基础上，我们进一步提出FMANet，这是一种新的端到端神经网络架构，内部化了双相分析和幅度调制为可学习模块。这使网络能够自适应地融合运动线索并关注于面部显著区域以进行分类。 在MMEW、SMIC、CASME-II和SAMM等广泛认可的标准数据集上的实验评估表明，我们提出的MM-COF表示法和FMANet优于现有方法，突显了可学习双相框架在推进微表情识别方面的潜力。
URL
https://arxiv.org/abs/2510.07810
PDF
-
Trajectory Conditioned Cross-embodiment Skill Transfer
2025-10-09 04:26:06YuHang Tang, Yixuan Lou, Pengfei Han, Haoming Song, Xinyi Ye, Dong Wang, Bin ZhaoAbstract
Learning manipulation skills from human demonstration videos presents a promising yet challenging problem, primarily due to the significant embodiment gap between human body and robot manipulators. Existing methods rely on paired datasets or hand-crafted rewards, which limit scalability and generalization. We propose TrajSkill, a framework for Trajectory Conditioned Cross-embodiment Skill Transfer, enabling robots to acquire manipulation skills directly from human demonstration videos. Our key insight is to represent human motions as sparse optical flow trajectories, which serve as embodiment-agnostic motion cues by removing morphological variations while preserving essential dynamics. Conditioned on these trajectories together with visual and textual inputs, TrajSkill jointly synthesizes temporally consistent robot manipulation videos and translates them into executable actions, thereby achieving cross-embodiment skill transfer. Extensive experiments are conducted, and the results on simulation data (MetaWorld) show that TrajSkill reduces FVD by 39.6\% and KVD by 36.6\% compared with the state-of-the-art, and improves cross-embodiment success rate by up to 16.7\%. Real-robot experiments in kitchen manipulation tasks further validate the effectiveness of our approach, demonstrating practical human-to-robot skill transfer across embodiments.
Abstract (translated)
从人类演示视频中学习操作技能是一个有前景但又极具挑战性的问题，主要原因是人手和机器人执行器之间存在显著的身体差异。现有的方法依赖于成对的数据集或手工制作的奖励机制，这限制了可扩展性和泛化能力。我们提出了一个名为TrajSkill的框架，这是一种基于轨迹条件跨体形技能转移的方法，能够使机器人直接从人类演示视频中学习操作技能。 我们的核心观点是将人类的动作表示为稀疏光流轨迹，这些轨迹通过消除形态变化而保留了关键的动力学特征，并作为与具体身体形式无关的动作线索。在这些轨迹、视觉和文本输入的条件下，TrajSkill可以同时生成时间上一致的机器人操作视频，并将其转化为可执行的操作动作，从而实现跨体形技能转移。 我们进行了广泛的实验，在模拟数据（MetaWorld）上的结果显示，TrajSkill相比现有最佳方法减少了39.6%的FVD分数和36.6%的KVD分数，并将跨身体形式的成功率提高了高达16.7%。在厨房操作任务中的真实机器人实验进一步验证了我们方法的有效性，展示了从人类到机器人的实际技能转移能力，能够跨越不同的身体形式。
URL
https://arxiv.org/abs/2510.07773
PDF
-
Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model
2025-10-07 20:29:27Danush Kumar Venkatesh, Adam Schmidt, Muhammad Abdullah Jamal, Omid MohareriAbstract
Surgical video datasets are essential for scene understanding, enabling procedural modeling and intra-operative support. However, these datasets are often heavily imbalanced, with rare actions and tools under-represented, which limits the robustness of downstream models. We address this challenge with $SurgiFlowVid$, a sparse and controllable video diffusion framework for generating surgical videos of under-represented classes. Our approach introduces a dual-prediction diffusion module that jointly denoises RGB frames and optical flow, providing temporal inductive biases to improve motion modeling from limited samples. In addition, a sparse visual encoder conditions the generation process on lightweight signals (e.g., sparse segmentation masks or RGB frames), enabling controllability without dense annotations. We validate our approach on three surgical datasets across tasks including action recognition, tool presence detection, and laparoscope motion prediction. Synthetic data generated by our method yields consistent gains of 10-20% over competitive baselines, establishing $SurgiFlowVid$ as a promising strategy to mitigate data imbalance and advance surgical video understanding methods.
Abstract (translated)
外科手术视频数据集对于场景理解至关重要，可以促进程序建模和术中支持。然而，这些数据集往往不平衡，罕见的动作和工具被严重低估，这限制了下游模型的鲁棒性。我们通过引入$SurgiFlowVid$来解决这一挑战，这是一个稀疏且可控的视频扩散框架，用于生成代表性不足类别的手术视频。我们的方法介绍了一个双预测扩散模块，该模块同时对RGB帧和光流进行去噪处理，提供时间归纳偏差以从有限样本中改进运动建模。此外，一个稀疏视觉编码器根据轻量级信号（例如稀疏分割掩码或RGB帧）来调节生成过程，从而在没有密集注释的情况下实现可控性。 我们在三个手术数据集上验证了我们的方法，涵盖了包括动作识别、工具存在检测和腹腔镜运动预测在内的任务。我们通过实验表明，由我们的方法生成的合成数据相对于竞争基线带来了10-20%的一致性能提升，从而将$SurgiFlowVid$确立为一种有前景的方法来缓解数据不平衡并推进手术视频理解技术的进步。
URL
https://arxiv.org/abs/2510.07345
PDF
-
Flow4Agent: Long-form Video Understanding via Motion Prior from Optical Flow
2025-10-07 12:01:57Ruyang Liu, Shangkun Sun, Haoran Tang, Ge Li, Wei GaoAbstract
Long-form video understanding has always been a challenging problem due to the significant redundancy in both temporal and spatial contents. This challenge is further exacerbated by the limited context length of Multimodal Large Language Models (MLLMs). To address this issue, many previous works have attempted to extract key video information, where the "key" is typically semantic-aware and heavily dependent on the CLIP model as prior. In this paper, we propose Flow4Agent, a novel framework that pioneeringly incorporates motion priors from optical flow to facilitate LLM-based long video understanding. Flow4Agent mitigates the redundancy in long videos at both temporal and spatial levels through two core modules: Temporal Granularity Optimization (TGO) adaptively refines framelevel hierarchies, which first leverages coarse flow priors to group similar visual contents and then applies semantic priors to filter out highly irrelevant scene information. Motion Token Pruning (MTP) further refines the intra-frame visual representations, pruning high-redundancy video tokens using fine-grained optical flow information. Extensive experiments demonstrate that our Flow4Agent outperforms existing methods across a wide range of video MLLM benchmarks, especially for hour-level video understanding tasks, achieving 64.7% on Video-MME, 71.4% on MLVU and 60.4% on LongVideoBench.
Abstract (translated)
长视频理解一直是一个具有挑战性的问题，原因是时间上和空间上的内容都存在大量冗余。这一难题由于多模态大型语言模型（MLLMs）的有限上下文长度而进一步加剧。为了应对这个问题，许多先前的研究试图提取关键视频信息，“关键”通常指的是语义感知的信息，并且高度依赖于CLIP模型作为先验知识。在本文中，我们提出了Flow4Agent，这是一个创新框架，它首次将来自光学流的运动先验引入到基于LLM（大型语言模型）的长视频理解中。通过两个核心模块——时间粒度优化（TGO）和运动标记修剪（MTP），Flow4Agent在时间和空间层面上减轻了长时间视频中的冗余。 具体来说： - **时间粒度优化（TGO）** 模块会根据粗略的光学流先验将相似视觉内容进行分组，然后利用语义先验过滤掉高度无关场景信息。 - **运动标记修剪（MTP）** 进一步精炼帧内视觉表示，通过细粒度的光学流信息删除高冗余视频标记。 实验结果表明，我们的Flow4Agent在广泛的视频MLLM基准测试中优于现有方法，特别是在小时级别的视频理解任务上表现出色。具体来说，在Video-MME、MLVU和LongVideoBench这些数据集上的表现分别为64.7%、71.4% 和 60.4%，显示了该框架在处理长时视频中的优越性。
URL
https://arxiv.org/abs/2510.05836
PDF
