Optical_Flow
-
TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring
2025-08-07 00:17:45Zhu Xu, Ting Lei, Zhimin Li, Guan Wang, Qingchao Chen, Yuxin Peng, Yang liuAbstract
Dynamic Scene Graph Generation (DSGG) aims to create a scene graph for each video frame by detecting objects and predicting their relationships. Weakly Supervised DSGG (WS-DSGG) reduces annotation workload by using an unlocalized scene graph from a single frame per video for training. Existing WS-DSGG methods depend on an off-the-shelf external object detector to generate pseudo labels for subsequent DSGG training. However, detectors trained on static, object-centric images struggle in dynamic, relation-aware scenarios required for DSGG, leading to inaccurate localization and low-confidence proposals. To address the challenges posed by external object detectors in WS-DSGG, we propose a Temporal-enhanced Relation-aware Knowledge Transferring (TRKT) method, which leverages knowledge to enhance detection in relation-aware dynamic scenarios. TRKT is built on two key components:(1)Relation-aware knowledge mining: we first employ object and relation class decoders that generate category-specific attention maps to highlight both object regions and interactive areas. Then we propose an Inter-frame Attention Augmentation strategy that exploits optical flow for neighboring frames to enhance the attention maps, making them motion-aware and robust to motion blur. This step yields relation- and motion-aware knowledge mining for WS-DSGG. (2) we introduce a Dual-stream Fusion Module that integrates category-specific attention maps into external detections to refine object localization and boost confidence scores for object proposals. Extensive experiments demonstrate that TRKT achieves state-of-the-art performance on Action Genome dataset. Our code is avaliable at this https URL.
Abstract (translated)
动态场景图生成（DSGG）的目标是通过检测物体并预测它们之间的关系，为视频中的每一帧创建一个场景图。弱监督的动态场景图生成（WS-DSGG）方法通过使用来自每个视频单个帧的未定位的场景图来减少标注工作量。现有的WS-DSGG方法依赖于现成的外部对象检测器来生成用于后续DSGG训练的伪标签。然而，基于静态、以物体为中心的图像进行训练的对象检测器在动态关系感知场景下表现不佳，导致定位不准确和低置信度建议。 为了解决WS-DSGG中外部对象检测器带来的挑战，我们提出了一种时间增强的关系感知知识迁移（TRKT）方法，该方法利用知识来提高在关系感知动态场景中的检测性能。TRKT建立在两个关键组件上： 1. **关系感知的知识挖掘**：首先使用物体和关系类解码器生成特定于类别的注意图以突出显示对象区域及其互动区。然后我们提出了一种跨帧注意力增强策略，利用光流来提高相邻帧中的注意图，使其能够感知运动，并且在面对运动模糊时更为鲁棒。这一步产生了适用于WS-DSGG的关系和运动感知的知识挖掘。 2. **双流融合模块**：此模块将特定于类别的注意图集成到外部检测结果中，以细化物体定位并提高对象建议的置信度得分。 广泛的实验表明，TRKT在Action Genome数据集上达到了最先进的性能。我们的代码可以在这里访问（原文中的URL链接）。
URL
https://arxiv.org/abs/2508.04943
PDF
-
Pseudo Depth Meets Gaussian: A Feed-forward RGB SLAM Baseline
2025-08-06 16:16:58Linqing Zhao, Xiuwei Xu, Yirui Wang, Hao Wang, Wenzhao Zheng, Yansong Tang, Haibin Yan, Jiwen LuAbstract
Incrementally recovering real-sized 3D geometry from a pose-free RGB stream is a challenging task in 3D reconstruction, requiring minimal assumptions on input data. Existing methods can be broadly categorized into end-to-end and visual SLAM-based approaches, both of which either struggle with long sequences or depend on slow test-time optimization and depth sensors. To address this, we first integrate a depth estimator into an RGB-D SLAM system, but this approach is hindered by inaccurate geometric details in predicted depth. Through further investigation, we find that 3D Gaussian mapping can effectively solve this problem. Building on this, we propose an online 3D reconstruction method using 3D Gaussian-based SLAM, combined with a feed-forward recurrent prediction module to directly infer camera pose from optical flow. This approach replaces slow test-time optimization with fast network inference, significantly improving tracking speed. Additionally, we introduce a local graph rendering technique to enhance robustness in feed-forward pose prediction. Experimental results on the Replica and TUM-RGBD datasets, along with a real-world deployment demonstration, show that our method achieves performance on par with the state-of-the-art SplaTAM, while reducing tracking time by more than 90\%.
Abstract (translated)
从无姿态RGB流中逐步恢复真实的3D几何形状是3D重建中的一个挑战性任务，该任务需要对输入数据做出最少的假设。现有的方法主要可以分为端到端和基于视觉SLAM（Simultaneous Localization and Mapping）的方法，这些方法要么难以处理长序列的问题，要么依赖于耗时的测试时间优化以及深度传感器。 为了解决这些问题，我们首先将一个深度估计器整合到了RGB-D SLAM系统中。然而这种方法由于预测出的深度不够准确导致几何细节不精确而受阻。通过进一步研究，我们发现基于3D高斯映射的方法可以有效解决这一问题。在此基础上，我们提出了一种使用基于3D高斯映射SLAM结合前馈递归预测模块直接从光流中推断相机姿态的在线3D重建方法。这种方法将耗时的测试时间优化替换为快速网络推理，显著提高了跟踪速度。此外，我们还引入了局部图渲染技术来增强前馈姿态预测的鲁棒性。 在Replica和TUM-RGBD数据集上的实验结果以及一个真实的部署演示表明，我们的方法与当前最先进的SplatMap方法性能相当，同时将跟踪时间减少了超过90%。
URL
https://arxiv.org/abs/2508.04597
PDF
-
Improving Tactile Gesture Recognition with Optical Flow
2025-08-06 11:33:21Shaohong Zhong, Alessandro Albini, Giammarco Caroleo, Giorgio Cannata, Perla MaiolinoAbstract
Tactile gesture recognition systems play a crucial role in Human-Robot Interaction (HRI) by enabling intuitive communication between humans and robots. The literature mainly addresses this problem by applying machine learning techniques to classify sequences of tactile images encoding the pressure distribution generated when executing the gestures. However, some gestures can be hard to differentiate based on the information provided by tactile images alone. In this paper, we present a simple yet effective way to improve the accuracy of a gesture recognition classifier. Our approach focuses solely on processing the tactile images used as input by the classifier. In particular, we propose to explicitly highlight the dynamics of the contact in the tactile image by computing the dense optical flow. This additional information makes it easier to distinguish between gestures that produce similar tactile images but exhibit different contact dynamics. We validate the proposed approach in a tactile gesture recognition task, showing that a classifier trained on tactile images augmented with optical flow information achieved a 9% improvement in gesture classification accuracy compared to one trained on standard tactile images.
Abstract (translated)
触觉手势识别系统在人机交互（HRI）中扮演着至关重要的角色，通过实现人类与机器人之间的直观交流来发挥作用。现有文献主要通过应用机器学习技术来分类编码手势执行过程中产生的压力分布的触觉图像序列，以此解决这一问题。然而，仅凭触觉图像是难以区分某些手势的。在这篇论文中，我们提出了一种简单而有效的方法来提高手势识别分类器的准确性。我们的方法专注于处理分类器使用的输入触觉图像。具体来说，我们建议通过计算密集光流来明确突出触觉图像中的接触动态，这将有助于区分那些虽然生成相似触觉图但表现出不同接触动态的手势。我们在一个触觉手势识别任务中验证了所提出的方法，并展示了基于光学流量信息增强的触觉图像训练得到的分类器在手势分类准确性方面相较于仅使用标准触觉图像训练提高了9%。
URL
https://arxiv.org/abs/2508.04338
PDF
-
Video Demoireing using Focused-Defocused Dual-Camera System
2025-08-05 13:49:49Xuan Dong, Xiangyuan Sun, Xia Wang, Jian Song, Ya Li, Weixin LiAbstract
Moire patterns, unwanted color artifacts in images and videos, arise from the interference between spatially high-frequency scene contents and the spatial discrete sampling of digital cameras. Existing demoireing methods primarily rely on single-camera image/video processing, which faces two critical challenges: 1) distinguishing moire patterns from visually similar real textures, and 2) preserving tonal consistency and temporal coherence while removing moire artifacts. To address these issues, we propose a dual-camera framework that captures synchronized videos of the same scene: one in focus (retaining high-quality textures but may exhibit moire patterns) and one defocused (with significantly reduced moire patterns but blurred textures). We use the defocused video to help distinguish moire patterns from real texture, so as to guide the demoireing of the focused video. We propose a frame-wise demoireing pipeline, which begins with an optical flow based alignment step to address any discrepancies in displacement and occlusion between the focused and defocused frames. Then, we leverage the aligned defocused frame to guide the demoireing of the focused frame using a multi-scale CNN and a multi-dimensional training loss. To maintain tonal and temporal consistency, our final step involves a joint bilateral filter to leverage the demoireing result from the CNN as the guide to filter the input focused frame to obtain the final output. Experimental results demonstrate that our proposed framework largely outperforms state-of-the-art image and video demoireing methods.
Abstract (translated)
摩尔纹是图像和视频中不需要的颜色伪影，它们源自空间高频场景内容与数字相机空间离散采样之间的干扰。现有的去摩尔纹方法主要依赖于单摄像头的图像或视频处理，面临着两个关键挑战：1）区分摩尔纹与视觉上相似的真实纹理；2）在去除摩尔纹的同时保持色调一致性和时间连贯性。为了解决这些问题，我们提出了一种双摄像头框架，该框架捕捉同一场景同步录制的两段视频：一段是聚焦模式（保留高质量的纹理但可能显示摩尔纹），另一段是非聚焦模式（显著减少摩尔纹但会模糊纹理）。利用非聚焦视频来帮助区分摩尔纹与真实纹理，并以此指导对聚焦视频进行去摩尔处理。我们提出了一种帧级去摩尔流水线，首先采用基于光学流的对齐步骤解决聚焦和非聚焦帧在位移和遮挡上的不一致问题。随后，利用已对齐的非聚焦帧来引导聚焦帧的去摩尔处理，通过多尺度卷积神经网络（CNN）及多维度训练损失实现这一过程。为了保持色调和时间的一致性，我们的最终步骤涉及联合双边滤波器，利用CNN输出的去摩尔结果作为指导过滤输入的聚焦视频，以获得最终输出。 实验结果显示，我们提出的框架在图像和视频去摩尔处理方法中远远超过了当前的最佳水平。
URL
https://arxiv.org/abs/2508.03449
PDF
-
ActionSink: Toward Precise Robot Manipulation with Dynamic Integration of Action Flow
2025-08-05 08:46:17Shanshan Guo, Xiwen Liang, Junfan Lin, Yuzheng Zhuang, Liang Lin, Xiaodan LiangAbstract
Language-instructed robot manipulation has garnered significant interest due to the potential of learning from collected data. While the challenges in high-level perception and planning are continually addressed along the progress of general large pre-trained models, the low precision of low-level action estimation has emerged as the key limiting factor in manipulation performance. To this end, this paper introduces a novel robot manipulation framework, i.e., ActionSink, to pave the way toward precise action estimations in the field of learning-based robot manipulation. As the name suggests, ActionSink reformulates the actions of robots as action-caused optical flows from videos, called "action flow", in a self-supervised manner, which are then used to be retrieved and integrated to enhance the action estimation. Specifically, ActionSink incorporates two primary modules. The first module is a coarse-to-fine action flow matcher, which continuously refines the accuracy of action flow via iterative retrieval and denoising process. The second module is a dynamic action flow integrator, which employs a working memory pool that dynamically and efficiently manages the historical action flows that should be used to integrate to enhance the current action estimation. In this module, a multi-layer fusion module is proposed to integrate direct estimation and action flows from both the current and the working memory, achieving highly accurate action estimation through a series of estimation-integration processes. Our ActionSink framework outperformed prior SOTA on the LIBERO benchmark by a 7.9\% success rate, and obtained nearly an 8\% accuracy gain on the challenging long-horizon visual task LIBERO-Long.
Abstract (translated)
基于指令的语言引导机器人操作由于可以从收集的数据中学习而引起了广泛的关注。尽管随着通用大型预训练模型的进步，高级感知和规划方面的挑战正在不断得到解决，但低级动作估计的精度不高已成为操控性能的主要限制因素。为此，本文提出了一种新的机器人操作框架——ActionSink，旨在推动基于学习的机器人操纵领域精确动作估算的发展。 正如其名称所暗示的那样，ActionSink 通过自监督的方式将机器人的动作重新表述为视频中由动作引起的光流（称为“动作流”），然后将其检索并整合以增强动作估计。具体来说，ActionSink 包含两个主要模块：一个是从粗到细的动作流匹配器和一个动态的动作流集成器。 第一个模块是一个从粗到细的动作流匹配器，通过迭代检索和去噪过程不断细化动作流的精度。第二个模块是动态动作流集成器，它使用一个工作内存池来动态且高效地管理需要用于当前动作估算的历史动作流。在这个模块中，提出了一个多层融合模块以整合直接估计和来自当前以及工作内存的动作流，通过一系列估计-集成过程实现了高度准确的动作估计。 我们的ActionSink框架在LIBERO基准测试中比之前的最先进方法（SOTA）高出7.9%的成功率，并且在具有挑战性的长时间视觉任务LIBERO-Long上获得了近8%的准确性提升。
URL
https://arxiv.org/abs/2508.03218
PDF
-
PMR: Physical Model-Driven Multi-Stage Restoration of Turbulent Dynamic Videos
2025-08-01 08:06:41Tao Wu, Jingyuan Ye, Ying FuAbstract
Geometric distortions and blurring caused by atmospheric turbulence degrade the quality of long-range dynamic scene videos. Existing methods struggle with restoring edge details and eliminating mixed distortions, especially under conditions of strong turbulence and complex dynamics. To address these challenges, we introduce a Dynamic Efficiency Index ($DEI$), which combines turbulence intensity, optical flow, and proportions of dynamic regions to accurately quantify video dynamic intensity under varying turbulence conditions and provide a high-dynamic turbulence training dataset. Additionally, we propose a Physical Model-Driven Multi-Stage Video Restoration ($PMR$) framework that consists of three stages: \textbf{de-tilting} for geometric stabilization, \textbf{motion segmentation enhancement} for dynamic region refinement, and \textbf{de-blurring} for quality restoration. $PMR$ employs lightweight backbones and stage-wise joint training to ensure both efficiency and high restoration quality. Experimental results demonstrate that the proposed method effectively suppresses motion trailing artifacts, restores edge details and exhibits strong generalization capability, especially in real-world scenarios characterized by high-turbulence and complex dynamics. We will make the code and datasets openly available.
Abstract (translated)
大气湍流引起的几何失真和模糊会降低长距离动态场景视频的质量。现有的方法在恢复边缘细节和消除混合失真方面面临挑战，尤其是在强湍流和复杂动态条件下表现不佳。为了解决这些问题，我们引入了一个动态效率指标（Dynamic Efficiency Index, $DEI$），它结合了湍流强度、光流以及动态区域的比例来准确量化不同湍流条件下的视频动态强度，并提供了高动态湍流训练数据集。此外，我们还提出了一种基于物理模型的多阶段视频恢复框架（Physical Model-Driven Multi-Stage Video Restoration, $PMR$），该框架包括三个阶段：**去倾斜**以实现几何稳定化、**运动分割增强**以优化动态区域、以及**去模糊**以提升图像质量。$PMR$ 使用轻量级骨干网络和分阶段联合训练，确保高效的同时保持高质量的恢复效果。 实验结果表明，所提出的方法能够有效抑制运动拖尾伪影，恢复边缘细节，并且在特征为高湍流和复杂动态的真实世界场景中表现出强大的泛化能力。我们将开源代码和数据集。
URL
https://arxiv.org/abs/2508.00406
PDF
-
Occlusion-robust Stylization for Drawing-based 3D Animation
2025-08-01 07:52:07Sunjae Yoon, Gwanhyeong Koo, Younghwan Lee, Ji Woo Hong, Chang D. YooAbstract
3D animation aims to generate a 3D animated video from an input image and a target 3D motion sequence. Recent advances in image-to-3D models enable the creation of animations directly from user-hand drawings. Distinguished from conventional 3D animation, drawing-based 3D animation is crucial to preserve artist's unique style properties, such as rough contours and distinct stroke patterns. However, recent methods still exhibit quality deterioration in style properties, especially under occlusions caused by overlapping body parts, leading to contour flickering and stroke blurring. This occurs due to a `stylization pose gap' between training and inference in stylization networks designed to preserve drawing styles in drawing-based 3D animation systems. The stylization pose gap denotes that input target poses used to train the stylization network are always in occlusion-free poses, while target poses encountered in an inference include diverse occlusions under dynamic motions. To this end, we propose Occlusion-robust Stylization Framework (OSF) for drawing-based 3D animation. We found that while employing object's edge can be effective input prior for guiding stylization, it becomes notably inaccurate when occlusions occur at inference. Thus, our proposed OSF provides occlusion-robust edge guidance for stylization network using optical flow, ensuring a consistent stylization even under occlusions. Furthermore, OSF operates in a single run instead of the previous two-stage method, achieving 2.4x faster inference and 2.1x less memory.
Abstract (translated)
3D动画的目标是从输入图像和目标3D运动序列生成3D动画视频。最近在图像到3D模型方面的发展使得能够直接从用户手绘的草图创建动画成为可能。与传统的3D动画相比，基于绘画的3D动画对于保留艺术家独特的风格属性（如粗糙轮廓和独特的笔触模式）至关重要。然而，尽管取得了这些进展，目前的方法仍然会在风格属性上表现出质量下降的问题，尤其是在由重叠身体部位导致遮挡时，会导致轮廓闪烁和笔迹模糊。这种现象是由于基于绘画的3D动画系统中用于保留绘制样式的风格化网络在训练与推理阶段之间存在的“姿态差距”（stylization pose gap）所引起的。这个姿态差距表示，在进行模型训练时使用的输入目标姿势总是没有遮挡的情况，而在进行推断时遇到的目标姿势则包含由于动态动作而产生的各种遮挡情况。 为此，我们提出了一种名为抗遮挡风格化框架（Occlusion-robust Stylization Framework, OSF）的方法来解决基于绘画的3D动画的问题。我们发现，虽然使用对象边缘作为引导风格化的有效输入先验是有效的，但在推断过程中发生遮挡时其准确性会显著降低。因此，我们的OSF通过利用光流为风格化网络提供抗遮挡的边缘指引，确保即使在存在遮挡的情况下也能保持一致的风格化效果。 此外，与先前采用的两阶段方法不同，OSF可以在单次运行中完成任务，从而实现了2.4倍更快的推理速度和2.1倍更少的记忆消耗。
URL
https://arxiv.org/abs/2508.00398
PDF
-
Video Forgery Detection with Optical Flow Residuals and Spatial-Temporal Consistency
2025-08-01 07:51:35Xi Xue, Kunio Suzuki, Nabarun Goswami, Takuya ShintateAbstract
The rapid advancement of diffusion-based video generation models has led to increasingly realistic synthetic content, presenting new challenges for video forgery detection. Existing methods often struggle to capture fine-grained temporal inconsistencies, particularly in AI-generated videos with high visual fidelity and coherent motion. In this work, we propose a detection framework that leverages spatial-temporal consistency by combining RGB appearance features with optical flow residuals. The model adopts a dual-branch architecture, where one branch analyzes RGB frames to detect appearance-level artifacts, while the other processes flow residuals to reveal subtle motion anomalies caused by imperfect temporal synthesis. By integrating these complementary features, the proposed method effectively detects a wide range of forged videos. Extensive experiments on text-to-video and image-to-video tasks across ten diverse generative models demonstrate the robustness and strong generalization ability of the proposed approach.
Abstract (translated)
基于扩散模型的视频生成技术的迅速发展，使得合成内容越来越逼真，这给视频伪造检测带来了新的挑战。现有的方法往往难以捕捉细微的时间不一致性问题，尤其是在视觉上高度逼真且运动连贯的人工智能生成视频中。为此，在这项工作中我们提出了一种利用时空一致性的检测框架，该框架结合了RGB外观特征和光流残差。模型采用双分支架构：一个分支分析RGB帧以检测外观层面的瑕疵，另一个分支处理光流残差以揭示由于时间合成不完美导致的细微运动异常。通过整合这些互补特征，所提出的方法能够有效检测各种伪造视频。 在十个不同的生成模型上进行的文字到视频和图像到视频任务的广泛实验表明了该方法具有强大的鲁棒性和泛化能力。
URL
https://arxiv.org/abs/2508.00397
PDF
-
Omni-Scan: Creating Visually-Accurate Digital Twin Object Models Using a Bimanual Robot with Handover and Gaussian Splat Merging
2025-08-01 06:36:19Tianshuang Qiu, Zehan Ma, Karim El-Refai, Hiya Shah, Chung Min Kim, Justin Kerr, Ken GoldbergAbstract
3D Gaussian Splats (3DGSs) are 3D object models derived from multi-view images. Such "digital twins" are useful for simulations, virtual reality, marketing, robot policy fine-tuning, and part inspection. 3D object scanning usually requires multi-camera arrays, precise laser scanners, or robot wrist-mounted cameras, which have restricted workspaces. We propose Omni-Scan, a pipeline for producing high-quality 3D Gaussian Splat models using a bi-manual robot that grasps an object with one gripper and rotates the object with respect to a stationary camera. The object is then re-grasped by a second gripper to expose surfaces that were occluded by the first gripper. We present the Omni-Scan robot pipeline using DepthAny-thing, Segment Anything, as well as RAFT optical flow models to identify and isolate objects held by a robot gripper while removing the gripper and the background. We then modify the 3DGS training pipeline to support concatenated datasets with gripper occlusion, producing an omni-directional (360 degree view) model of the object. We apply Omni-Scan to part defect inspection, finding that it can identify visual or geometric defects in 12 different industrial and household objects with an average accuracy of 83%. Interactive videos of Omni-Scan 3DGS models can be found at this https URL
Abstract (translated)
3D高斯点（3D Gaussian Splats，简称3DGS）是从多视角图像中生成的三维物体模型。这些“数字孪生”在仿真、虚拟现实、市场营销、机器人策略微调和部件检测等众多领域都非常有用。传统的三维物体扫描通常需要使用多个摄像头阵列、精密激光扫描仪或安装于机械臂腕部的相机，而后者的工作空间受到限制。 我们提出了一种名为Omni-Scan的新方法，该方法采用双臂机器人的抓取系统来生成高质量的3DGS模型。具体来说，一个机器人夹爪抓住物体并相对于固定摄像头旋转该对象，以使其不同侧面朝向摄像机；随后由第二个夹爪重新抓取同一物体，以便于先前被第一个夹爪遮挡的部分可以露出。我们利用DepthAny-thing、Segment Anything以及RAFT光流模型来识别和分离机器人夹持器所握的物件，并去除夹持器本身及背景部分。 在上述步骤完成后，我们会修改3DGS训练管道，以适应带有夹具遮挡的拼接数据集，最终生成一个全方位（360度视角）的对象模型。我们还利用Omni-Scan技术进行零件缺陷检测实验，在12种不同的工业和家用物品上实现了平均83%的准确率。 互动视频展示Omni-Scan 3DGS模型的具体应用可以访问以下链接：[此处插入实际URL]
URL
https://arxiv.org/abs/2508.00354
PDF
-
World Consistency Score: A Unified Metric for Video Generation Quality
2025-07-31 20:06:47Akshat Rakheja, Aarsh Ashdhir, Aryan Bhattacharjee, Vanshika SharmaAbstract
We introduce World Consistency Score (WCS), a novel unified evaluation metric for generative video models that emphasizes internal world consistency of the generated videos. WCS integrates four interpretable sub-components - object permanence, relation stability, causal compliance, and flicker penalty - each measuring a distinct aspect of temporal and physical coherence in a video. These submetrics are combined via a learned weighted formula to produce a single consistency score that aligns with human judgments. We detail the motivation for WCS in the context of existing video evaluation metrics, formalize each submetric and how it is computed with open-source tools (trackers, action recognizers, CLIP embeddings, optical flow), and describe how the weights of the WCS combination are trained using human preference data. We also outline an experimental validation blueprint: using benchmarks like VBench-2.0, EvalCrafter, and LOVE to test WCS's correlation with human evaluations, performing sensitivity analyses, and comparing WCS against established metrics (FVD, CLIPScore, VBench, FVMD). The proposed WCS offers a comprehensive and interpretable framework for evaluating video generation models on their ability to maintain a coherent "world" over time, addressing gaps left by prior metrics focused only on visual fidelity or prompt alignment.
Abstract (translated)
我们介绍了一种新颖的统一评估指标——世界一致性得分（World Consistency Score，WCS），用于评价生成视频模型。该指标特别强调了所生成视频内部的世界一致性。WCS集成了四个可解释的子组件：物体恒常性、关系稳定性、因果合规性和闪烁惩罚，每个子组件分别衡量视频在时间和物理连贯性方面的不同方面。这些子度量通过一个基于学习权重公式组合起来，产生一个单一的一致性得分，并且该得分能够与人类判断保持一致。 我们详细阐述了WCS相对于现有视频评估指标的动机、形式化定义每个子度量以及如何使用开源工具（跟踪器、动作识别器、CLIP嵌入、光流）进行计算。此外，还描述了如何通过利用人类偏好数据来训练WCS组合权重。 我们也概述了一个实验验证蓝图：使用VBench-2.0、EvalCrafter和LOVE等基准测试来检验WCS与人类评价的相关性；执行敏感性分析，并将WCS与其他已建立的度量（如FVD、CLIPScore、VBench、FVMD）进行比较。 提出的WCS提供了一个全面且可解释的框架，用于评估视频生成模型在其维持随时间变化的一致“世界”方面的能力。这弥补了之前仅关注视觉保真度或指令对齐的指标所留下的不足之处。
URL
https://arxiv.org/abs/2508.00144
PDF
-
Enhanced Velocity Field Modeling for Gaussian Video Reconstruction
2025-07-31 16:26:22Zhenyang Li, Xiaoyang Bai, Tongchen Zhang, Pengfei Shen, Weiwei Xu, Yifan PengAbstract
High-fidelity 3D video reconstruction is essential for enabling real-time rendering of dynamic scenes with realistic motion in virtual and augmented reality (VR/AR). The deformation field paradigm of 3D Gaussian splatting has achieved near-photorealistic results in video reconstruction due to the great representation capability of deep deformation networks. However, in videos with complex motion and significant scale variations, deformation networks often overfit to irregular Gaussian trajectories, leading to suboptimal visual quality. Moreover, the gradient-based densification strategy designed for static scene reconstruction proves inadequate to address the absence of dynamic content. In light of these challenges, we propose a flow-empowered velocity field modeling scheme tailored for Gaussian video reconstruction, dubbed FlowGaussian-VR. It consists of two core components: a velocity field rendering (VFR) pipeline which enables optical flow-based optimization, and a flow-assisted adaptive densification (FAD) strategy that adjusts the number and size of Gaussians in dynamic regions. We validate our model's effectiveness on multi-view dynamic reconstruction and novel view synthesis with multiple real-world datasets containing challenging motion scenarios, demonstrating not only notable visual improvements (over 2.5 dB gain in PSNR) and less blurry artifacts in dynamic textures, but also regularized and trackable per-Gaussian trajectories.
Abstract (translated)
高保真3D视频重建对于在虚拟现实（VR）和增强现实（AR）中实时渲染具有真实运动的动态场景至关重要。基于三维高斯点云的变形场范式，由于深度变形网络的强大表征能力，在视频重建方面已经实现了接近照片级的真实效果。然而，在复杂运动和显著尺度变化的视频中，变形网络往往会过度拟合不规则的高斯轨迹，导致视觉质量下降。此外，用于静态场景重建的基于梯度的稠密化策略在处理缺乏动态内容的问题上显得不足。 针对这些挑战，我们提出了一种专为高斯视频重建设计的速度场建模方案——FlowGaussian-VR。该方案包含两个核心组件：速度场渲染（VFR）管线，它支持以光流为基础进行优化；以及一种受流动辅助的自适应稠密化（FAD）策略，它可以调整动态区域中高斯的数量和大小。 我们使用多个包含挑战性运动场景的真实世界数据集，在多视角动态重建和新视点合成上验证了模型的有效性。结果显示，我们的方法不仅在视觉质量上有显著提升（PSNR值提高了超过2.5dB），而且减少了动态纹理中的模糊伪影，并确保每颗高斯的轨迹更加规律且可追踪。
URL
https://arxiv.org/abs/2507.23704
PDF
-
Learning to Prune Branches in Modern Tree-Fruit Orchards
2025-07-30 18:24:20Abhinav Jain, Cindy Grimm, Stefan LeeAbstract
Dormant tree pruning is labor-intensive but essential to maintaining modern highly-productive fruit orchards. In this work we present a closed-loop visuomotor controller for robotic pruning. The controller guides the cutter through a cluttered tree environment to reach a specified cut point and ensures the cutters are perpendicular to the branch. We train the controller using a novel orchard simulation that captures the geometric distribution of branches in a target apple orchard configuration. Unlike traditional methods requiring full 3D reconstruction, our controller uses just optical flow images from a wrist-mounted camera. We deploy our learned policy in simulation and the real-world for an example V-Trellis envy tree with zero-shot transfer, achieving a 30% success rate -- approximately half the performance of an oracle planner.
Abstract (translated)
休眠树木修剪是一项劳动密集型工作，但对维持现代高产果园至关重要。在这项工作中，我们提出了一种用于机器人修剪的闭环视觉运动控制器。该控制器引导切割工具穿过茂密的树丛环境以到达指定的切割点，并确保切割面与树枝垂直。我们使用一种新颖的果园模拟器进行控制器训练，这种模拟器捕捉到了目标苹果园配置中枝条的几何分布情况。与传统方法需要完整的3D重建不同，我们的控制器仅使用来自手腕安装摄像头提供的光学流图像。我们将学到的策略部署在仿真和实际环境中，用于处理一个示例V型支架艳丽树（envy tree），实现了零样本迁移学习，在测试中达到了约30%的成功率——这大约是理想规划器（oracle planner）性能的一半。
URL
https://arxiv.org/abs/2507.23015
PDF
-
Estimating 2D Camera Motion with Hybrid Motion Basis
2025-07-30 08:30:37Haipeng Li, Tianhao Zhou, Zhanglei Yang, Yi Wu, Yan Chen, Zijing Mao, Shen Cheng, Bing Zeng, Shuaicheng LiuAbstract
Estimating 2D camera motion is a fundamental computer vision task that models the projection of 3D camera movements onto the 2D image plane. Current methods rely on either homography-based approaches, limited to planar scenes, or meshflow techniques that use grid-based local homographies but struggle with complex non-linear transformations. A key insight of our work is that combining flow fields from different homographies creates motion patterns that cannot be represented by any single homography. We introduce CamFlow, a novel framework that represents camera motion using hybrid motion bases: physical bases derived from camera geometry and stochastic bases for complex scenarios. Our approach includes a hybrid probabilistic loss function based on the Laplace distribution that enhances training robustness. For evaluation, we create a new benchmark by masking dynamic objects in existing optical flow datasets to isolate pure camera motion. Experiments show CamFlow outperforms state-of-the-art methods across diverse scenarios, demonstrating superior robustness and generalization in zero-shot settings. Code and datasets are available at our project page: this https URL.
Abstract (translated)
估计2D相机运动是计算机视觉中的一个基本任务，它模拟了3D相机移动在2D图像平面上的投影。目前的方法依赖于基于齐次变换的方法（仅限于平面场景）或使用网格局部齐次变换的网流技术，但后者难以处理复杂的非线性转换。我们工作的一个关键见解是，结合来自不同齐次变换的流动场可以创建出任何单一齐次变换都无法表示的运动模式。为此，我们提出了CamFlow，这是一个新的框架，它利用混合运动基来表示相机运动：物理基来源于相机几何学，而随机基用于处理复杂场景。我们的方法包括一个基于拉普拉斯分布的混合概率损失函数，以增强训练过程中的鲁棒性。为了评估性能，我们在现有的光流数据集中遮蔽动态对象，从而创建了一个新的基准测试来隔离纯粹的相机运动。实验结果显示，在各种不同场景下，CamFlow均优于当前最先进的方法，并在无模型设置中展示了更强的稳健性和泛化能力。该项目的代码和数据集可在项目页面上获取：此链接。
URL
https://arxiv.org/abs/2507.22480
PDF
-
Unleashing the Power of Motion and Depth: A Selective Fusion Strategy for RGB-D Video Salient Object Detection
2025-07-29 14:30:48Jiahao He, Daerji Suolang, Keren Fu, Qijun ZhaoAbstract
Applying salient object detection (SOD) to RGB-D videos is an emerging task called RGB-D VSOD and has recently gained increasing interest, due to considerable performance gains of incorporating motion and depth and that RGB-D videos can be easily captured now in daily life. Existing RGB-D VSOD models have different attempts to derive motion cues, in which extracting motion information explicitly from optical flow appears to be a more effective and promising alternative. Despite this, there remains a key issue that how to effectively utilize optical flow and depth to assist the RGB modality in SOD. Previous methods always treat optical flow and depth equally with respect to model designs, without explicitly considering their unequal contributions in individual scenarios, limiting the potential of motion and depth. To address this issue and unleash the power of motion and depth, we propose a novel selective cross-modal fusion framework (SMFNet) for RGB-D VSOD, incorporating a pixel-level selective fusion strategy (PSF) that achieves optimal fusion of optical flow and depth based on their actual contributions. Besides, we propose a multi-dimensional selective attention module (MSAM) to integrate the fused features derived from PSF with the remaining RGB modality at multiple dimensions, effectively enhancing feature representation to generate refined features. We conduct comprehensive evaluation of SMFNet against 19 state-of-the-art models on both RDVS and DVisal datasets, making the evaluation the most comprehensive RGB-D VSOD benchmark up to date, and it also demonstrates the superiority of SMFNet over other models. Meanwhile, evaluation on five video benchmark datasets incorporating synthetic depth validates the efficacy of SMFNet as well. Our code and benchmark results are made publicly available at this https URL.
Abstract (translated)
将显著目标检测（SOD）应用于RGB-D视频是一种新兴的任务，被称为RGB-D VSOD（Video Salient Object Detection），由于结合运动和深度信息可以显著提高性能，并且现在日常生活中很容易捕捉到RGB-D视频，因此最近引起了越来越多的关注。现有的RGB-D VSOD模型在推导运动线索方面有不同的尝试，其中从光流中显式提取运动信息似乎是一种更有效且有前景的替代方法。尽管如此，仍然存在一个关键问题是如何有效地利用光流和深度来辅助RGB模态进行SOD。先前的方法总是将光流和深度在模型设计上等同对待，没有明确考虑它们在不同场景中的贡献差异，这限制了运动和深度的潜力。 为了解决这一问题并释放运动和深度的力量，我们提出了一种新颖的选择性跨模式融合框架（SMFNet），用于RGB-D VSOD。该框架结合了一个像素级别的选择性融合策略（PSF），基于光流和深度的实际贡献实现了最优融合。此外，我们还提出了一个多维选择性注意模块（MSAM），以在多个维度上将通过PSF获得的融合特征与剩余的RGB模态集成起来，从而有效增强特征表示生成精炼的特征。 我们在RDVS和DVisal数据集上对SMFNet进行了全面评估，使其成为迄今为止最全面的RGB-D VSOD基准测试，并且它也展示了SMFNet相对于其他模型的优势。同时，在包含合成深度的五个视频基准测试数据集上的评估验证了SMFNet的有效性。 我们的代码和基准结果公开提供在以下网址：[此链接]（请将方括号中的内容替换为实际提供的URL）。
URL
https://arxiv.org/abs/2507.21857
PDF
-
Hanging Around: Cognitive Inspired Reasoning for Reactive Robotics
2025-07-28 13:39:03Mihai Pomarlan, Stefano De Giorgis, Rachel Ringe, Maria M. Hedblom, Nikolaos TsiogkasAbstract
Situationally-aware artificial agents operating with competence in natural environments face several challenges: spatial awareness, object affordance detection, dynamic changes and unpredictability. A critical challenge is the agent's ability to identify and monitor environmental elements pertinent to its objectives. Our research introduces a neurosymbolic modular architecture for reactive robotics. Our system combines a neural component performing object recognition over the environment and image processing techniques such as optical flow, with symbolic representation and reasoning. The reasoning system is grounded in the embodied cognition paradigm, via integrating image schematic knowledge in an ontological structure. The ontology is operatively used to create queries for the perception system, decide on actions, and infer entities' capabilities derived from perceptual data. The combination of reasoning and image processing allows the agent to focus its perception for normal operation as well as discover new concepts for parts of objects involved in particular interactions. The discovered concepts allow the robot to autonomously acquire training data and adjust its subsymbolic perception to recognize the parts, as well as making planning for more complex tasks feasible by focusing search on those relevant object parts. We demonstrate our approach in a simulated world, in which an agent learns to recognize parts of objects involved in support relations. While the agent has no concept of handle initially, by observing examples of supported objects hanging from a hook it learns to recognize the parts involved in establishing support and becomes able to plan the establishment/destruction of the support relation. This underscores the agent's capability to expand its knowledge through observation in a systematic way, and illustrates the potential of combining deep reasoning [...].
Abstract (translated)
情景感知型人工智能代理在自然环境中操作时面临若干挑战：空间意识、物体可利用性检测、动态变化和不可预测性。其中关键的一个挑战是代理能够识别并监控与其目标相关联的环境元素的能力。我们的研究介绍了一种用于反应式机器人的神经符号模块化架构。该系统结合了执行对象识别及图像处理技术（如光流）的神经网络组件，以及基于本体论结构整合图式知识的符号表示和推理机制。 本体论在此被操作性地用来创建感知系统的查询、决定行动，并从感知数据中推断实体的能力。这种推理与图像处理相结合的方法使代理能够聚焦于正常运行所需的感知任务，同时还能发现参与特定交互的部分物体的新概念。所发现的概念允许机器人自主获取训练数据并调整其亚符号感知以识别这些部分；这使得通过集中搜索相关对象部件来进行更复杂任务的规划成为可能。 我们在一个模拟环境中展示了我们的方法：代理学会识别支持关系中涉及的对象部分。尽管一开始没有“把手”的概念，但通过观察挂在钩子上的支撑物体的例子，它学会了识别建立和支持关系的相关部分，并能够计划这种关系的确立或破坏。这强调了代理人系统地扩展其知识的能力，并展示了将深度推理与感知相结合的潜力。
URL
https://arxiv.org/abs/2507.20832
PDF
-
Video Self-Distillation for Single-Image Encoders: A Step Toward Physically Plausible Perception
2025-07-25 13:52:31Marcel Simon, Tae-Ho Kim, Seul-Ki YeomAbstract
Self-supervised image encoders such as DINO have recently gained significant interest for learning robust visual features without labels. However, most SSL methods train on static images and miss the temporal cues inherent in videos. We introduce a video-distilled single-image encoder trained to predict the next-frame representation from the current frame. This simple objective injects 3D spatial and temporal priors without optical flow or tracking. When pre-training on a single 2-hour video, our approach raises the mean Intersection-over-Union (mIoU) on ADE20K from 35.0 (DoRA) to 36.4 while remaining a drop-in replacement for image-only pipelines. Our results highlight video self-distillation as a lightweight route to geometry-aware perception an essential ingredient for physically plausible world models and Physical AI.
Abstract (translated)
近期，像DINO这样的自监督图像编码器因能够在无标签的情况下学习稳健的视觉特征而备受关注。然而，大多数自监督学习（SSL）方法都是在静态图像上进行训练，忽视了视频中固有的时间线索。为此，我们提出了一种基于单帧预测下一帧表示的视频蒸馏单一图像编码器。这种简单的目标无需光学流或跟踪技术就能注入3D空间和时间先验信息。当仅使用一段2小时长的视频进行预训练时，我们的方法使ADE20K数据集上的平均交并比（mIoU）从35.0（DoRA结果）提升到了36.4，并且可以作为图像管道中的直接替代方案。我们的成果强调了视频自蒸馏作为一种轻量级的方法，有助于几何感知的视觉理解，这对于构建物理上合理的世界模型和实现物理人工智能至关重要。
URL
https://arxiv.org/abs/2507.19272
PDF
-
EndoControlMag: Robust Endoscopic Vascular Motion Magnification with Periodic Reference Resetting and Hierarchical Tissue-aware Dual-Mask Contro
2025-07-21 06:47:44An Wanga, Rulin Zhou, Mengya Xu, Yiru Ye, Longfei Gou, Yiting Chang, Hao Chen, Chwee Ming Lim, Jiankun Wang, Hongliang RenAbstract
Visualizing subtle vascular motions in endoscopic surgery is crucial for surgical precision and decision-making, yet remains challenging due to the complex and dynamic nature of surgical scenes. To address this, we introduce EndoControlMag, a training-free, Lagrangian-based framework with mask-conditioned vascular motion magnification tailored to endoscopic environments. Our approach features two key modules: a Periodic Reference Resetting (PRR) scheme that divides videos into short overlapping clips with dynamically updated reference frames to prevent error accumulation while maintaining temporal coherence, and a Hierarchical Tissue-aware Magnification (HTM) framework with dual-mode mask dilation. HTM first tracks vessel cores using a pretrained visual tracking model to maintain accurate localization despite occlusions and view changes. It then applies one of two adaptive softening strategies to surrounding tissues: motion-based softening that modulates magnification strength proportional to observed tissue displacement, or distance-based exponential decay that simulates biomechanical force attenuation. This dual-mode approach accommodates diverse surgical scenarios-motion-based softening excels with complex tissue deformations while distance-based softening provides stability during unreliable optical flow conditions. We evaluate EndoControlMag on our EndoVMM24 dataset spanning four different surgery types and various challenging scenarios, including occlusions, instrument disturbance, view changes, and vessel deformations. Quantitative metrics, visual assessments, and expert surgeon evaluations demonstrate that EndoControlMag significantly outperforms existing methods in both magnification accuracy and visual quality while maintaining robustness across challenging surgical conditions. The code, dataset, and video results are available at this https URL.
Abstract (translated)
在内窥镜手术中，可视化微小的血管运动对于提高手术精度和决策至关重要，但由于手术场景的复杂性和动态性，这一过程仍然极具挑战。为了应对这些难题，我们提出了一种无需训练、基于拉格朗日方法且适用于内窥镜环境的框架——EndoControlMag，该框架专门针对血管运动放大进行了掩码条件优化。 我们的方法包含两个关键模块：周期参考重置（PRR）方案和层次化组织感知放大（HTM）框架。其中： - **周期参考重置（PRR）** 方案将视频分割成一系列短的、具有动态更新参考帧的重叠片段，以防止错误积累并保持时间上的连贯性。 - **层次化组织感知放大（HTM）** 框架包含双模式掩码膨胀。HTM首先使用预训练的视觉跟踪模型来追踪血管核心，即使在存在遮挡和视角变化的情况下也能维持精确的位置定位。随后，它采用两种自适应软化策略之一来处理周围组织：一种是基于运动的软化策略，根据观察到的组织位移调整放大强度；另一种是基于距离的指数衰减策略，模拟生物力学力的减弱。这种双模式方法能够应对各种手术场景——基于运动的软化策略在复杂组织变形的情况下表现优异，而基于距离的软化策略则为不稳定的光学流条件下提供稳定性。 我们利用涵盖四种不同手术类型及多种具有挑战性情景（包括遮挡、器械干扰、视角变化和血管变形）的EndoVMM24数据集对EndoControlMag进行了评估。定量指标、视觉评价以及专家外科医生的评价表明，EndoControlMag在放大精度和视觉质量方面显著优于现有方法，并且能够应对具有挑战性的手术条件。 我们的代码、数据集及视频结果均可在此网址获取：[https://this-url.com]（请将 [https://this-url.com] 替换为实际链接）。
URL
https://arxiv.org/abs/2507.15292
PDF
-
Systole-Conditioned Generative Cardiac Motion
2025-07-20 14:44:40Shahar Zuler, Gal Lifshitz, Hadar Averbuch-Elor, Dan RavivAbstract
Accurate motion estimation in cardiac computed tomography (CT) imaging is critical for assessing cardiac function and surgical planning. Data-driven methods have become the standard approach for dense motion estimation, but they rely on vast amounts of labeled data with dense ground-truth (GT) motion annotations, which are often unfeasible to obtain. To address this limitation, we present a novel approach that synthesizes realistically looking pairs of cardiac CT frames enriched with dense 3D flow field annotations. Our method leverages a conditional Variational Autoencoder (CVAE), which incorporates a novel multi-scale feature conditioning mechanism and is trained to generate 3D flow fields conditioned on a single CT frame. By applying the generated flow field to warp the given frame, we create pairs of frames that simulate realistic myocardium deformations across the cardiac cycle. These pairs serve as fully annotated data samples, providing optical flow GT annotations. Our data generation pipeline could enable the training and validation of more complex and accurate myocardium motion models, allowing for substantially reducing reliance on manual annotations. Our code, along with animated generated samples and additional material, is available on our project page: this https URL.
Abstract (translated)
在心脏计算机断层扫描（CT）成像中，准确的运动估计对于评估心脏功能和手术规划至关重要。数据驱动方法已成为密集运动估算的标准方法，但它们依赖于大量带有密集真实标签（GT）运动注释的数据集，这些数据往往难以获取。为了解决这一限制，我们提出了一种新方法，该方法通过生成具有密集3D流场标注的真实外观心脏CT帧对来合成数据。 我们的方法利用了条件变分自动编码器（CVAE），并引入了一个新颖的多尺度特征调节机制，使其能够根据单个CT帧生成3D流场。通过应用生成的流场来变形给定的帧，我们创建了一对模拟心脏周期中真实心肌变形的帧。这些配对作为完全标注的数据样本，提供光学流GT注释。 我们的数据生成流水线可以促进更复杂和准确的心肌运动模型的训练和验证，从而大大减少对手动注释的依赖。我们的代码、动画生成示例和其他材料可在项目页面上获取：this https URL。
URL
https://arxiv.org/abs/2507.15894
PDF
-
Motion Segmentation and Egomotion Estimation from Event-Based Normal Flow
2025-07-19 06:11:09Zhiyuan Hua, Dehao Yuan, Cornelia Ferm\"ullerAbstract
This paper introduces a robust framework for motion segmentation and egomotion estimation using event-based normal flow, tailored specifically for neuromorphic vision sensors. In contrast to traditional methods that rely heavily on optical flow or explicit depth estimation, our approach exploits the sparse, high-temporal-resolution event data and incorporates geometric constraints between normal flow, scene structure, and inertial measurements. The proposed optimization-based pipeline iteratively performs event over-segmentation, isolates independently moving objects via residual analysis, and refines segmentations using hierarchical clustering informed by motion similarity and temporal consistency. Experimental results on the EVIMO2v2 dataset validate that our method achieves accurate segmentation and translational motion estimation without requiring full optical flow computation. This approach demonstrates significant advantages at object boundaries and offers considerable potential for scalable, real-time robotic and navigation applications.
Abstract (translated)
本文介绍了一个基于事件的法线流进行运动分割和自运动估计的稳健框架，专门针对神经形态视觉传感器。与依赖于光流或显式深度估计的传统方法不同，我们的方法利用了稀疏、高时间分辨率的事件数据，并结合了法线流、场景结构和惯性测量之间的几何约束条件。所提出的优化管道迭代地执行事件过度分割，通过残差分析隔离独立移动的对象，并使用基于运动相似性和时间一致性信息的分层聚类来细化分割结果。在EVIMO2v2数据集上的实验结果验证了我们的方法能够实现准确的分割和无须完整光流计算的平移运动估计。此方法在对象边界处表现出了显著优势，为可扩展、实时的机器人和导航应用提供了巨大的潜力。
URL
https://arxiv.org/abs/2507.14500
PDF
-
DUSTrack: Semi-automated point tracking in ultrasound videos
2025-07-18 21:22:39Praneeth Namburi, Roger Pallar\`es-L\'opez, Jessica Rosendorf, Duarte Folgado, Brian W. AnthonyAbstract
Ultrasound technology enables safe, non-invasive imaging of dynamic tissue behavior, making it a valuable tool in medicine, biomechanics, and sports science. However, accurately tracking tissue motion in B-mode ultrasound remains challenging due to speckle noise, low edge contrast, and out-of-plane movement. These challenges complicate the task of tracking anatomical landmarks over time, which is essential for quantifying tissue dynamics in many clinical and research applications. This manuscript introduces DUSTrack (Deep learning and optical flow-based toolkit for UltraSound Tracking), a semi-automated framework for tracking arbitrary points in B-mode ultrasound videos. We combine deep learning with optical flow to deliver high-quality and robust tracking across diverse anatomical structures and motion patterns. The toolkit includes a graphical user interface that streamlines the generation of high-quality training data and supports iterative model refinement. It also implements a novel optical-flow-based filtering technique that reduces high-frequency frame-to-frame noise while preserving rapid tissue motion. DUSTrack demonstrates superior accuracy compared to contemporary zero-shot point trackers and performs on par with specialized methods, establishing its potential as a general and foundational tool for clinical and biomechanical research. We demonstrate DUSTrack's versatility through three use cases: cardiac wall motion tracking in echocardiograms, muscle deformation analysis during reaching tasks, and fascicle tracking during ankle plantarflexion. As an open-source solution, DUSTrack offers a powerful, flexible framework for point tracking to quantify tissue motion from ultrasound videos. DUSTrack is available at this https URL.
Abstract (translated)
超声技术能够实现安全、无创的动态组织成像，因此在医学、生物力学和体育科学领域中具有重要价值。然而，在B模式超声图像中准确追踪组织运动仍然面临挑战，如斑点噪声（speckle noise）、低边缘对比度以及平面外移动等，这些因素使得随时间追踪解剖标志变得复杂，这对于许多临床和研究应用中的量化组织动力学至关重要。 本文介绍了一种基于深度学习和光流的超声跟踪工具包DUSTrack，这是一种半自动框架，用于在B模式超声视频中追踪任意点。我们结合了深度学习与光流技术，以实现跨不同解剖结构和运动模式的高质量、鲁棒性追踪。该工具包包括一个图形用户界面，可以简化高质量训练数据的生成，并支持迭代模型优化。此外，它还实施了一种新颖的基于光流的过滤技术，可以在减少帧间高频噪声的同时保留快速组织移动。 DUSTrack相较于当代零样本点跟踪器展现了更优的准确性，并且在性能上与专门的方法相当，证明了其作为临床和生物力学研究中通用而基础工具的潜力。我们通过三种应用场景展示了DUSTrack的多功能性：心脏壁运动追踪（使用心超图像）、伸手任务期间肌肉变形分析以及脚踝跖屈时肌纤维追踪。 作为一种开源解决方案，DUSTrack为从超声视频量化组织运动提供了强大且灵活的框架。更多关于DUSTrack的信息可以在提供的网址上获取。
URL
https://arxiv.org/abs/2507.14368
PDF
