Abstract
Surgical video datasets are essential for scene understanding, enabling procedural modeling and intra-operative support. However, these datasets are often heavily imbalanced, with rare actions and tools under-represented, which limits the robustness of downstream models. We address this challenge with $SurgiFlowVid$, a sparse and controllable video diffusion framework for generating surgical videos of under-represented classes. Our approach introduces a dual-prediction diffusion module that jointly denoises RGB frames and optical flow, providing temporal inductive biases to improve motion modeling from limited samples. In addition, a sparse visual encoder conditions the generation process on lightweight signals (e.g., sparse segmentation masks or RGB frames), enabling controllability without dense annotations. We validate our approach on three surgical datasets across tasks including action recognition, tool presence detection, and laparoscope motion prediction. Synthetic data generated by our method yields consistent gains of 10-20% over competitive baselines, establishing $SurgiFlowVid$ as a promising strategy to mitigate data imbalance and advance surgical video understanding methods.
Abstract (translated)
外科手术视频数据集对于场景理解至关重要，可以促进程序建模和术中支持。然而，这些数据集往往不平衡，罕见的动作和工具被严重低估，这限制了下游模型的鲁棒性。我们通过引入$SurgiFlowVid$来解决这一挑战，这是一个稀疏且可控的视频扩散框架，用于生成代表性不足类别的手术视频。我们的方法介绍了一个双预测扩散模块，该模块同时对RGB帧和光流进行去噪处理，提供时间归纳偏差以从有限样本中改进运动建模。此外，一个稀疏视觉编码器根据轻量级信号（例如稀疏分割掩码或RGB帧）来调节生成过程，从而在没有密集注释的情况下实现可控性。 我们在三个手术数据集上验证了我们的方法，涵盖了包括动作识别、工具存在检测和腹腔镜运动预测在内的任务。我们通过实验表明，由我们的方法生成的合成数据相对于竞争基线带来了10-20%的一致性能提升，从而将$SurgiFlowVid$确立为一种有前景的方法来缓解数据不平衡并推进手术视频理解技术的进步。
URL
https://arxiv.org/abs/2510.07345
