Abstract
To continuously enhance model adaptability in surgical video scene parsing, recent studies incrementally update it to progressively learn to segment an increasing number of surgical instruments over time. However, prior works constantly overlooked the potential of positive forward knowledge transfer, i.e., how past knowledge could help learn new classes, and positive backward knowledge transfer, i.e., how learning new classes could help refine past knowledge. In this paper, we propose a self-reflection hierarchical prompt framework that unlocks the power of positive forward and backward knowledge transfer in class incremental segmentation, aiming to proficiently learn new instruments, improve existing skills of regular instruments, and avoid catastrophic forgetting of old instruments. Our framework is built on a frozen, pre-trained model that adaptively appends instrument-aware prompts for new classes throughout training episodes. To enable positive forward knowledge transfer, we organize instrument prompts into a hierarchical prompt parsing tree with the instrument-shared prompt partition as the root node, n-part-shared prompt partitions as intermediate nodes and instrument-distinct prompt partitions as leaf nodes, to expose the reusable historical knowledge for new classes to simplify their learning. Conversely, to encourage positive backward knowledge transfer, we conduct self-reflection refining on existing knowledge by directed-weighted graph propagation, examining the knowledge associations recorded in the tree to improve its representativeness without causing catastrophic forgetting. Our framework is applicable to both CNN-based models and advanced transformer-based foundation models, yielding more than 5% and 11% improvements over the competing methods on two public benchmarks respectively.
Abstract (translated)
为持续增强手术视频场景解析中的模型适应性，近期研究通过增量更新使其逐步学会随时间推移分割越来越多的手术器械。然而，先前工作始终忽视了正向知识迁移的潜力（即过往知识如何帮助学习新类别）与反向知识迁移的潜力（即学习新类别如何帮助优化过往知识）。本文提出一种自反思层次提示框架，释放类别增量分割中正向与反向知识迁移的效能，旨在熟练掌握新器械、提升常规器械的现有技能，并避免旧器械的灾难性遗忘。该框架基于冻结的预训练模型构建，能在训练过程中自适应地为新类别追加器械感知提示。为实现正向知识迁移，我们将器械提示组织为层次化提示解析树：以器械共享提示分区为根节点，以n部分共享提示分区为中间节点，以器械专属提示分区为叶节点，从而为新类别暴露可复用的历史知识以简化其学习过程。反之，为促进反向知识迁移，我们通过有向加权图传播对现有知识进行自反思精炼，检验树中记录的知识关联以提升其表征性，同时避免灾难性遗忘。该框架既适用于基于CNN的模型，也适用于先进的基于Transformer的基础模型，在两个公开基准测试上分别较竞争方法取得超过5%和11%的性能提升。
URL
https://arxiv.org/abs/2604.02877
