Abstract
Integration of hyperspectral imaging into fluorescence-guided neurosurgery has the potential to improve surgical decision making by providing quantitative fluorescence measurements in real-time. Quantitative fluorescence requires paired spectral data in fluorescence (blue light) and reflectance (white light) mode. Blue and white image acquisition needs to be performed sequentially in a potentially dynamic surgical environment. A key component to the fluorescence quantification process is therefore the ability to find dense cross-modal image correspondences between two hyperspectral images taken under these drastically different lighting conditions. We address this challenge with the introduction of X-RAFT, a Recurrent All-Pairs Field Transforms (RAFT) optical flow model modified for cross-modal inputs. We propose using distinct image encoders for each modality pair, and fine-tune these in a self-supervised manner using flow-cycle-consistency on our neurosurgical hyperspectral data. We show an error reduction of 36.6% across our evaluation metrics when comparing to a naive baseline and 27.83% reduction compared to an existing cross-modal optical flow method (CrossRAFT). Our code and models will be made publicly available after the review process.
Abstract (translated)
将高光谱成像技术集成到荧光引导神经外科手术中，有望通过提供实时的定量荧光测量来改善手术决策。定量荧光需要在荧光（蓝光）和反射率（白光）模式下采集配对的光谱数据。由于手术环境可能具有动态性，在这种截然不同的照明条件下获取蓝色和白色图像必须依次进行。因此，荧光量化过程的一个关键组成部分是能够在两种高光谱图像之间找到密集的跨模态图像对应关系。 为了解决这一挑战，我们引入了X-RAFT模型，这是一种针对跨模态输入进行了修改的递归全对场变换（Recurrent All-Pairs Field Transforms, RAFT）光学流模型。我们建议为每一对模式使用不同的图像编码器，并通过在神经外科高光谱数据上进行自我监督的方式微调这些编码器，以保持流动循环一致性。当与基准方法相比时，我们在评估指标上的误差减少了36.6%，而与现有的跨模态光学流方法（CrossRAFT）相比则减少了27.83%。 我们的代码和模型将在审稿流程后公开发布。
URL
https://arxiv.org/abs/2507.07747
