Abstract
Radiology report generation aims to automatically generate detailed and coherent descriptive reports alongside radiology images. Previous work mainly focused on refining fine-grained image features or leveraging external knowledge. However, the precise alignment of fine-grained image features with corresponding text descriptions has not been considered. This paper presents a novel method called Fine-grained Image-Text Aligner (FITA) to construct fine-grained alignment for image and text features. It has three novel designs: Image Feature Refiner (IFR), Text Feature Refiner (TFR) and Contrastive Aligner (CA). IFR and TFR aim to learn fine-grained image and text features, respectively. We achieve this by leveraging saliency maps to effectively fuse symptoms with corresponding abnormal visual regions, and by utilizing a meticulously constructed triplet set for training. Finally, CA module aligns fine-grained image and text features using contrastive loss for precise alignment. Results show that our method surpasses existing methods on the widely used benchmark
Abstract (translated)
影像报告生成旨在自动生成详细的、连贯的描述性报告，并与影像图像一起显示。之前的工作主要集中在改进细粒度图像特征或利用外部知识。然而，没有考虑细粒度图像特征与相应文本描述的精确对齐。本文提出了一种名为细粒度图像-文本对齐器（FITA）的新方法来构建图像和文本特征的细粒度对齐。它具有三个新颖的设计：图像特征优化器（IFR）、文本特征优化器（TFR）和对比对齐器（CA）。IFR和TFR旨在分别学习细粒度图像和文本特征。我们通过利用置信度图有效地将症状与相应的异常视觉区域相结合，并使用精心构建的三元组集进行训练来实现这一目标。最后，CA模块通过对比损失对细粒度图像和文本特征进行精确对齐。结果表明，我们的方法在广泛使用的基准测试中都超过了现有方法。
URL
https://arxiv.org/abs/2405.00962
