Abstract
Latent Diffusion Models have shown remarkable results in text-guided image synthesis in recent years. In the domain of natural (RGB) images, recent works have shown that such models can be adapted to various vision-language downstream tasks with little to no supervision involved. On the contrary, text-to-image Latent Diffusion Models remain relatively underexplored in the field of medical imaging, primarily due to limited data availability (e.g., due to privacy concerns). In this work, focusing on the chest X-ray modality, we first demonstrate that a standard text-conditioned Latent Diffusion Model has not learned to align clinically relevant information in free-text radiology reports with the corresponding areas of the given scan. Then, to alleviate this issue, we propose a fine-tuning framework to improve multi-modal alignment in a pre-trained model such that it can be efficiently repurposed for downstream tasks such as phrase grounding. Our method sets a new state-of-the-art on a standard benchmark dataset (MS-CXR), while also exhibiting robust performance on out-of-distribution data (VinDr-CXR). Our code will be made publicly available.
Abstract (translated)
近年来，潜在扩散模型（Latent Diffusion Models）在文本引导的图像合成领域取得了显著成果。在自然图像（RGB）领域，最近的研究表明，这类模型可以在几乎没有或不需要监督的情况下适应各种视觉-语言下游任务。然而，在医学成像领域，基于文本到图像的潜在扩散模型研究相对较少，主要原因是可用数据有限（例如，由于隐私问题）。在这项工作中，我们专注于胸部X光影像模式，并首先展示了标准的条件文本的潜在扩散模型未能学会将放射学报告中的临床相关信息与给定扫描中的相应区域对齐。为了解决这个问题，我们提出了一种微调框架，以改进预训练模型中的多模态对齐，使其可以高效地重新应用于下游任务（如短语定位）。我们的方法在标准基准数据集（MS-CXR）上建立了新的最先进水平，并且在分布外数据（VinDr-CXR）上也表现出稳健的性能。我们将公开发布代码。
URL
https://arxiv.org/abs/2506.10633
