Abstract
Image-to-image translation aims to learn a mapping between a source and a target domain, enabling tasks such as style transfer, appearance transformation, and domain adaptation. In this work, we explore a diffusion-based framework for image-to-image translation by adapting Diffusion Transformers (DiT), which combine the denoising capabilities of diffusion models with the global modeling power of transformers. To guide the translation process, we condition the model on image embeddings extracted from a pre-trained CLIP encoder, allowing for fine-grained and structurally consistent translations without relying on text or class labels. We incorporate both a CLIP similarity loss to enforce semantic consistency and an LPIPS perceptual loss to enhance visual fidelity during training. We validate our approach on two benchmark datasets: face2comics, which translates real human faces to comic-style illustrations, and edges2shoes, which translates edge maps to realistic shoe images. Experimental results demonstrate that DiT, combined with CLIP-based conditioning and perceptual similarity objectives, achieves high-quality, semantically faithful translations, offering a promising alternative to GAN-based models for paired image-to-image translation tasks.
Abstract (translated)
图像到图像翻译旨在学习源域和目标域之间的映射，从而实现如风格转换、外观变换和领域适应等任务。在这项工作中，我们探索了一种基于扩散框架的图像到图像翻译方法，通过调整 Diffusion Transformers（DiT）来实现这一目的。Diffusion Transformers 结合了扩散模型的去噪能力与变压器的整体建模能力。为了引导翻译过程，我们将模型条件化设置为从预训练 CLIP 编码器中提取的图像嵌入，从而使翻译更加精细且结构一致，并且无需依赖文本或类别标签。我们引入了一种基于 CLIP 的相似性损失来强制执行语义一致性以及一种 LPIPS 视觉感知损失以在训练期间增强视觉保真度。 我们在两个基准数据集上验证了我们的方法：face2comics，该数据集将真实的人脸翻译成漫画风格的插图；edges2shoes，该数据集则将边缘地图转换为逼真的鞋类图像。实验结果表明，DiT 结合 CLIP 基础条件化和感知相似性目标，能够实现高质量且语义忠实的翻译，这为配对图像到图像翻译任务提供了一种有前景的GAN模型替代方案。
URL
https://arxiv.org/abs/2505.16001
