Abstract
Diffusion models are pivotal for generating high-quality images and videos. Inspired by the success of OpenAI's Sora, the backbone of diffusion models is evolving from U-Net to Transformer, known as Diffusion Transformers (DiTs). However, generating high-quality content necessitates longer sequence lengths, exponentially increasing the computation required for the attention mechanism, and escalating DiTs inference latency. Parallel inference is essential for real-time DiTs deployments, but relying on a single parallel method is impractical due to poor scalability at large scales. This paper introduces xDiT, a comprehensive parallel inference engine for DiTs. After thoroughly investigating existing DiTs parallel approaches, xDiT chooses Sequence Parallel (SP) and PipeFusion, a novel Patch-level Pipeline Parallel method, as intra-image parallel strategies, alongside CFG parallel for inter-image parallelism. xDiT can flexibly combine these parallel approaches in a hybrid manner, offering a robust and scalable solution. Experimental results on two 8xL40 GPUs (PCIe) nodes interconnected by Ethernet and an 8xA100 (NVLink) node showcase xDiT's exceptional scalability across five state-of-the-art DiTs. Notably, we are the first to demonstrate DiTs scalability on Ethernet-connected GPU clusters. xDiT is available at this https URL.
Abstract (translated)
扩散模型对于生成高质量的图像和视频至关重要。受到OpenAI的Sora成功的启发，扩散模型的核心正在从U-Net向Transformer演变，这被称为扩散Transformer（DiTs）。然而，生成高质量内容需要更长的序列长度，这会导致注意力机制所需的计算呈指数级增长，并增加DiTs推理延迟。并行推理对于实时部署DiTs至关重要，但仅依赖单一并行方法在大规模情况下是不切实际的，因为其可扩展性较差。本文介绍了一种名为xDiT的全面并行推理引擎，专为DiTs设计。通过深入研究现有的DiTs并行方法，xDiT选择了序列并行（SP）和一种新型的Patch级别的流水线并行方法PipeFusion作为图像内的并行策略，并且还采用了CFG并行来实现图像间的并行处理。xDiT能够灵活地以混合方式组合这些并行方法，提供一个强大而可扩展的解决方案。实验结果在两个通过以太网互连、各配备8xL40 GPU（PCIe）节点以及一个配备8xA100（NVLink）节点上显示了xDiT在五种最先进的DiTs模型上的卓越扩展能力。值得注意的是，我们是第一个展示DiTs在以太网连接的GPU集群中可扩展性的研究团队。xDiT可以在提供的链接https://...获取。
URL
https://arxiv.org/abs/2411.01738
