Abstract
Diffusion Transformers (DiTs) are a powerful yet underexplored class of generative models compared to U-Net-based diffusion models. To bridge this gap, we introduce TIDE (Temporal-aware Sparse Autoencoders for Interpretable Diffusion transformErs), a novel framework that enhances temporal reconstruction within DiT activation layers across denoising steps. TIDE employs Sparse Autoencoders (SAEs) with a sparse bottleneck layer to extract interpretable and hierarchical features, revealing that diffusion models inherently learn hierarchical features at multiple levels (e.g., 3D, semantic, class) during generative pre-training. Our approach achieves state-of-the-art reconstruction performance, with a mean squared error (MSE) of 1e-3 and a cosine similarity of 0.97, demonstrating superior accuracy in capturing activation dynamics along the denoising trajectory. Beyond interpretability, we showcase TIDE's potential in downstream applications such as sparse activation-guided image editing and style transfer, enabling improved controllability for generative systems. By providing a comprehensive training and evaluation protocol tailored for DiTs, TIDE contributes to developing more interpretable, transparent, and trustworthy generative models.
Abstract (translated)
扩散变换器（DiT）相较于基于U-Net的扩散模型，是一个强大但尚未充分探索的生成模型类别。为了缩小这一差距，我们引入了TIDE（用于可解释性扩散转换器的时间感知稀疏自编码框架），这是一个新的框架，在扩散过程中增强了时间重建能力，并在整个去噪步骤中改善了DiT激活层的表现。 TIDE利用具有稀疏瓶颈层的稀疏自编码器（SAE）来提取可解释和分层特征，揭示了扩散模型在生成预训练期间内在地学习多层级特征（例如3D、语义、类别）。我们的方法实现了最先进的重建性能，在均方误差（MSE）为1e-3和余弦相似度为0.97的情况下表现出色，证明其在捕捉去噪轨迹上的激活动力学方面具有出色的准确性。 除了可解释性之外，我们展示了TIDE在下游应用中的潜力，如稀疏激活引导的图像编辑和风格转移，使生成系统获得了更好的可控性。通过提供专门针对DiTs的综合训练和评估协议，TIDE有助于开发更可解释、透明和值得信赖的生成模型。
URL
https://arxiv.org/abs/2503.07050
