Abstract
Video Diffusion Transformers (DiTs) have been synthesizing high-quality video with high fidelity from given text descriptions involving motion. However, understanding how Video DiTs convert motion words into video remains insufficient. Furthermore, while prior studies on interpretable saliency maps primarily target objects, motion-related behavior in Video DiTs remains largely unexplored. In this paper, we investigate concrete motion features that specify when and which object moves for a given motion concept. First, to spatially localize, we introduce GramCol, which adaptively produces per-frame saliency maps for any text concept, including both motion and non-motion. Second, we propose a motion-feature selection algorithm to obtain an Interpretable Motion-Attentive Map (IMAP) that localizes motion spatially and temporally. Our method discovers concept saliency maps without the need for any gradient calculation or parameter update. Experimentally, our method shows outstanding localization capability on the motion localization task and zero-shot video semantic segmentation, providing interpretable and clearer saliency maps for both motion and non-motion concepts.
Abstract (translated)
视频扩散变换器（DiTs）已经能够从给定的包含运动描述的文本中合成高质量且忠实度高的视频。然而，关于Video DiT如何将运动词汇转化为视频的理解还很不充分。此外，之前的研究主要集中在可解释性的注意力图谱上，这些研究大多针对的是物体本身，而与运动相关的特性在Video DiTs中的表现则很少被探讨。 本文旨在探究对于给定的运动概念而言，在特定时间和哪一对象进行移动的具体运动特征。首先，为了空间定位，我们引入了GramCol技术，它可以为包括运动和非运动在内的任何文本概念自适应地生成每帧注意力图谱。其次，我们提出了一种选择性运动特征算法，以获得可解释的运动注意图（IMAP），该方法可以对运动进行时空定位。 我们的方法能够在不需要计算梯度或参数更新的情况下发现概念性的注意力图谱。实验表明，我们的方法在运动定位任务和零样本视频语义分割上展示了卓越的空间定位能力，并提供了更清晰、更具可解释性的运动和非运动概念的注意力图谱。
URL
https://arxiv.org/abs/2603.02919
