Abstract
This paper focuses on Winograd transformation in 3D convolutional neural networks (CNNs) that are more over-parameterized compared with the 2D version. The over-increasing Winograd parameters not only exacerbate training complexity but also barricade the practical speedups due simply to the volume of element-wise products in the Winograd domain. We attempt to reduce trainable parameters by introducing a low-rank Winograd transformation, a novel training paradigm that decouples the original large tensor into two less storage-required trainable tensors, leading to a significant complexity reduction. Built upon our low-rank Winograd transformation, we take one step ahead by proposing a low-rank oriented sparse granularity that measures column-wise parameter importance. By simply involving the non-zero columns in the element-wise product, our sparse granularity is empowered with the ability to produce a very regular sparse pattern to acquire effectual Winograd speedups. To better understand the efficacy of our method, we perform extensive experiments on 3D CNNs. Results manifest that our low-rank Winograd transformation well outperforms the vanilla Winograd transformation. We also show that our proposed low-rank oriented sparse granularity permits practical Winograd acceleration compared with the vanilla counterpart.
Abstract (translated)
本 paper 关注于在 3D 卷积神经网络(CNNs)中比 2D 版本更加超参数化的 Winograd 变换。超参数的增加不仅加剧了训练的复杂性,还因为 Winograd 域中的元素级积的体积增加而限制了实际速度的提高。我们试图通过引入低秩的 Winograd 变换来减少可训练参数的数量,这是一个新的训练范式,将原始大型向量分离成两个存储要求较少的训练向量,导致巨大的复杂性减少。基于我们低秩的 Winograd 变换,我们向前迈出了一步,提出了一种低秩方向稀疏颗粒度,以衡量列向参数重要性。仅仅涉及元素级积中非零列的稀疏颗粒度,使我们的稀疏颗粒度获得了生成非常稀疏的稀疏模式的能力,从而获得有效的 Winograd 速度提高。为了更好地理解我们的方法的效果,我们对 3D CNN 进行了广泛的实验。结果表明,我们的低秩 Winograd 变换远远超越了传统的 Winograd 变换。我们还表明,与我们传统的解决方案相比,我们提出的低秩方向稀疏颗粒度能够允许实用的 Winograd 加速。
URL
https://arxiv.org/abs/2301.11180
