Paper Reading AI Learner

Towards Higher Ranks via Adversarial Weight Pruning

2023-11-29 10:04:39
Yuchuan Tian, Hanting Chen, Tianyu Guo, Chao Xu, Yunhe Wang

Abstract

Convolutional Neural Networks (CNNs) are hard to deploy on edge devices due to its high computation and storage complexities. As a common practice for model compression, network pruning consists of two major categories: unstructured and structured pruning, where unstructured pruning constantly performs better. However, unstructured pruning presents a structured pattern at high pruning rates, which limits its performance. To this end, we propose a Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an adversarial manner. In each step, we minimize the low-rank approximation error for the weight matrices using singular value decomposition, and maximize their distance by pushing the weight matrices away from its low rank approximation. This rank-based optimization objective guides sparse weights towards a high-rank topology. The proposed method is conducted in a gradual pruning fashion to stabilize the change of rank during training. Experimental results on various datasets and different tasks demonstrate the effectiveness of our algorithm in high sparsity. The proposed RPG outperforms the state-of-the-art performance by 1.13% top-1 accuracy on ImageNet in ResNet-50 with 98% sparsity. The codes are available at this https URL and this https URL.

Abstract (translated)

卷积神经网络(CNNs)在边缘设备上部署困难,因为它们具有高计算和存储复杂性。作为一种常见的模型压缩实践,网络剪枝包括两大类:无结构和有结构剪枝,其中无结构剪枝在高剪枝率下始终表现更好。然而,无结构剪枝在剪枝率较高时呈现出一维结构模式,这限制了其性能。为此,我们提出了一种基于排名的剪枝方法,以在对抗方式下维护稀疏权重的等级。在每一步中,我们通过稀疏值分解最小化权重矩阵的低秩近似误差,并通过将权重矩阵推离其低秩近似来最大化它们的距离。基于排名的优化目标引导稀疏权重走向高维拓扑结构。在训练过程中,我们采用逐步剪枝方式来稳定秩的变化。各种数据集和不同任务上的实验结果表明,我们的算法在高稀疏情况下具有很好的效果。与最先进的性能相比,我们提出的RPG方法在ImageNet上ResNet-50的98%稀疏率上提高了1.13%的top-1准确率。代码可在此https://url和https://url处获取。

URL

https://arxiv.org/abs/2311.17493

PDF

https://arxiv.org/pdf/2311.17493.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot