Paper Reading AI Learner

An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning

2025-01-07 03:50:11
Fatemeh Ghofrani, Pooyan Jamshidi

Abstract

Self-supervised learning (SSL) has significantly advanced image representation learning, yet efficiency challenges persist, particularly with adversarial training. Many SSL methods require extensive epochs to achieve convergence, a demand further amplified in adversarial settings. To address this inefficiency, we revisit the robust EMP-SSL framework, emphasizing the importance of increasing the number of crops per image to accelerate learning. Unlike traditional contrastive learning, robust EMP-SSL leverages multi-crop sampling, integrates an invariance term and regularization, and reduces training epochs, enhancing time efficiency. Evaluated with both standard linear classifiers and multi-patch embedding aggregation, robust EMP-SSL provides new insights into SSL evaluation strategies. Our results show that robust crop-based EMP-SSL not only accelerates convergence but also achieves a superior balance between clean accuracy and adversarial robustness, outperforming multi-crop embedding aggregation. Additionally, we extend this approach with free adversarial training in Multi-Crop SSL, introducing the Cost-Free Adversarial Multi-Crop Self-Supervised Learning (CF-AMC-SSL) method. CF-AMC-SSL demonstrates the effectiveness of free adversarial training in reducing training time while simultaneously improving clean accuracy and adversarial robustness. These findings underscore the potential of CF-AMC-SSL for practical SSL applications. Our code is publicly available at this https URL.

Abstract (translated)

自监督学习(Self-supervised Learning,SSL)在图像表示学习方面取得了显著进展,但仍然存在效率挑战,尤其是在对抗训练中。许多SSL方法需要大量的迭代周期才能达到收敛状态,而在对抗环境中这一需求进一步增加。为了应对这种低效问题,我们重新审视了鲁棒的EMP-SSL框架,并强调通过增加每张图片的采样数量来加速学习过程的重要性。 与传统的对比学习不同,鲁棒的EMP-SSL利用多作物抽样、整合不变性项和正则化,并减少了训练周期,从而提高了时间效率。该方法在标准线性分类器和多补丁嵌入聚合评估中提供了关于SSL评价策略的新见解。我们的实验结果显示,基于农作物的鲁棒EMP-SSL不仅加速了收敛过程,还实现了干净准确性和对抗健壮性的最佳平衡,优于多作物嵌入聚合。 此外,我们还在多作物SSL中引入了一种自由对抗训练的方法——成本免费对抗多作物自监督学习(Cost-Free Adversarial Multi-Crop Self-Supervised Learning, CF-AMC-SSL)。CF-AMC-SSL证明了自由对抗训练在减少训练时间的同时能够提升干净准确性和对抗健壮性。这些发现强调了CF-AMC-SSL在实际SSL应用中的潜力。 我们的代码可以在以下URL公开获取:[此链接](请将“this https URL”替换为实际的URL)。

URL

https://arxiv.org/abs/2501.03507

PDF

https://arxiv.org/pdf/2501.03507.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot