Abstract
Although larger datasets are crucial for training large deep models, the rapid growth of dataset size has brought a significant challenge in terms of considerable training costs, which even results in prohibitive computational expenses. Dataset Distillation becomes a popular technique recently to reduce the dataset size via learning a highly compact set of representative exemplars, where the model trained with these exemplars ideally should have comparable performance with respect to the one trained with the full dataset. While most of existing works upon dataset distillation focus on supervised datasets, we instead aim to distill images and their self-supervisedly trained representations into a distilled set. This procedure, named as Self-Supervised Dataset Distillation, effectively extracts rich information from real datasets, yielding the distilled sets with enhanced cross-architecture generalizability. Particularly, in order to preserve the key characteristics of original dataset more faithfully and compactly, several novel techniques are proposed: 1) we introduce an innovative parameterization upon images and representations via distinct low-dimensional bases, where the base selection for parameterization is experimentally shown to play a crucial role; 2) we tackle the instability induced by the randomness of data augmentation -- a key component in self-supervised learning but being underestimated in the prior work of self-supervised dataset distillation -- by utilizing predetermined augmentations; 3) we further leverage a lightweight network to model the connections among the representations of augmented views from the same image, leading to more compact pairs of distillation. Extensive experiments conducted on various datasets validate the superiority of our approach in terms of distillation efficiency, cross-architecture generalization, and transfer learning performance.
Abstract (translated)
尽管更大的数据集对于训练大型深度模型至关重要，但数据集规模的快速膨胀也带来了显著的成本挑战，这甚至导致了高昂的计算费用。为此，数据集蒸馏作为一种流行的技术被提出，旨在通过学习一组具有代表性的样本（称为示例）来减少数据集大小，在这种情况下，使用这些示例训练得到的模型应该与使用完整数据集训练出来的模型性能相当。然而，大多数现有的数据集蒸馏工作主要集中在监督学习的数据集上，我们则试图将图像及其自监督训练表示转化为一个浓缩后的集合。这一过程被称为自监督数据集蒸馏（Self-Supervised Dataset Distillation），它能够从真实世界的数据集中有效提取丰富信息，并生成具有增强跨架构泛化能力的浓缩集合。 为了更忠实且紧凑地保留原始数据集的关键特性，我们提出了几种新颖的技术： 1. 我们通过不同的低维基底引入了一种创新参数化的图像和表示方法，实验表明这种基础选择对参数化起着至关重要的作用； 2. 为了解决由数据增强的随机性所导致的不稳定问题——这是一个自监督学习中的关键组成部分，但在之前的自监督数据集蒸馏工作中被低估了——我们利用预定的数据增强策略来应对这一挑战； 3. 我们进一步通过一个轻量级网络建模来自同一图像的不同视图表示之间的连接关系，从而生成更为紧凑的蒸馏对。 在多个数据集上进行的广泛实验验证了我们在蒸馏效率、跨架构泛化能力以及迁移学习性能方面的优势。
URL
https://arxiv.org/abs/2507.21455
