Abstract
Transformer architecture search (TAS) aims to automatically discover efficient vision transformers (ViTs), reducing the need for manual design. Existing TAS methods typically train an over-parameterized network (i.e., a supernet) that encompasses all candidate architectures (i.e., subnets). However, all subnets share the same set of weights, which leads to interference that degrades the smaller subnets severely. We have found that well-trained small subnets can serve as a good foundation for training larger ones. Motivated by this, we propose a progressive training framework, dubbed GrowTAS, that begins with training small subnets and incorporate larger ones gradually. This enables reducing the interference and stabilizing a training process. We also introduce GrowTAS+ that fine-tunes a subset of weights only to further enhance the performance of large subnets. Extensive experiments on ImageNet and several transfer learning benchmarks, including CIFAR-10/100, Flowers, CARS, and INAT-19, demonstrate the effectiveness of our approach over current TAS methods
Abstract (translated)
翻译： Transformer架构搜索（TAS）旨在自动发现高效的视觉变换器（ViTs），减少手动设计的需求。现有的TAS方法通常训练一个过度参数化的网络（即超网），该网络包含所有候选架构（即子网）。然而，所有的子网都共享同一组权重，这导致了干扰，严重影响较小的子网的表现。我们发现，经过良好训练的小型子网可以作为训练更大规模子网的良好基础。受此启发，我们提出了一种渐进式训练框架，称为GrowTAS，该框架从训练小型子网开始，并逐渐加入更大的子网。这有助于减少干扰并稳定训练过程。此外，我们还引入了GrowTAS+，它仅微调部分权重，以进一步提升大规模子网的表现。 在ImageNet和多个迁移学习基准测试（包括CIFAR-10/100、Flowers、Cars和INAT-19）上的广泛实验表明，我们的方法比当前的TAS方法更有效。
URL
https://arxiv.org/abs/2512.12296
