Abstract
Designing effective neural networks is a cornerstone of deep learning, and Neural Architecture Search (NAS) has emerged as a powerful tool for automating this process. Among the existing NAS approaches, Differentiable Architecture Search (DARTS) has gained prominence for its efficiency and ease of use, inspiring numerous advancements. Since the rise of Vision Transformers (ViT), researchers have applied NAS to explore ViT architectures, often focusing on macro-level search spaces and relying on discrete methods like evolutionary algorithms. While these methods ensure reliability, they face challenges in discovering innovative architectural designs, demand extensive computational resources, and are time-intensive. To address these limitations, we introduce Differentiable Architecture Search for Vision Transformer (DASViT), which bridges the gap in differentiable search for ViTs and uncovers novel designs. Experiments show that DASViT delivers architectures that break traditional Transformer encoder designs, outperform ViT-B/16 on multiple datasets, and achieve superior efficiency with fewer parameters and FLOPs.
Abstract (translated)
设计有效的神经网络是深度学习的核心，而神经架构搜索（NAS）作为自动化这一过程的强有力工具已经崭露头角。在现有的NAS方法中，可微分架构搜索（DARTS）因其效率和易用性而备受推崇，并激发了众多的进步与发展。随着视觉变换器（ViT）的兴起，研究人员开始将NAS应用于探索ViT架构，通常集中在宏观级别的搜索空间上，并依赖于如进化算法之类的离散方法。尽管这些方法能够保证可靠性，但它们在发现创新性的架构设计方面存在挑战，同时需要大量的计算资源和时间。 为了解决这些问题，我们提出了用于视觉变换器（Vision Transformer）的可微分架构搜索（DASViT）。这一方法填补了ViTs中不同iable搜索的空白，并揭示出新颖的设计。实验表明，DASViT能够打破传统的Transformer编码器设计模式，在多个数据集上超越了ViT-B/16模型，并且在使用更少参数和计算量的情况下达到了更高的效率。 通过这种方式，DASViT不仅为ViTs架构搜索提供了一种新的视角，而且还展示了可微分搜索方法的巨大潜力。
URL
https://arxiv.org/abs/2507.13079
