Abstract
Architecture design is inherently complex. Existing approaches rely on either handcrafted rules, which demand extensive empirical expertise, or automated methods like neural architecture search, which are computationally intensive. In this paper, we introduce DMAO, an architecture optimization framework that employs a grow-and-drop strategy to automatically reallocate parameters during training. This reallocation shifts resources from less-utilized areas to those parts of the model where they are most beneficial. Notably, DMAO only introduces negligible training overhead at a given model complexity. We evaluate DMAO through experiments with CTC on LibriSpeech, TED-LIUM-v2 and Switchboard datasets. The results show that, using the same amount of training resources, our proposed DMAO consistently improves WER by up to 6% relatively across various architectures, model sizes, and datasets. Furthermore, we analyze the pattern of parameter redistribution and uncover insightful findings.
Abstract (translated)
架构设计本质上是复杂的。现有的方法要么依赖于需要大量经验知识的手动编写规则，要么使用像神经架构搜索这样的自动化方法，这些方法在计算上非常耗费资源。在这篇论文中，我们介绍了DMAO（动态模型架构优化）框架，它采用了一种“生长与舍弃”策略，在训练过程中自动重新分配参数。这种重新分配将资源从利用率较低的区域转移到模型中最需要的地方。值得注意的是，DMAO在给定模型复杂度的情况下仅引入了可以忽略不计的训练开销。 我们通过在LibriSpeech、TED-LIUM-v2和Switchboard数据集上进行CTC（连接时序分类）实验来评估DMAO的效果。结果显示，在使用相同的训练资源条件下，我们的提议方法DMAO可以在各种架构、模型大小以及数据集中始终如一地将错误词率(WER)相对提高最多6%。此外，我们还分析了参数重新分配的模式，并发现了具有洞察力的结果。 该研究证明了通过智能优化模型结构和参数分布，能够在不显著增加训练成本的情况下提升性能，为自动架构搜索提供了一种新的有效策略。
URL
https://arxiv.org/abs/2506.13180
