Abstract
The demand for efficient natural language processing (NLP) systems has led to the development of lightweight language models. Previous work in this area has primarily focused on manual design or training-based neural architecture search (NAS) methods. Recently, zero-shot NAS methods have been proposed for evaluating language models without the need for training. However, prevailing approaches to zero-shot NAS often face challenges such as biased evaluation metrics and computational inefficiencies. In this paper, we introduce weight-weighted PCA (W-PCA), a novel zero-shot NAS method specifically tailored for lightweight language models. Our approach utilizes two evaluation proxies: the parameter count and the number of principal components with cumulative contribution exceeding $\eta$ in the feed-forward neural (FFN) layer. Additionally, by eliminating the need for gradient computations, we optimize the evaluation time, thus enhancing the efficiency of designing and evaluating lightweight language models. We conduct a comparative analysis on the GLUE and SQuAD datasets to evaluate our approach. The results demonstrate that our method significantly reduces training time compared to one-shot NAS methods and achieves higher scores in the testing phase compared to previous state-of-the-art training-based methods. Furthermore, we perform ranking evaluations on a dataset sampled from the FlexiBERT search space. Our approach exhibits superior ranking correlation and further reduces solving time compared to other zero-shot NAS methods that require gradient computation.
Abstract (translated)
对于高效的自然语言处理（NLP）系统的市场需求，已经开发出了一系列轻量级的语言模型。在此领域的先前研究主要集中在手动设计或基于训练的神经架构搜索（NAS）方法上。最近，提出了一些零样本NAS方法用于在无需进行培训的情况下评估语言模型。然而，流行的零样本NAS方法往往面临着如评估标准偏见和计算效率低下的挑战。在这篇论文中，我们介绍了一种新颖的零样本NAS方法——权重量化PCA（W-PCA），该方法特别适用于轻量级的语言模型。我们的方法采用了两个评估代理：参数计数以及在前馈神经网络层中累计贡献超过$\eta$的主要成分数量。此外，通过消除对梯度计算的需求，我们优化了评估时间，从而提高了设计和评估轻量级语言模型的效率。 我们在GLUE和SQuAD数据集上进行了对比分析以评估我们的方法。结果显示，与一次样本NAS方法相比，我们的方法显著减少了训练时间；而在测试阶段，与先前基于训练的最佳方法相比，我们取得了更高的分数。此外，我们在从FlexiBERT搜索空间中抽取的数据库上执行了排名评估。我们的方法在排名相关性方面表现出色，并进一步降低了求解时间，优于其他需要梯度计算的零样本NAS方法。
URL
https://arxiv.org/abs/2504.15983
