NAS
-
Modular Machine Learning: An Indispensable Path towards New-Generation Large Language Models
2025-04-28 17:42:02Xin Wang, Haoyang Li, Zeyang Zhang, Haibo Chen, Wenwu ZhuAbstract
Large language models (LLMs) have dramatically advanced machine learning research including natural language processing, computer vision, data mining, etc., yet they still exhibit critical limitations in reasoning, factual consistency, and interpretability. In this paper, we introduce a novel learning paradigm -- Modular Machine Learning (MML) -- as an essential approach toward new-generation LLMs. MML decomposes the complex structure of LLMs into three interdependent components: modular representation, modular model, and modular reasoning, aiming to enhance LLMs' capability of counterfactual reasoning, mitigating hallucinations, as well as promoting fairness, safety, and transparency. Specifically, the proposed MML paradigm can: i) clarify the internal working mechanism of LLMs through the disentanglement of semantic components; ii) allow for flexible and task-adaptive model design; iii) enable interpretable and logic-driven decision-making process. We present a feasible implementation of MML-based LLMs via leveraging advanced techniques such as disentangled representation learning, neural architecture search and neuro-symbolic learning. We critically identify key challenges, such as the integration of continuous neural and discrete symbolic processes, joint optimization, and computational scalability, present promising future research directions that deserve further exploration. Ultimately, the integration of the MML paradigm with LLMs has the potential to bridge the gap between statistical (deep) learning and formal (logical) reasoning, thereby paving the way for robust, adaptable, and trustworthy AI systems across a wide range of real-world applications.
Abstract (translated)
大型语言模型（LLMs）已经在机器学习研究中取得了显著进展，包括自然语言处理、计算机视觉和数据挖掘等领域，但它们在推理能力、事实一致性以及可解释性方面仍然存在关键性的局限。本文介绍了一种新颖的学习范式——模块化机器学习（Modular Machine Learning, MML），作为下一代LLMs的重要方法。MML将复杂结构的LLMs分解为三个相互依存的部分：模块表示、模块模型和模块推理，旨在增强LLMs在反事实推理能力上的表现，减少幻觉现象，并促进公平性、安全性和透明度。具体而言，所提出的MML范式可以： 1) 通过分离语义组件来阐明LLMs的内部工作机制； 2) 允许灵活且任务适应性的模型设计； 3) 实现可解释和逻辑驱动的决策过程。 我们通过利用诸如解耦表示学习、神经架构搜索以及符号-神经混合学习等先进技术，提出了基于MML范式的LLMs可行实现方法。同时，我们也批判性地指出了关键挑战，如连续神经处理与离散符号处理的融合、联合优化及计算可扩展性等问题，并提出了一些值得进一步探索的未来研究方向。最终，将MML范式集成到LLMs中有可能弥合统计（深度）学习和形式（逻辑）推理之间的差距，从而为各种现实世界应用中的稳健、适应性和可信的人工智能系统铺平道路。
URL
https://arxiv.org/abs/2504.20020
PDF
-
DNAD: Differentiable Neural Architecture Distillation
2025-04-25 08:49:31Xuan Rao, Bo Zhao, Derong LiuAbstract
To meet the demand for designing efficient neural networks with appropriate trade-offs between model performance (e.g., classification accuracy) and computational complexity, the differentiable neural architecture distillation (DNAD) algorithm is developed based on two cores, namely search by deleting and search by imitating. Primarily, to derive neural architectures in a space where cells of the same type no longer share the same topology, the super-network progressive shrinking (SNPS) algorithm is developed based on the framework of differentiable architecture search (DARTS), i.e., search by deleting. Unlike conventional DARTS-based approaches which yield neural architectures with simple structures and derive only one architecture during the search procedure, SNPS is able to derive a Pareto-optimal set of architectures with flexible structures by forcing the dynamic super-network shrink from a dense structure to a sparse one progressively. Furthermore, since knowledge distillation (KD) has shown great effectiveness to train a compact network with the assistance of an over-parameterized model, we integrate SNPS with KD to formulate the DNAD algorithm, i.e., search by imitating. By minimizing behavioral differences between the super-network and teacher network, the over-fitting of one-level DARTS is avoided and well-performed neural architectures are derived. Experiments on CIFAR-10 and ImageNet classification tasks demonstrate that both SNPS and DNAD are able to derive a set of architectures which achieve similar or lower error rates with fewer parameters and FLOPs. Particularly, DNAD achieves the top-1 error rate of 23.7% on ImageNet classification with a model of 6.0M parameters and 598M FLOPs, which outperforms most DARTS-based methods.
Abstract (translated)
为了满足设计高效神经网络的需求，这些神经网络在模型性能（如分类准确率）和计算复杂性之间做出适当权衡，可区分神经架构蒸馏（DNAD）算法基于两种核心方法开发：通过删除搜索和通过模仿搜索。主要地，为了推导出在一个空间中的神经架构，在这个空间中相同类型的单元不再具有相同的拓扑结构，超级网络逐步缩小(SNPS)算法根据可微架构搜索(DARTS)框架进行开发，也就是通过删除来搜索的方法。不同于常规基于DARTS的方法只能生成简单结构的神经架构，并且在搜索过程中仅生成一个架构，SNPS能够通过强制动态超级网络从密集结构逐步缩减到稀疏结构的方式推导出一组具有灵活结构的帕累托最优架构。 此外，由于知识蒸馏(KD)展示了利用过度参数化模型来训练紧凑型网络的有效性，我们将SNPS与KD相结合以制定DNAD算法，即通过模仿进行搜索的方法。通过最小化超级网络和教师网络之间的行为差异，可以避免单层DARTS的过拟合并推导出性能良好的神经架构。在CIFAR-10和ImageNet分类任务上的实验表明，无论是SNPS还是DNAD都能够推导出一组实现相似或更低错误率、同时具有更少参数和FLOPs的架构。 特别地，DNAD在使用6.0M参数和598M FLOPs模型的情况下，在ImageNet分类上达到了23.7%的顶级1误差率，超过了大多数基于DARTS的方法。
URL
https://arxiv.org/abs/2504.20080
PDF
-
FX-DARTS: Designing Topology-unconstrained Architectures with Differentiable Architecture Search and Entropy-based Super-network Shrinking
2025-04-25 08:34:29Xuan Rao, Bo Zhao, Derong Liu, Cesare AlippiAbstract
Strong priors are imposed on the search space of Differentiable Architecture Search (DARTS), such that cells of the same type share the same topological structure and each intermediate node retains two operators from distinct nodes. While these priors reduce optimization difficulties and improve the applicability of searched architectures, they hinder the subsequent development of automated machine learning (Auto-ML) and prevent the optimization algorithm from exploring more powerful neural networks through improved architectural flexibility. This paper aims to reduce these prior constraints by eliminating restrictions on cell topology and modifying the discretization mechanism for super-networks. Specifically, the Flexible DARTS (FX-DARTS) method, which leverages an Entropy-based Super-Network Shrinking (ESS) framework, is presented to address the challenges arising from the elimination of prior constraints. Notably, FX-DARTS enables the derivation of neural architectures without strict prior rules while maintaining the stability in the enlarged search space. Experimental results on image classification benchmarks demonstrate that FX-DARTS is capable of exploring a set of neural architectures with competitive trade-offs between performance and computational complexity within a single search procedure.
Abstract (translated)
在不同iable Architecture Search (DARTS) 的搜索空间中强加了先验条件，使得同类型的细胞具有相同的拓扑结构，并且每个中间节点从不同的节点保留两个操作符。尽管这些先验条件减少了优化的难度并提高了所搜寻架构的应用性，但它们限制了自动化机器学习（Auto-ML）的发展，并阻碍了优化算法通过提高架构灵活性来探索更强大的神经网络的能力。本文旨在通过消除对细胞拓扑结构的限制和修改超级网络的离散化机制，减少这些先验约束条件的影响。特别地，本文提出了一种利用基于熵的超级网络缩小（ESS）框架的Flexible DARTS (FX-DARTS) 方法来应对由于取消先前约束所引发的挑战。值得注意的是，FX-DARTS 允许在不严格遵循先验规则的情况下导出神经架构，并且能够在扩大的搜索空间中保持稳定性。实验结果表明，在图像分类基准测试上，FX-DARTS 能够通过单一搜索过程探索一组具有竞争力的性能和计算复杂度权衡的神经网络架构。
URL
https://arxiv.org/abs/2504.20079
PDF
-
Evolution Meets Diffusion: Efficient Neural Architecture Generation
2025-04-24 03:09:04Bingye Zhou, Caiyang YuAbstract
Neural Architecture Search (NAS) has gained widespread attention for its transformative potential in deep learning model design. However, the vast and complex search space of NAS leads to significant computational and time costs. Neural Architecture Generation (NAG) addresses this by reframing NAS as a generation problem, enabling the precise generation of optimal architectures for specific tasks. Despite its promise, mainstream methods like diffusion models face limitations in global search capabilities and are still hindered by high computational and time demands. To overcome these challenges, we propose Evolutionary Diffusion-based Neural Architecture Generation (EDNAG), a novel approach that achieves efficient and training-free architecture generation. EDNAG leverages evolutionary algorithms to simulate the denoising process in diffusion models, using fitness to guide the transition from random Gaussian distributions to optimal architecture distributions. This approach combines the strengths of evolutionary strategies and diffusion models, enabling rapid and effective architecture generation. Extensive experiments demonstrate that EDNAG achieves state-of-the-art (SOTA) performance in architecture optimization, with an improvement in accuracy of up to 10.45%. Furthermore, it eliminates the need for time-consuming training and boosts inference speed by an average of 50 times, showcasing its exceptional efficiency and effectiveness.
Abstract (translated)
神经架构搜索（NAS）因其在深度学习模型设计方面的变革潜力而备受关注。然而，NAS的巨大且复杂的搜索空间导致了显著的计算和时间成本问题。神经架构生成（NAG）通过将NAS重新定义为一个生成问题来解决这一难题，从而能够精确地为特定任务生成最优架构。尽管NAG展现出巨大前景，但主流方法如扩散模型仍然面临着全局搜索能力有限及高计算与时间需求的问题。 为了克服这些挑战，我们提出了进化式扩散神经架构生成（EDNAG），这是一种新型的方法，实现了高效且无需训练的架构生成。EDNAG利用进化算法模拟了扩散模型中的去噪过程，并通过适应度来引导随机高斯分布向最优架构分布转化。该方法结合了进化策略和扩散模型的优势，能够实现快速有效的架构生成。 广泛的实验表明，EDNAG在架构优化方面达到了最先进的性能（SOTA），其准确率提高了高达10.45%。此外，它消除了耗时的训练需求，并将推理速度平均提升了50倍，展示了其卓越的效率和有效性。
URL
https://arxiv.org/abs/2504.17827
PDF
-
Regularizing Differentiable Architecture Search with Smooth Activation
2025-04-22 22:49:38Yanlin Zhou, Mostafa El-Khamy, Kee-Bong SongAbstract
Differentiable Architecture Search (DARTS) is an efficient Neural Architecture Search (NAS) method but suffers from robustness, generalization, and discrepancy issues. Many efforts have been made towards the performance collapse issue caused by skip dominance with various regularization techniques towards operation weights, path weights, noise injection, and super-network redesign. It had become questionable at a certain point if there could exist a better and more elegant way to retract the search to its intended goal -- NAS is a selection problem. In this paper, we undertake a simple but effective approach, named Smooth Activation DARTS (SA-DARTS), to overcome skip dominance and discretization discrepancy challenges. By leveraging a smooth activation function on architecture weights as an auxiliary loss, our SA-DARTS mitigates the unfair advantage of weight-free operations, converging to fanned-out architecture weight values, and can recover the search process from skip-dominance initialization. Through theoretical and empirical analysis, we demonstrate that the SA-DARTS can yield new state-of-the-art (SOTA) results on NAS-Bench-201, classification, and super-resolution. Further, we show that SA-DARTS can help improve the performance of SOTA models with fewer parameters, such as Information Multi-distillation Network on the super-resolution task.
Abstract (translated)
不同可微架构搜索（DARTS）是一种高效的神经网络架构搜索（NAS）方法，但在鲁棒性、泛化能力和差异性方面存在问题。针对由跳跃连接主导导致的性能崩溃问题，已经采取了多种正则化技术来改善操作权重、路径权重、噪声注入和超级网络的设计。在某个阶段，人们开始质疑是否存在一种更好的且更优雅的方法可以使搜索回归到它的目标——即NAS本质上是一个选择问题。在这篇论文中，我们采用了一种简单但有效的方法，命名为平滑激活DARTS（SA-DARTS），以克服跳跃连接主导和离散化差异的挑战。通过在架构权重上应用一个平滑激活函数作为辅助损失，我们的SA-DARTS减轻了无权操作不公正的优势，收敛到扇出式的架构权重值，并能从跳跃主导初始化中恢复搜索过程。通过理论分析和实验验证，我们证明了SA-DARTS可以在NAS-Bench-201、分类任务以及超分辨率上产生新的最先进的（SOTA）结果。此外，我们展示了SA-DARTS可以帮助提升具有较少参数的最新模型在特定任务上的性能，例如信息多重蒸馏网络在超分辨率任务中的应用。
URL
https://arxiv.org/abs/2504.16306
PDF
-
W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models
2025-04-22 15:33:01Shang WangAbstract
The demand for efficient natural language processing (NLP) systems has led to the development of lightweight language models. Previous work in this area has primarily focused on manual design or training-based neural architecture search (NAS) methods. Recently, zero-shot NAS methods have been proposed for evaluating language models without the need for training. However, prevailing approaches to zero-shot NAS often face challenges such as biased evaluation metrics and computational inefficiencies. In this paper, we introduce weight-weighted PCA (W-PCA), a novel zero-shot NAS method specifically tailored for lightweight language models. Our approach utilizes two evaluation proxies: the parameter count and the number of principal components with cumulative contribution exceeding $\eta$ in the feed-forward neural (FFN) layer. Additionally, by eliminating the need for gradient computations, we optimize the evaluation time, thus enhancing the efficiency of designing and evaluating lightweight language models. We conduct a comparative analysis on the GLUE and SQuAD datasets to evaluate our approach. The results demonstrate that our method significantly reduces training time compared to one-shot NAS methods and achieves higher scores in the testing phase compared to previous state-of-the-art training-based methods. Furthermore, we perform ranking evaluations on a dataset sampled from the FlexiBERT search space. Our approach exhibits superior ranking correlation and further reduces solving time compared to other zero-shot NAS methods that require gradient computation.
Abstract (translated)
对于高效的自然语言处理（NLP）系统的市场需求，已经开发出了一系列轻量级的语言模型。在此领域的先前研究主要集中在手动设计或基于训练的神经架构搜索（NAS）方法上。最近，提出了一些零样本NAS方法用于在无需进行培训的情况下评估语言模型。然而，流行的零样本NAS方法往往面临着如评估标准偏见和计算效率低下的挑战。在这篇论文中，我们介绍了一种新颖的零样本NAS方法——权重量化PCA（W-PCA），该方法特别适用于轻量级的语言模型。我们的方法采用了两个评估代理：参数计数以及在前馈神经网络层中累计贡献超过$\eta$的主要成分数量。此外，通过消除对梯度计算的需求，我们优化了评估时间，从而提高了设计和评估轻量级语言模型的效率。 我们在GLUE和SQuAD数据集上进行了对比分析以评估我们的方法。结果显示，与一次样本NAS方法相比，我们的方法显著减少了训练时间；而在测试阶段，与先前基于训练的最佳方法相比，我们取得了更高的分数。此外，我们在从FlexiBERT搜索空间中抽取的数据库上执行了排名评估。我们的方法在排名相关性方面表现出色，并进一步降低了求解时间，优于其他需要梯度计算的零样本NAS方法。
URL
https://arxiv.org/abs/2504.15983
PDF
-
MedNNS: Supernet-based Medical Task-Adaptive Neural Network Search
2025-04-22 13:04:40Lotfi Abdelkrim Mecharbat, Ibrahim Elmakky, Martin Takac, Mohammed YaqubAbstract
Deep learning (DL) has achieved remarkable progress in the field of medical imaging. However, adapting DL models to medical tasks remains a significant challenge, primarily due to two key factors: (1) architecture selection, as different tasks necessitate specialized model designs, and (2) weight initialization, which directly impacts the convergence speed and final performance of the models. Although transfer learning from ImageNet is a widely adopted strategy, its effectiveness is constrained by the substantial differences between natural and medical images. To address these challenges, we introduce Medical Neural Network Search (MedNNS), the first Neural Network Search framework for medical imaging applications. MedNNS jointly optimizes architecture selection and weight initialization by constructing a meta-space that encodes datasets and models based on how well they perform together. We build this space using a Supernetwork-based approach, expanding the model zoo size by 51x times over previous state-of-the-art (SOTA) methods. Moreover, we introduce rank loss and Fréchet Inception Distance (FID) loss into the construction of the space to capture inter-model and inter-dataset relationships, thereby achieving more accurate alignment in the meta-space. Experimental results across multiple datasets demonstrate that MedNNS significantly outperforms both ImageNet pre-trained DL models and SOTA Neural Architecture Search (NAS) methods, achieving an average accuracy improvement of 1.7% across datasets while converging substantially faster. The code and the processed meta-space is available at this https URL.
Abstract (translated)
深度学习（DL）在医学成像领域取得了显著进展。然而，将深度学习模型应用于医疗任务仍面临重大挑战，主要是由于两个关键因素：一是架构选择，不同的任务需要专门设计的模型；二是权重初始化，这直接影响模型的收敛速度和最终性能。虽然从ImageNet进行迁移学习是一种广泛采用的战略，但其有效性受限于自然图像与医学图像之间的巨大差异。为解决这些挑战，我们引入了Medical Neural Network Search（MedNNS），这是首个针对医学成像应用的神经网络搜索框架。MedNNS通过构建一个元空间来联合优化架构选择和权重初始化，该元空间根据数据集和模型共同表现的程度编码它们。 我们采用基于超级网络的方法建立这一空间，并将模型动物园的规模扩大了51倍，超过了以往最先进的（SOTA）方法。此外，我们在空间构造中引入了等级损失和Fréchet inception距离（FID）损失，以捕捉跨模型及跨数据集的关系，从而在元空间内实现更精确的对齐。 多项实验结果显示，MedNNS显著优于ImageNet预训练的深度学习模型以及最先进的神经架构搜索（NAS）方法，在多个数据集中平均准确率提升了1.7%，同时收敛速度更快。该代码和处理后的元空间可在以下链接获取：[提供具体URL]。
URL
https://arxiv.org/abs/2504.15865
PDF
-
Introduction to Quantum Machine Learning and Quantum Architecture Search
Abstract
Recent advancements in quantum computing (QC) and machine learning (ML) have fueled significant research efforts aimed at integrating these two transformative technologies. Quantum machine learning (QML), an emerging interdisciplinary field, leverages quantum principles to enhance the performance of ML algorithms. Concurrently, the exploration of systematic and automated approaches for designing high-performance quantum circuit architectures for QML tasks has gained prominence, as these methods empower researchers outside the quantum computing domain to effectively utilize quantum-enhanced tools. This tutorial will provide an in-depth overview of recent breakthroughs in both areas, highlighting their potential to expand the application landscape of QML across diverse fields.
Abstract (translated)
最近在量子计算（QC）和机器学习（ML）领域的进展激发了大量旨在将这两种变革性技术融合的研究努力。量子机器学习（QML），作为一门新兴的跨学科领域，利用量子原理来提升机器学习算法的表现。同时，为设计高性能量子电路架构以应对QML任务而探索系统化与自动化的方法也日益受到重视，这些方法使非量子计算领域的研究人员也能有效地使用增强型量子工具。本教程将全面介绍这两个领域的最新突破，并强调它们在扩展QML应用领域方面的潜力，尤其是在不同学科中的应用。
URL
https://arxiv.org/abs/2504.16131
PDF
-
Transferrable Surrogates in Expressive Neural Architecture Search Spaces
2025-04-17 14:22:28Shiwen Qin, Gabriela Kadlecov\'a, Martin Pil\'at, Shay B. Cohen, Roman Neruda, Elliot J. Crowley, Jovita Lukasik, Linus EricssonAbstract
Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of architectures to effectively search such spaces. We investigate surrogate model training for improving search in highly expressive NAS search spaces based on context-free grammars. We show that i) surrogate models trained either using zero-cost-proxy metrics and neural graph features (GRAF) or by fine-tuning an off-the-shelf LM have high predictive power for the performance of architectures both within and across datasets, ii) these surrogates can be used to filter out bad architectures when searching on novel datasets, thereby significantly speeding up search and achieving better final performances, and iii) the surrogates can be further used directly as the search objective for huge speed-ups.
Abstract (translated)
神经架构搜索（NAS）在平衡探索表达丰富、广泛的搜索空间以实现架构创新和高效评估架构的需求之间面临着挑战。我们研究了基于上下文无关语法的高度表现力的NAS搜索空间中，通过训练替代模型来改进搜索的方法。我们的发现包括： 1. 使用零成本代理指标或神经图特征（GRAF）训练的替代模型，以及微调现成的语言模型得到的替代模型，对于预测架构在数据集内部和跨不同数据集上的性能具有很高的预测能力。 2. 这些替代模型可以用于过滤新数据集中较差的架构，从而显著加快搜索速度并达到更好的最终性能。 3. 此外，这些替代模型可以直接用作搜索目标，以实现巨大的加速效果。
URL
https://arxiv.org/abs/2504.12971
PDF
-
Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?
2025-04-13 18:36:03Christophe El Zeinaty, Wassim Hamidouche, Glenn Herrou, Daniel Menard, Merouane DebbahAbstract
This paper introduces a novel framework for designing efficient neural network architectures specifically tailored to tiny machine learning (TinyML) platforms. By leveraging large language models (LLMs) for neural architecture search (NAS), a vision transformer (ViT)-based knowledge distillation (KD) strategy, and an explainability module, the approach strikes an optimal balance between accuracy, computational efficiency, and memory usage. The LLM-guided search explores a hierarchical search space, refining candidate architectures through Pareto optimization based on accuracy, multiply-accumulate operations (MACs), and memory metrics. The best-performing architectures are further fine-tuned using logits-based KD with a pre-trained ViT-B/16 model, which enhances generalization without increasing model size. Evaluated on the CIFAR-100 dataset and deployed on an STM32H7 microcontroller (MCU), the three proposed models, LMaNet-Elite, LMaNet-Core, and QwNet-Core, achieve accuracy scores of 74.50%, 74.20% and 73.00%, respectively. All three models surpass current state-of-the-art (SOTA) models, such as MCUNet-in3/in4 (69.62% / 72.86%) and XiNet (72.27%), while maintaining a low computational cost of less than 100 million MACs and adhering to the stringent 320 KB static random-access memory (SRAM) constraint. These results demonstrate the efficiency and performance of the proposed framework for TinyML platforms, underscoring the potential of combining LLM-driven search, Pareto optimization, KD, and explainability to develop accurate, efficient, and interpretable models. This approach opens new possibilities in NAS, enabling the design of efficient architectures specifically suited for TinyML.
Abstract (translated)
这篇论文介绍了一种为小型机器学习（TinyML）平台设计高效神经网络架构的新型框架。该方法通过利用大型语言模型（LLM）进行神经架构搜索（NAS），结合基于视觉变换器（ViT）的知识蒸馏（KD）策略和可解释性模块，实现了准确度、计算效率以及内存使用之间的最佳平衡。由LLM引导的搜索在层次化的搜索空间中探索，通过以精度、乘积累加操作（MACs）和内存指标为基础的帕累托优化来精炼候选架构。最优表现的架构进一步利用预训练的ViT-B/16模型进行基于logits的KD微调，从而提升泛化能力而不增加模型大小。 在CIFAR-100数据集上评估并部署到STM32H7微控制器（MCU）上的三个提出的模型——LMaNet-Elite、LMaNet-Core和QwNet-Core，分别达到了74.50%、74.20% 和 73.00% 的精度。这三个模型均超越了当前最先进的（SOTA）模型，如MCUNet-in3/in4（69.62%/72.86%）和XiNet（72.27%），同时保持了低于1亿MACs的低计算成本，并且遵守严格的320KB静态随机存取内存（SRAM）限制。这些结果展示了该框架在TinyML平台上的效率与性能，强调了结合LLM驱动搜索、帕累托优化、KD和可解释性开发准确、高效及可解释模型的巨大潜力。 这种方法为NAS开辟了新的可能性，使得设计适合TinyML的高效架构成为可能。
URL
https://arxiv.org/abs/2504.09685
PDF
-
Kernel-Level Energy-Efficient Neural Architecture Search for Tabular Dataset
Abstract
Many studies estimate energy consumption using proxy metrics like memory usage, FLOPs, and inference latency, with the assumption that reducing these metrics will also lower energy consumption in neural networks. This paper, however, takes a different approach by introducing an energy-efficient Neural Architecture Search (NAS) method that directly focuses on identifying architectures that minimize energy consumption while maintaining acceptable accuracy. Unlike previous methods that primarily target vision and language tasks, the approach proposed here specifically addresses tabular datasets. Remarkably, the optimal architecture suggested by this method can reduce energy consumption by up to 92% compared to architectures recommended by conventional NAS.
Abstract (translated)
许多研究通过使用代理指标（如内存使用量、FLOPs 和推理延迟）来估算能源消耗，假设减少这些指标也会降低神经网络的能耗。然而，本文采用了一种不同的方法，引入了一种节能型神经架构搜索（NAS）方法，该方法直接专注于识别在保持可接受精度的同时最小化能耗的架构。与以往主要针对视觉和语言任务的方法不同，这里提出的方法特别针对表格数据集。值得注意的是，此方法建议的最佳架构相比传统 NAS 推荐的架构可以减少高达 92% 的能源消耗。
URL
https://arxiv.org/abs/2504.08359
PDF
-
MicroNAS: An Automated Framework for Developing a Fall Detection System
2025-04-10 02:32:47Seyed Mojtaba Mohasel, John Sheppard, Lindsey K. Molina, Richard R. Neptune, Shane R. Wurdeman, Corey A. PewAbstract
This work presents MicroNAS, an automated neural architecture search tool specifically designed to create models optimized for microcontrollers with small memory resources. The ESP32 microcontroller, with 320 KB of memory, is used as the target platform. The artificial intelligence contribution lies in a novel method for optimizing convolutional neural network and gated recurrent unit architectures by considering the memory size of the target microcontroller as a guide. A comparison is made between memory-driven model optimization and traditional two-stage methods, which use pruning, to show the effectiveness of the proposed framework. To demonstrate the engineering application of MicroNAS, a fall detection system (FDS) for lower-limb amputees is developed as a pilot study. A critical challenge in fall detection studies, class imbalance in the dataset, is addressed. The results show that MicroNAS models achieved higher F1-scores than alternative approaches, such as ensemble methods and H2O Automated Machine Learning, presenting a significant step forward in real-time FDS development. Biomechanists using body-worn sensors for activity detection can adopt the open-source code to design machine learning models tailored for microcontroller platforms with limited memory.
Abstract (translated)
这项工作介绍了MicroNAS，这是一种自动神经架构搜索工具，专门设计用于为内存资源有限的微控制器创建优化模型。目标平台是ESP32微控制器，其内存容量为320 KB。人工智能方面的贡献在于提出了一种新颖的方法，通过考虑目标微控制器的内存大小来优化卷积神经网络和门控循环单元架构。研究将基于内存驱动的模型优化方法与传统的两阶段方法（使用剪枝技术）进行了比较，以展示所提框架的有效性。为了展示MicroNAS的实际工程应用价值，开发了一个用于下肢截肢者的跌倒检测系统(FDS)作为试点研究项目。在这一过程中，还解决了跌倒检测研究中数据集不平衡的关键挑战。结果显示，MicroNAS模型的F1分数高于包括集成方法和H2O自动化机器学习在内的其他方法，这表明在实时FDS开发方面取得了重大进展。从事生物力学研究并使用穿戴式传感器进行活动检测的研究人员可以采用开源代码来设计针对内存受限微控制器平台的机器学习模型。
URL
https://arxiv.org/abs/2504.07397
PDF
-
Federated Neural Architecture Search with Model-Agnostic Meta Learning
Abstract
Federated Learning (FL) often struggles with data heterogeneity due to the naturally uneven distribution of user data across devices. Federated Neural Architecture Search (NAS) enables collaborative search for optimal model architectures tailored to heterogeneous data to achieve higher accuracy. However, this process is time-consuming due to extensive search space and retraining. To overcome this, we introduce FedMetaNAS, a framework that integrates meta-learning with NAS within the FL context to expedite the architecture search by pruning the search space and eliminating the retraining stage. Our approach first utilizes the Gumbel-Softmax reparameterization to facilitate relaxation of the mixed operations in the search space. We then refine the local search process by incorporating Model-Agnostic Meta-Learning, where a task-specific learner adapts both weights and architecture parameters (alphas) for individual tasks, while a meta learner adjusts the overall model weights and alphas based on the gradient information from task learners. Following the meta-update, we propose soft pruning using the same trick on search space to gradually sparsify the architecture, ensuring that the performance of the chosen architecture remains robust after pruning which allows for immediate use of the model without retraining. Experimental evaluations demonstrate that FedMetaNAS significantly accelerates the search process by more than 50\% with higher accuracy compared to FedNAS.
Abstract (translated)
联邦学习（FL）通常会因为用户数据在不同设备上的分布不均而面临数据异质性的问题。联邦神经架构搜索（NAS）通过协作搜索，旨在为异构数据找到最优的模型结构以提高准确性。然而，这一过程由于搜索空间广泛以及需要重新训练而耗时较长。为了克服这个问题，我们引入了FedMetaNAS框架，该框架将元学习与NAS集成在联邦学习环境中，通过缩小搜索空间和消除重训阶段来加速架构搜索。 我们的方法首先使用Gumbel-Softmax再参数化技术使搜索空间中的混合操作松弛。随后，我们利用模型无关的元学习来优化局部搜索过程，在这种过程中，特定任务的学习者同时调整权重和架构参数（alphas）以适应个体任务，而元学习者则根据来自各个任务学习者的梯度信息调整整个模型的权重和alphas。 在完成元更新后，我们提出了一种使用同样技巧进行软修剪的方法来逐渐稀疏化架构。这种方法确保了在剪枝后的选定架构性能仍然稳健，并且可以在不重新训练的情况下直接使用该模型。 实验评估表明，FedMetaNAS与FedNAS相比，在提高搜索过程的速度超过50%的同时还能获得更高的准确性。
URL
https://arxiv.org/abs/2504.06457
PDF
-
Comparative Analysis of Unsupervised and Supervised Autoencoders for Nuclei Classification in Clear Cell Renal Cell Carcinoma Images
2025-04-04 03:52:32Fatemeh Javadian, Zahra Aminparast, Johannes Stegmaier, Abin JoseAbstract
This study explores the application of supervised and unsupervised autoencoders (AEs) to automate nuclei classification in clear cell renal cell carcinoma (ccRCC) images, a diagnostic task traditionally reliant on subjective visual grading by pathologists. We evaluate various AE architectures, including standard AEs, contractive AEs (CAEs), and discriminative AEs (DAEs), as well as a classifier-based discriminative AE (CDAE), optimized using the hyperparameter tuning tool Optuna. Bhattacharyya distance is selected from several metrics to assess class separability in the latent space, revealing challenges in distinguishing adjacent grades using unsupervised models. CDAE, integrating a supervised classifier branch, demonstrated superior performance in both latent space separation and classification accuracy. Given that CDAE-CNN achieved notable improvements in classification metrics, affirming the value of supervised learning for class-specific feature extraction, F1 score was incorporated into the tuning process to optimize classification performance. Results show significant improvements in identifying aggressive ccRCC grades by leveraging the classification capability of AE through latent clustering followed by fine-grained classification. Our model outperforms the current state of the art, CHR-Network, across all evaluated metrics. These findings suggest that integrating a classifier branch in AEs, combined with neural architecture search and contrastive learning, enhances grading automation in ccRCC pathology, particularly in detecting aggressive tumor grades, and may improve diagnostic accuracy.
Abstract (translated)
这项研究探讨了监督式和非监督式自动编码器（AEs）在透明细胞肾细胞癌（ccRCC）图像中自动化核分类的应用，这是一个传统上依赖病理学家主观视觉评估的诊断任务。我们评估了各种AE架构，包括标准AE、收缩AE（CAE）、判别性AE（DAE），以及使用超参数优化工具Optuna优化的基于分类器的判别性AE（CDAE）。从多个度量中选择了Bhattacharyya距离来评估潜在空间中的类分离情况，揭示了使用无监督模型区分相邻等级时面临的挑战。集成有监督分类分支的CDAE在潜在空间分离和分类准确性方面表现出色。 鉴于CDAE-CNN在分类指标上取得了显著改进，证实了针对特定类别特征提取的监督学习的价值，F1分数被纳入到优化过程中以提高分类性能。结果表明，通过使用自动编码器进行潜在聚类后紧随其后的精细分类能力，在识别侵袭性ccRCC等级方面有了明显的改善。我们的模型在所有评估指标上都优于当前最先进的CHR-Network。这些发现表明，结合神经架构搜索和对比学习的自动编码器中的分类分支，可以提高ccRCC病理学中分级自动化的能力，特别是在检测侵袭性肿瘤等级方面，并可能提高诊断准确性。
URL
https://arxiv.org/abs/2504.03146
PDF
-
LLM-Guided Evolution: An Autonomous Model Optimization for Object Detection
2025-04-03 05:06:06YiMing Yu, Jason ZuttyAbstract
In machine learning, Neural Architecture Search (NAS) requires domain knowledge of model design and a large amount of trial-and-error to achieve promising performance. Meanwhile, evolutionary algorithms have traditionally relied on fixed rules and pre-defined building blocks. The Large Language Model (LLM)-Guided Evolution (GE) framework transformed this approach by incorporating LLMs to directly modify model source code for image classification algorithms on CIFAR data and intelligently guide mutations and crossovers. A key element of LLM-GE is the "Evolution of Thought" (EoT) technique, which establishes feedback loops, allowing LLMs to refine their decisions iteratively based on how previous operations performed. In this study, we perform NAS for object detection by improving LLM-GE to modify the architecture of You Only Look Once (YOLO) models to enhance performance on the KITTI dataset. Our approach intelligently adjusts the design and settings of YOLO to find the optimal algorithms against objective such as detection accuracy and speed. We show that LLM-GE produced variants with significant performance improvements, such as an increase in Mean Average Precision from 92.5% to 94.5%. This result highlights the flexibility and effectiveness of LLM-GE on real-world challenges, offering a novel paradigm for automated machine learning that combines LLM-driven reasoning with evolutionary strategies.
Abstract (translated)
在机器学习领域，神经架构搜索（NAS）需要模型设计的专业知识和大量的试错才能取得令人满意的表现。与此同时，传统的进化算法依赖于固定的规则和预先定义的构建模块。大型语言模型（LLM）引导演化（GE）框架通过将LLMs融入其中，直接修改图像分类算法在CIFAR数据集上的模型源代码，并智能地指导变异与交叉过程，从而改变了这一方法。LLM-GE的关键元素是“思想进化”(EoT)技术，它建立了反馈循环机制，使LLMs能够根据之前的操作表现来迭代改进其决策。 在这项研究中，我们通过优化LLM-GE框架，在对象检测任务上进行神经架构搜索（NAS），以改进You Only Look Once (YOLO)模型的架构，并在KITTI数据集上提高性能。我们的方法智能地调整和设置YOLO的设计参数，以便找到针对准确性和速度等目标的最佳算法。我们展示了LLM-GE生成了具有显著性能提升的新变体，例如平均精度均值从92.5%提升至94.5%。 这一结果强调了LLM-GE在解决实际问题时的灵活性和有效性，为自动机器学习提供了一种新的范式，结合了由大型语言模型驱动的推理与进化策略。
URL
https://arxiv.org/abs/2504.02280
PDF
-
Spectral Architecture Search for Neural Networks
2025-04-01 15:14:30Gianluca Peri, Lorenzo Giambagli, Lorenzo Chicchi, Duccio FanelliAbstract
Architecture design and optimization are challenging problems in the field of artificial neural networks. Working in this context, we here present SPARCS (SPectral ARchiteCture Search), a novel architecture search protocol which exploits the spectral attributes of the inter-layer transfer matrices. SPARCS allows one to explore the space of possible architectures by spanning continuous and differentiable manifolds, thus enabling for gradient-based optimization algorithms to be eventually employed. With reference to simple benchmark models, we show that the newly proposed method yields a self-emerging architecture with a minimal degree of expressivity to handle the task under investigation and with a reduced parameter count as compared to other viable alternatives.
Abstract (translated)
在人工神经网络领域，架构设计和优化是极具挑战性的问题。在此背景下，我们提出了SPARCS（SPectral ARchiteCture Search），这是一种新颖的架构搜索协议，利用了层间转移矩阵的谱属性。通过探索连续且可微流形的空间，SPARCS使得基于梯度的优化算法能够最终被采用。参考简单的基准模型，我们展示了新提出的方法可以生成一种具有处理任务所需最小表达力和参数量较少的自出现架构，相较于其他可行选项而言更为优越。
URL
https://arxiv.org/abs/2504.00885
PDF
-
Multi-Task Neural Architecture Search Using Architecture Embedding and Transfer Rank
2025-04-01 13:23:11TingJie Zhang, HaiLin LiuAbstract
Multi-task neural architecture search (NAS) enables transferring architectural knowledge among different tasks. However, ranking disorder between the source task and the target task degrades the architecture performance on the downstream task. We propose KTNAS, an evolutionary cross-task NAS algorithm, to enhance transfer efficiency. Our data-agnostic method converts neural architectures into graphs and uses architecture embedding vectors for the subsequent architecture performance prediction. The concept of transfer rank, an instance-based classifier, is introduced into KTNAS to address the performance degradation issue. We verify the search efficiency on NASBench-201 and transferability to various vision tasks on Micro TransNAS-Bench-101. The scalability of our method is demonstrated on DARTs search space including CIFAR-10/100, MNIST/Fashion-MNIST, MedMNIST. Experimental results show that KTNAS outperforms peer multi-task NAS algorithms in search efficiency and downstream task performance. Ablation studies demonstrate the vital importance of transfer rank for transfer performance.
Abstract (translated)
多任务神经架构搜索（NAS）使不同任务之间的网络结构知识得以转移。然而，源任务与目标任务间的排名顺序混乱会降低在下游任务上构建的架构性能。我们提出了KTNAS，一种进化型跨任务NAS算法，以提高迁移效率。我们的数据无关方法将神经网络结构转换为图，并使用架构嵌入向量来进行后续的架构性能预测。为了应对性能下降问题，我们在KTNAS中引入了基于实例分类器的概念“转移秩”。我们已经在NASBench-201上验证了搜索效率，在Micro TransNAS-Bench-101上的各种视觉任务上验证了可迁移性，并在包括CIFAR-10/100、MNIST/Fashion-MNIST和MedMNIST在内的DARTS搜索空间中展示了方法的扩展能力。实验结果显示，KTNAS在搜索效率和下游任务性能方面优于同行中的多任务NAS算法。消融研究表明，转移秩对于迁移性能至关重要。
URL
https://arxiv.org/abs/2504.00772
PDF
-
FACETS: Efficient Once-for-all Object Detection via Constrained Iterative Search
2025-03-27 21:35:02Tony Tran, Bin HuAbstract
Neural Architecture Search (NAS) for deep learning object detection frameworks typically involves multiple modules, each performing distinct tasks. These modules contribute to a vast search space, resulting in searches that can take several GPU hours or even days, depending on the complexity of the search space. This makes joint optimization both challenging and computationally expensive. Furthermore, satisfying target device constraints across modules adds additional complexity to the optimization process. To address these challenges, we propose \textbf{FACETS}, e\textbf{\underline{F}}ficient Once-for-\textbf{\underline{A}}ll Object Detection via \textbf{\underline{C}}onstrained it\textbf{\underline{E}}ra\textbf{\underline{T}}ive\textbf{\underline{S}}earch, a novel unified iterative NAS method that refines the architecture of all modules in a cyclical manner. FACETS leverages feedback from previous iterations, alternating between fixing one module's architecture and optimizing the others. This approach reduces the overall search space while preserving interdependencies among modules and incorporates constraints based on the target device's computational budget. In a controlled comparison against progressive and single-module search strategies, FACETS achieves architectures with up to $4.75\%$ higher accuracy twice as fast as progressive search strategies in earlier stages, while still being able to achieve a global optimum. Moreover, FACETS demonstrates the ability to iteratively refine the search space, producing better performing architectures over time. The refined search space yields candidates with a mean accuracy up to $27\%$ higher than global search and $5\%$ higher than progressive search methods via random sampling.
Abstract (translated)
神经架构搜索（NAS）在深度学习目标检测框架中通常涉及多个执行不同任务的模块。这些模块共同构成了一个巨大的搜索空间，导致搜索过程可能需要数小时甚至几天的时间，具体取决于搜索空间的复杂性。这使得联合优化既具有挑战性又计算成本高昂。此外，在满足目标设备跨模块约束的情况下进行优化增加了额外的复杂度。为了解决这些问题，我们提出了**FACETS**（通过受限迭代搜索实现高效的一次性所有对象检测）这一新颖的统一迭代NAS方法，该方法在循环中对所有模块的架构进行精细化调整。 FACETS 利用前一次迭代的反馈，在每次迭代中交替固定一个模块的架构并优化其他模块。这种方法减少了总体搜索空间，同时保持了各模块之间的依赖关系，并根据目标设备的计算预算引入了约束条件。通过与渐进式和单一模块搜索策略进行受控对比实验，FACETS 在早期阶段实现了最高提升27%平均精度的目标架构，其准确性比渐进式搜索策略高出4.75%，速度却是后者的两倍，并且依然能够达到全局最优解。此外，在整个优化过程中，FACETS 有能力逐步细化搜索空间，从而随着时间的推移生成性能更好的架构。 具体而言，通过随机采样，FACETS 的精细化搜索空间产生的候选架构在平均准确度上比全球搜索方法高出27%，比渐进式搜索方法高5%。
URL
https://arxiv.org/abs/2503.21999
PDF
-
LightSNN: Lightweight Architecture Search for Sparse and Accurate Spiking Neural Networks
2025-03-27 16:38:13Yesmine Abdennadher, Giovanni Perin, Riccardo Mazzieri, Jacopo Pegoraro, Michele RossiAbstract
Spiking Neural Networks (SNNs) are highly regarded for their energy efficiency, inherent activation sparsity, and suitability for real-time processing in edge devices. However, most current SNN methods adopt architectures resembling traditional artificial neural networks (ANNs), leading to suboptimal performance when applied to SNNs. While SNNs excel in energy efficiency, they have been associated with lower accuracy levels than traditional ANNs when utilizing conventional architectures. In response, in this work we present LightSNN, a rapid and efficient Neural Network Architecture Search (NAS) technique specifically tailored for SNNs that autonomously leverages the most suitable architecture, striking a good balance between accuracy and efficiency by enforcing sparsity. Based on the spiking NAS network (SNASNet) framework, a cell-based search space including backward connections is utilized to build our training-free pruning-based NAS mechanism. Our technique assesses diverse spike activation patterns across different data samples using a sparsity-aware Hamming distance fitness evaluation. Thorough experiments are conducted on both static (CIFAR10 and CIFAR100) and neuromorphic datasets (DVS128-Gesture). Our LightSNN model achieves state-of-the-art results on CIFAR10 and CIFAR100, improves performance on DVS128Gesture by 4.49%, and significantly reduces search time, most notably offering a 98x speedup over SNASNet and running 30% faster than the best existing method on DVS128Gesture.
Abstract (translated)
脉冲神经网络（SNNs）因其高能效、固有的激活稀疏性和边缘设备实时处理的适用性而备受推崇。然而，目前大多数SNN方法采用类似于传统人工神经网络（ANNs）的架构，在应用于SNN时导致性能不佳。尽管SNN在能耗方面表现出色，但在使用常规架构的情况下，其准确性往往低于传统的ANNS。为解决这一问题，我们在此提出LightSNN，这是一种专门针对SNN设计的快速高效的神经网络架构搜索（NAS）技术，能够自主选择最合适的架构，在保持高准确性的基础上提高效率，并通过强制稀疏性来实现最佳平衡。基于脉冲NAS网络（SNASNet）框架，我们在包括反向连接的单元搜索空间上构建了一个无需训练的剪枝基NAS机制。我们的方法使用稀疏感知汉明距离评估不同数据样本中的多种脉冲激活模式。我们在静态（CIFAR10和CIFAR100）和神经形态数据集（DVS128-Gesture）上进行了详尽的实验。我们的LightSNN模型在CIFAR10和CIFAR100数据集上取得了最佳结果，在DVS128Gesture数据集上的性能提升了4.49%，并且显著减少了搜索时间，相比SNASNet速度提升高达98倍，并且在DVS128Gesture上比现有最优方法快30%。
URL
https://arxiv.org/abs/2503.21846
PDF
-
Neural Architecture Search by Learning a Hierarchical Search Space
2025-03-27 00:20:13Mehraveh Javan Roshtkhari, Matthew Toews, Marco PedersoliAbstract
Monte-Carlo Tree Search (MCTS) is a powerful tool for many non-differentiable search related problems such as adversarial games. However, the performance of such approach highly depends on the order of the nodes that are considered at each branching of the tree. If the first branches cannot distinguish between promising and deceiving configurations for the final task, the efficiency of the search is exponentially reduced. In Neural Architecture Search (NAS), as only the final architecture matters, the visiting order of the branching can be optimized to improve learning. In this paper, we study the application of MCTS to NAS for image classification. We analyze several sampling methods and branching alternatives for MCTS and propose to learn the branching by hierarchical clustering of architectures based on their similarity. The similarity is measured by the pairwise distance of output vectors of architectures. Extensive experiments on two challenging benchmarks on CIFAR10 and ImageNet show that MCTS, if provided with a good branching hierarchy, can yield promising solutions more efficiently than other approaches for NAS problems.
Abstract (translated)
蒙特卡洛树搜索（MCTS）是一种强大的工具，适用于许多与非微分相关的问题，如对抗性游戏。然而，这种方法的性能很大程度上取决于在每棵树分支时考虑节点的顺序。如果最初的几个分支无法区分对最终任务有前景和误导性的配置，那么搜索效率将呈指数级下降。在神经架构搜索（NAS）中，只有最后的架构才重要，因此可以通过优化分枝访问顺序来改进学习过程。在这篇论文中，我们研究了MCTS在图像分类领域应用于NAS的应用。我们分析了几种采样方法和MCTS分支选择的替代方案，并提出通过基于架构相似性的层次聚类来学习分支选择。相似性是通过架构输出向量之间的成对距离进行度量的。在两个具有挑战性的CIFAR10和ImageNet基准测试上的广泛实验表明，如果为MCTS提供一个良好的分枝层级结构，它能够比其他NAS方法更高效地生成有前景的解决方案。
URL
https://arxiv.org/abs/2503.21061
PDF
