NAS
-
ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures
2025-10-06 15:43:36Shiwen Qin, Alexander Auras, Shay B. Cohen, Elliot J. Crowley, Michael Moeller, Linus Ericsson, Jovita LukasikAbstract
Neural architecture search (NAS) automates the design process of high-performing architectures, but remains bottlenecked by expensive performance evaluation. Most existing studies that achieve faster evaluation are mostly tied to cell-based search spaces and graph encodings tailored to those individual search spaces, limiting their flexibility and scalability when applied to more expressive search spaces. In this work, we aim to close the gap of individual search space restrictions and search space dependent network representations. We present ONNX-Bench, a benchmark consisting of a collection of neural networks in a unified format based on ONNX files. ONNX-Bench includes all open-source NAS-bench-based neural networks, resulting in a total size of more than 600k {architecture, accuracy} pairs. This benchmark allows creating a shared neural network representation, ONNX-Net, able to represent any neural architecture using natural language descriptions acting as an input to a performance predictor. This text-based encoding can accommodate arbitrary layer types, operation parameters, and heterogeneous topologies, enabling a single surrogate to generalise across all neural architectures rather than being confined to cell-based search spaces. Experiments show strong zero-shot performance across disparate search spaces using only a small amount of pretraining samples, enabling the unprecedented ability to evaluate any neural network architecture instantly.
Abstract (translated)
神经架构搜索（NAS）自动化了高性能架构的设计过程，但其瓶颈在于昂贵的性能评估。大多数现有的实现更快评估的研究主要依赖于基于单元格的搜索空间和针对特定搜索空间量身定制的图编码，这在应用于更复杂的搜索空间时限制了它们的灵活性和可扩展性。在这项工作中，我们旨在弥合单一搜索空间限制和与搜索空间相关的网络表示之间的差距。为此，我们提出了ONNX-Bench，这是一个由基于ONNX文件的统一格式组成的神经网络集合基准测试。ONNX-Bench包括所有开源NAS-bench基础的神经网络，在总规模上达到了超过60万个{架构，精度}对。这一基准测试使得创建一个共享的神经网络表示——ONNX-Net成为可能，它可以使用自然语言描述作为输入来表示任何神经架构，并且用于性能预测器。这种基于文本的编码可以容纳任意类型的层、操作参数和异构拓扑结构，这使得单个代理能够跨所有神经架构进行泛化，而不是局限于单元格基搜索空间。实验表明，在仅使用少量预训练样本的情况下，该方法在不同的搜索空间中具有强大的零样本性能表现，从而使即时评估任何神经网络架构成为可能，这是前所未有的能力。
URL
https://arxiv.org/abs/2510.04938
PDF
-
AutoMaAS: Self-Evolving Multi-Agent Architecture Search for Large Language Models
2025-10-03 01:57:07Bo Ma, Hang Li, ZeHua Hu, XiaoFan Gui, LuYao Liu, Simon LiuAbstract
Multi-agent systems powered by large language models have demonstrated remarkable capabilities across diverse domains, yet existing automated design approaches seek monolithic solutions that fail to adapt resource allocation based on query complexity and domain requirements. This paper introduces AutoMaAS, a self-evolving multi-agent architecture search framework that leverages neural architecture search principles to automatically discover optimal agent configurations through dynamic operator lifecycle management and automated machine learning techniques. Our approach incorporates four key innovations: (1) automatic operator generation, fusion, and elimination based on performance-cost analysis, (2) dynamic cost-aware optimization with real-time parameter adjustment, (3) online feedback integration for continuous architecture refinement, and (4) enhanced interpretability through decision tracing mechanisms. Extensive experiments across six benchmarks demonstrate that AutoMaAS achieves 1.0-7.1\% performance improvement while reducing inference costs by 3-5\% compared to state-of-the-art methods. The framework shows superior transferability across datasets and LLM backbones, establishing a new paradigm for automated multi-agent system design in the era of large language models.
Abstract (translated)
由大型语言模型驱动的多代理系统在多个领域展示了显著的能力，但现有的自动化设计方法寻求的是单一解决方案，无法根据查询复杂度和特定领域的需要来调整资源分配。本文介绍了AutoMaAS，这是一个自我演化的多代理架构搜索框架，利用神经结构搜索原则通过动态操作生命周期管理和自动机器学习技术自动发现最优的代理配置。 我们的方法包含四个关键创新点： 1. **基于性能-成本分析的自动化操作生成、融合和消除**：这种方法可以优化多个代理之间的工作流程。 2. **实时参数调整的成本意识型动态优化**：根据查询需求，系统能够灵活地调整其运行策略以适应各种场景。 3. **在线反馈整合用于持续架构改进**：通过用户或环境的即时反馈，框架能够在使用过程中不断优化和增强自身性能。 4. **决策追踪机制提升可解释性**：这种方法使得复杂且自动化的架构搜索过程变得更加透明。 在六个基准测试中的广泛实验表明，与当前最佳方法相比，AutoMaAS能够实现1.0%-7.1%的性能改进，并将推理成本降低3-5%。该框架显示出了跨数据集和大型语言模型骨干网络的良好迁移能力，在大规模语言模型时代为自动化多代理系统设计确立了新的范式。 通过这些创新点，AutoMaAS不仅提高了多代理系统的效率和效果，还增强了它们的适应性和可解释性，从而推动了这一领域的发展。
URL
https://arxiv.org/abs/2510.02669
PDF
-
SAGE-LD: Towards Scalable and Generalizable End-to-End Language Diarization via Simulated Data Augmentation
2025-10-01 07:01:33Sangmin Lee, Woongjib Choi, Jihyun Kim, Hong-Goo KangAbstract
In this paper, we present a neural spoken language diarization model that supports an unconstrained span of languages within a single framework. Our approach integrates a learnable query-based architecture grounded in multilingual awareness, with large-scale pretraining on simulated code-switching data. By jointly leveraging these two components, our method overcomes the limitations of conventional approaches in data scarcity and architecture optimization, and generalizes effectively to real-world multilingual settings across diverse environments. Experimental results demonstrate that our approach achieves state-of-the-art performance on several language diarization benchmarks, with a relative performance improvement of 23% to 52% over previous methods. We believe that this work not only advances research in language diarization but also establishes a foundational framework for code-switching speech technologies.
Abstract (translated)
在这篇论文中，我们提出了一种神经口语语言身份识别模型，该模型在一个单一框架内支持无限制的语言跨度。我们的方法结合了一个基于多语种意识的可学习查询架构，并在模拟代码转换数据上进行了大规模预训练。通过联合利用这两个组件，我们的方法克服了传统方法在数据稀缺和架构优化方面的局限性，并能够有效地推广到各种环境下的现实世界多语言设置中。实验结果表明，我们所提出的方法在几种语言身份识别基准测试中实现了最先进的性能，在相对性能方面比以往的方法提高了23%至52%。我们认为这项工作不仅推动了语言身份识别的研究进展，还为代码转换语音技术建立了基础框架。
URL
https://arxiv.org/abs/2510.00582
PDF
-
CoLLM-NAS: Collaborative Large Language Models for Efficient Knowledge-Guided Neural Architecture Search
2025-09-30 10:12:49Zhe Li, Zhiwei Lin, Yongtao WangAbstract
The integration of Large Language Models (LLMs) with Neural Architecture Search (NAS) has introduced new possibilities for automating the design of neural architectures. However, most existing methods face critical limitations, including architectural invalidity, computational inefficiency, and inferior performance compared to traditional NAS. In this work, we present Collaborative LLM-based NAS (CoLLM-NAS), a two-stage NAS framework with knowledge-guided search driven by two complementary LLMs. Specifically, we propose a Navigator LLM to guide search direction and a Generator LLM to synthesize high-quality candidates, with a dedicated Coordinator module to manage their interaction. CoLLM-NAS efficiently guides the search process by combining LLMs' inherent knowledge of structured neural architectures with progressive knowledge from iterative feedback and historical trajectory. Experimental results on ImageNet and NAS-Bench-201 show that CoLLM-NAS surpasses existing NAS methods and conventional search algorithms, achieving new state-of-the-art results. Furthermore, CoLLM-NAS consistently enhances the performance and efficiency of various two-stage NAS methods (e.g., OFA, SPOS, and AutoFormer) across diverse search spaces (e.g., MobileNet, ShuffleNet, and AutoFormer), demonstrating its excellent generalization.
Abstract (translated)
大型语言模型（LLMs）与神经架构搜索（NAS）的结合为自动设计神经网络结构带来了新的可能性。然而，大多数现有方法面临着诸如架构无效性、计算效率低下以及相较于传统NAS性能较差的关键限制。在此项研究中，我们提出了基于协作大语言模型的神经架构搜索框架（CoLLM-NAS），这是一个两阶段的NAS框架，由两个互补的大语言模型驱动的知识引导搜索，并配备了一个专用的协调模块来管理它们之间的互动。 具体来说，我们提出了一种导航器大语言模型来指导搜索方向，并且还设计了一个生成器大语言模型来合成高质量候选结构。此外，还有一个专有的协调模块负责管理和促进这两种大语言模型之间的协作过程。CoLLM-NAS通过结合大语言模型对结构化神经架构的固有知识以及从迭代反馈和历史轨迹中获得的进步性知识，有效地引导搜索流程。 在ImageNet和NAS-Bench-201数据集上的实验结果表明，CoLLM-NAS超越了现有的NAS方法和传统的搜索算法，并取得了新的最先进的性能。此外，在不同的搜索空间（例如MobileNet、ShuffleNet以及AutoFormer）上对各种两阶段NAS方法（如OFA、SPOS和AutoFormer）进行评估时，CoLLM-NAS持续提高了这些方法的性能和效率，展示了其卓越的泛化能力。
URL
https://arxiv.org/abs/2509.26037
PDF
-
CIMNAS: A Joint Framework for Compute-In-Memory-Aware Neural Architecture Search
2025-09-30 06:57:49Olga Krestinskaya, Mohammed E. Fouda, Ahmed Eltawil, Khaled N. SalamaAbstract
To maximize hardware efficiency and performance accuracy in Compute-In-Memory (CIM)-based neural network accelerators for Artificial Intelligence (AI) applications, co-optimizing both software and hardware design parameters is essential. Manual tuning is impractical due to the vast number of parameters and their complex interdependencies. To effectively automate the design and optimization of CIM-based neural network accelerators, hardware-aware neural architecture search (HW-NAS) techniques can be applied. This work introduces CIMNAS, a joint model-quantization-hardware optimization framework for CIM architectures. CIMNAS simultaneously searches across software parameters, quantization policies, and a broad range of hardware parameters, incorporating device-, circuit-, and architecture-level co-optimizations. CIMNAS experiments were conducted over a search space of 9.9x10^85 potential parameter combinations with the MobileNet model as a baseline and RRAM-based CIM architecture. Evaluated on the ImageNet dataset, CIMNAS achieved a reduction in energy-delay-area product (EDAP) ranging from 90.1x to 104.5x, an improvement in TOPS/W between 4.68x and 4.82x, and an enhancement in TOPS/mm^2 from 11.3x to 12.78x relative to various baselines, all while maintaining an accuracy of 73.81%. The adaptability and robustness of CIMNAS are demonstrated by extending the framework to support the SRAM-based ResNet50 architecture, achieving up to an 819.5x reduction in EDAP. Unlike other state-of-the-art methods, CIMNAS achieves EDAP-focused optimization without any accuracy loss, generating diverse software-hardware parameter combinations for high-performance CIM-based neural network designs. The source code of CIMNAS is available at this https URL.
Abstract (translated)
为了在基于计算内存储器（CIM）的神经网络加速器中最大化硬件效率和性能准确性，特别是在人工智能（AI）应用中，同时优化软件和硬件设计参数是必不可少的。由于参数数量庞大且相互关系复杂，手动调优变得不切实际。为有效自动化基于CIM的神经网络加速器的设计与优化过程，可以采用硬件感知型神经架构搜索（HW-NAS）技术。这项工作提出了一个名为CIMNAS的联合模型量化和硬件优化框架，专门针对CIM架构。 CIMNAS同时在软件参数、量化策略以及广泛的硬件参数上进行搜索，涵盖了设备级、电路级与体系结构级别的协同优化。该研究使用MobileNet模型作为基准，并基于RRAM（阻变式存储器）的CIM架构进行了9.9×10^85种潜在参数组合的实验空间探索。在ImageNet数据集上的评估显示，CIMNAS将能源延迟面积乘积（EDAP）减少了90.1到104.5倍，同时在每瓦特TOPS（万亿次操作/秒）性能上提高了4.68至4.82倍，在每平方毫米的TOPS值上提升了11.3至12.78倍，并且保持了73.81%的准确性。通过将框架扩展以支持基于SRAM的ResNet50架构，CIMNAS展示了其适应性和鲁棒性，实现了高达819.5倍的EDAP减少。 与现有的最先进的方法不同，CIMNAS在不损失任何精度的情况下专注于EDAP优化，并生成多种软件和硬件参数组合，用于高性能的基于CIM的神经网络设计。CIMNAS的源代码可在以下网址获取：[此链接]（请将"this https URL"替换为实际提供的URL）。
URL
https://arxiv.org/abs/2509.25862
PDF
-
Deep Learning Approaches with Explainable AI for Differentiating Alzheimer Disease and Mild Cognitive Impairment
2025-09-27 16:17:14Fahad Mostafa, Kannon Hossain, Hafiz KhanAbstract
Early and accurate diagnosis of Alzheimer Disease is critical for effective clinical intervention, particularly in distinguishing it from Mild Cognitive Impairment, a prodromal stage marked by subtle structural changes. In this study, we propose a hybrid deep learning ensemble framework for Alzheimer Disease classification using structural magnetic resonance imaging. Gray and white matter slices are used as inputs to three pretrained convolutional neural networks such as ResNet50, NASNet, and MobileNet, each fine tuned through an end to end process. To further enhance performance, we incorporate a stacked ensemble learning strategy with a meta learner and weighted averaging to optimally combine the base models. Evaluated on the Alzheimer Disease Neuroimaging Initiative dataset, the proposed method achieves state of the art accuracy of 99.21% for Alzheimer Disease vs. Mild Cognitive Impairment and 91.0% for Mild Cognitive Impairment vs. Normal Controls, outperforming conventional transfer learning and baseline ensemble methods. To improve interpretability in image based diagnostics, we integrate Explainable AI techniques by Gradient weighted Class Activation, which generates heatmaps and attribution maps that highlight critical regions in gray and white matter slices, revealing structural biomarkers that influence model decisions. These results highlight the frameworks potential for robust and scalable clinical decision support in neurodegenerative disease diagnostics.
Abstract (translated)
阿尔茨海默病的早期和准确诊断对于有效的临床干预至关重要，特别是在将其与轻度认知障碍（MCI）区分开来时。MCI是一个前驱阶段，以细微的结构变化为标志。在本研究中，我们提出了一种使用结构磁共振成像进行阿尔茨海默病分类的混合深度学习集成框架。灰质和白质切片被用作三个预训练卷积神经网络（如ResNet50、NASNet和MobileNet）的输入，每个模型通过端到端的过程进行了微调以适应特定任务。 为了进一步提高性能，我们采用了一种堆叠式集成学习策略，并结合了一个元学习器以及加权平均法来优化地组合基础模型。在阿尔茨海默病神经成像倡议（ADNI）数据集上评估后，所提出的方法实现了99.21%的最高准确率用于区分阿尔茨海默病与轻度认知障碍，以及91.0%的准确性用于区分轻度认知障碍与正常对照组，超越了传统的迁移学习和基准集成方法。 为了提高基于图像诊断的解释性，我们通过梯度加权类激活（Grad-CAM）技术整合了解释人工智能的方法。这种方法生成热图和归因图，突出显示灰质和白质切片中的关键区域，并揭示影响模型决策的结构生物标志物。 这些结果强调了该框架在神经退行性疾病诊断中提供强大且可扩展临床决策支持方面的潜力。
URL
https://arxiv.org/abs/2510.00048
PDF
-
DPFNAS: Differential Privacy-Enhanced Federated Neural Architecture Search for 6G Edge Intelligence
2025-09-27 01:03:26Yang Lv, Jin Cao, Ben Niu, Zhe Sun, Fengwei Wang, Fenghua Li, Hui LiAbstract
The Sixth-Generation (6G) network envisions pervasive artificial intelligence (AI) as a core goal, enabled by edge intelligence through on-device data utilization. To realize this vision, federated learning (FL) has emerged as a key paradigm for collaborative training across edge devices. However, the sensitivity and heterogeneity of edge data pose key challenges to FL: parameter sharing risks data reconstruction, and a unified global model struggles to adapt to diverse local distributions. In this paper, we propose a novel federated learning framework that integrates personalized differential privacy (DP) and adaptive model design. To protect training data, we leverage sample-level representations for knowledge sharing and apply a personalized DP strategy to resist reconstruction attacks. To ensure distribution-aware adaptation under privacy constraints, we develop a privacy-aware neural architecture search (NAS) algorithm that generates locally customized architectures and hyperparameters. To the best of our knowledge, this is the first personalized DP solution tailored for representation-based FL with theoretical convergence guarantees. Our scheme achieves strong privacy guarantees for training data while significantly outperforming state-of-the-art methods in model performance. Experiments on benchmark datasets such as CIFAR-10 and CIFAR-100 demonstrate that our scheme improves accuracy by 6.82\% over the federated NAS method PerFedRLNAS, while reducing model size to 1/10 and communication cost to 1/20.
Abstract (translated)
第六代（6G）网络将普及的人工智能视为核心目标，通过设备边缘智能和本地数据利用来实现。为了实现这一愿景，联合学习（FL）作为一种关键范式在边缘设备间的协作训练中崭露头角。然而，边缘数据的敏感性和异质性给FL带来了重大挑战：参数共享存在数据重建的风险，而统一的全局模型难以适应多样化的本地分布情况。为此，在本文中我们提出了一种新颖的联邦学习框架，该框架结合了个性化的差分隐私（DP）和自适应模型设计。为了保护训练数据，我们利用样本级别的表示来进行知识分享，并应用个性化DP策略以抵御重建攻击。在确保受隐私约束的情况下进行分布感知型适应方面，我们开发出了一种隐私感知的神经架构搜索（NAS）算法，该算法能够生成本地定制化的架构和超参数。 据我们所知，这是第一个针对基于表示的FL、具备理论收敛保证的个性化DP解决方案。我们的方案在提供强大训练数据隐私保护的同时，其模型性能显著超越了现有的最先进技术方法。我们在CIFAR-10和CIFAR-100等基准数据集上的实验表明：相较于联邦NAS方法PerFedRLNAS，我们的方案能够提高6.82%的准确性，并且将模型大小减少到原来的十分之一，通信成本降低至五分之一。
URL
https://arxiv.org/abs/2509.23030
PDF
-
RAM-NAS: Resource-aware Multiobjective Neural Architecture Search Method for Robot Vision Tasks
2025-09-25 02:42:57Shouren Mao, Minghao Qin, Wei Dong, Huajian Liu, Yongzhuo GaoAbstract
Neural architecture search (NAS) has shown great promise in automatically designing lightweight models. However, conventional approaches are insufficient in training the supernet and pay little attention to actual robot hardware resources. To meet such challenges, we propose RAM-NAS, a resource-aware multi-objective NAS method that focuses on improving the supernet pretrain and resource-awareness on robot hardware devices. We introduce the concept of subnets mutual distillation, which refers to mutually distilling all subnets sampled by the sandwich rule. Additionally, we utilize the Decoupled Knowledge Distillation (DKD) loss to enhance logits distillation performance. To expedite the search process with consideration for hardware resources, we used data from three types of robotic edge hardware to train Latency Surrogate predictors. These predictors facilitated the estimation of hardware inference latency during the search phase, enabling a unified multi-objective evolutionary search to balance model accuracy and latency trade-offs. Our discovered model family, RAM-NAS models, can achieve top-1 accuracy ranging from 76.7% to 81.4% on ImageNet. In addition, the resource-aware multi-objective NAS we employ significantly reduces the model's inference latency on edge hardware for robots. We conducted experiments on downstream tasks to verify the scalability of our methods. The inference time for detection and segmentation is reduced on all three hardware types compared to MobileNetv3-based methods. Our work fills the gap in NAS for robot hardware resource-aware.
Abstract (translated)
神经架构搜索（NAS）在自动设计轻量级模型方面展现了巨大的潜力。然而，传统方法在训练超网和关注实际机器人硬件资源方面存在不足。为应对这些挑战，我们提出了一种新的资源感知多目标NAS方法——RAM-NAS，旨在改进超网的预训练并提高对机器人硬件设备的认识度。 该方法引入了子网络互蒸馏的概念，即根据三明治规则抽样的所有子网络之间的相互蒸馏过程。此外，我们采用解耦知识蒸馏（DKD）损失来增强日志概率蒸馏的效果。为了加速考虑硬件资源的搜索进程，我们利用来自三种不同机器人边缘硬件的数据训练了时延替代预测器。这些预测器在搜索阶段能够估算硬件推理延迟，从而实现统一多目标进化搜索以平衡模型精度与时延之间的权衡。 通过RAM-NAS方法发现的模型系列，在ImageNet数据集上的Top-1准确率范围为76.7%到81.4%之间。此外，我们所采用的资源感知多目标NAS显著降低了机器人边缘硬件上模型的推理延迟时间。我们在下游任务中进行了实验以验证该方法的可扩展性，结果显示相较于基于MobileNetv3的方法，在三种不同的硬件类型上检测和分割任务的推理时间均有所减少。 我们的工作填补了NAS在机器人硬件资源感知方面的空白，为今后的研究提供了宝贵的参考与借鉴。
URL
https://arxiv.org/abs/2509.20688
PDF
-
Quantum Architecture Search for Solving Quantum Machine Learning Tasks
2025-09-14 09:55:38Michael K\"olle, Simon Salfer, Tobias Rohe, Philipp Altmann, Claudia Linnhoff-PopienAbstract
Quantum computing leverages quantum mechanics to address computational problems in ways that differ fundamentally from classical approaches. While current quantum hardware remains error-prone and limited in scale, Variational Quantum Circuits offer a noise-resilient framework suitable for today's devices. The performance of these circuits strongly depends on the underlying architecture of their parameterized quantum components. Identifying efficient, hardware-compatible quantum circuit architectures -- known as Quantum Architecture Search (QAS) -- is therefore essential. Manual QAS is complex and error-prone, motivating efforts to automate it. Among various automated strategies, Reinforcement Learning (RL) remains underexplored, particularly in Quantum Machine Learning contexts. This work introduces RL-QAS, a framework that applies RL to discover effective circuit architectures for classification tasks. We evaluate RL-QAS using the Iris and binary MNIST datasets. The agent autonomously discovers low-complexity circuit designs that achieve high test accuracy. Our results show that RL is a viable approach for automated architecture search in quantum machine learning. However, applying RL-QAS to more complex tasks will require further refinement of the search strategy and performance evaluation mechanisms.
Abstract (translated)
量子计算利用量子力学来解决计算问题，这与传统的计算方法从根本上不同。尽管当前的量子硬件仍然存在错误并且规模有限，但变分量子电路提供了一种适用于现有设备的抗噪框架。这些电路的性能在很大程度上取决于其参数化量子组件的基础架构。因此，识别高效的、硬件兼容的量子电路架构——称为量子架构搜索（QAS）——是至关重要的。手动进行QAS既复杂又容易出错，这促使了自动化它的努力。在各种自动策略中，尽管强化学习（RL）在量子机器学习环境中尚未得到充分探索，但本文介绍了一种名为RL-QAS的框架，该框架利用RL来发现适用于分类任务的有效电路架构。我们使用鸢尾花和二进制MNIST数据集对RL-QAS进行了评估。代理自主发现了低复杂度且测试准确率高的电路设计。我们的结果表明，RL是一种在量子机器学习中进行自动架构搜索的可行方法。然而，要将RL-QAS应用于更复杂的任务，则需要进一步优化搜索策略和性能评估机制。
URL
https://arxiv.org/abs/2509.11198
PDF
-
Evaluating the Efficiency of Latent Spaces via the Coupling-Matrix
2025-09-08 03:36:47Mehmet Can Yavuz, Berrin YanikogluAbstract
A central challenge in representation learning is constructing latent embeddings that are both expressive and efficient. In practice, deep networks often produce redundant latent spaces where multiple coordinates encode overlapping information, reducing effective capacity and hindering generalization. Standard metrics such as accuracy or reconstruction loss provide only indirect evidence of such redundancy and cannot isolate it as a failure mode. We introduce a redundancy index, denoted rho(C), that directly quantifies inter-dimensional dependencies by analyzing coupling matrices derived from latent representations and comparing their off-diagonal statistics against a normal distribution via energy distance. The result is a compact, interpretable, and statistically grounded measure of representational quality. We validate rho(C) across discriminative and generative settings on MNIST variants, Fashion-MNIST, CIFAR-10, and CIFAR-100, spanning multiple architectures and hyperparameter optimization strategies. Empirically, low rho(C) reliably predicts high classification accuracy or low reconstruction error, while elevated redundancy is associated with performance collapse. Estimator reliability grows with latent dimension, yielding natural lower bounds for reliable analysis. We further show that Tree-structured Parzen Estimators (TPE) preferentially explore low-rho regions, suggesting that rho(C) can guide neural architecture search and serve as a redundancy-aware regularization target. By exposing redundancy as a universal bottleneck across models and tasks, rho(C) offers both a theoretical lens and a practical tool for evaluating and improving the efficiency of learned representations.
Abstract (translated)
在表示学习中的一个核心挑战是构建既能表达丰富信息又能保持高效性的潜在嵌入。实践中，深度网络常常产生冗余的潜在空间，在这种空间中，多个坐标编码重叠的信息，这会减少有效容量并阻碍泛化能力。标准指标如准确率或重构损失只能间接证明这一冗余现象，并且无法将其作为失败模式进行隔离。我们引入了一个冗余度指数ρ(C)，通过分析从潜在表示推导出的耦合矩阵的对角线之外统计量与正态分布之间的能量距离来直接量化维度间的相互依赖性，从而提供了一种紧凑、可解释和基于统计数据的质量衡量标准。 我们在多个数据集上验证了ρ(C)的有效性，包括MNIST变体、Fashion-MNIST、CIFAR-10以及CIFAR-100，并且涵盖了多种架构与超参数优化策略。从经验来看，低ρ(C)值能够可靠地预测高分类准确率或低重构误差，而较高的冗余度则常常导致性能下降。随着潜在维度的增加，估计器可靠性也相应提升，从而为可靠的分析提供了自然下界。 此外我们还表明，树结构帕尔森估算器(TPE)倾向于探索低ρ区域，这暗示着ρ(C)可以作为神经架构搜索中的指引工具，并且可以用作考虑冗余度的正则化目标。通过揭示在各种模型和任务中普遍存在的冗余瓶颈问题，ρ(C)不仅为评估和改进学习表示效率提供了理论视角，还提供了一种实用工具。
URL
https://arxiv.org/abs/2509.06314
PDF
-
LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encoding
2025-09-06 09:26:39Yuxuan Hu, Jihao Liu, Ke Wang, Jinliang Zhen, Weikang Shi, Manyuan Zhang, Qi Dou, Rui Liu, Aojun Zhou, Hongsheng LiAbstract
Recent progress in Large Language Models (LLMs) has opened new avenues for solving complex optimization problems, including Neural Architecture Search (NAS). However, existing LLM-driven NAS approaches rely heavily on prompt engineering and domain-specific tuning, limiting their practicality and scalability across diverse tasks. In this work, we propose LM-Searcher, a novel framework that leverages LLMs for cross-domain neural architecture optimization without the need for extensive domain-specific adaptation. Central to our approach is NCode, a universal numerical string representation for neural architectures, which enables cross-domain architecture encoding and search. We also reformulate the NAS problem as a ranking task, training LLMs to select high-performing architectures from candidate pools using instruction-tuning samples derived from a novel pruning-based subspace sampling strategy. Our curated dataset, encompassing a wide range of architecture-performance pairs, encourages robust and transferable learning. Comprehensive experiments demonstrate that LM-Searcher achieves competitive performance in both in-domain (e.g., CNNs for image classification) and out-of-domain (e.g., LoRA configurations for segmentation and generation) tasks, establishing a new paradigm for flexible and generalizable LLM-based architecture search. The datasets and models will be released at this https URL.
Abstract (translated)
最近在大型语言模型（LLMs）方面的进展为解决复杂的优化问题开辟了新的途径，其中包括神经架构搜索（NAS）。然而，现有的基于LLM的NAS方法高度依赖于提示工程和特定领域的调整，这限制了它们在各种任务中的实用性和可扩展性。为此，我们提出了一种新型框架LM-Searcher，它利用大型语言模型进行跨领域神经架构优化，并且无需广泛的领域特定适应。 我们的方法核心在于NCode，这是一种用于表示神经架构的通用数值字符串表示法，它可以实现跨领域的架构编码和搜索。此外，我们将NAS问题重新表述为一个排序任务，通过使用一种基于剪枝的新颖子空间采样策略获得的指令微调样本训练LLM，来从候选池中选择高性能的架构。 我们精心策划的数据集涵盖了广泛的架构-性能对组合，鼓励了稳健且可迁移的学习。全面的实验表明，LM-Searcher在领域内（例如用于图像分类的CNNs）和跨领域任务（如分割和生成中的LoRA配置）上均实现了具有竞争力的表现，从而确立了一种基于LLM的新灵活通用架构搜索范例。 相关数据集与模型将会在此链接处发布：[提供具体的URL]。
URL
https://arxiv.org/abs/2509.05657
PDF
-
OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search
Abstract
Neural architecture search (NAS) is a hard computationally expensive optimization problem with a discrete, vast, and spiky search space. One of the key research efforts dedicated to this space focuses on accelerating NAS via certain proxy evaluations of neural architectures. Different from the prevalent predictor-based methods using surrogate models and differentiable architecture search via supernetworks, we propose an optimization proxy to streamline the NAS as an end-to-end optimization framework, named OptiProxy-NAS. In particular, using a proxy representation, the NAS space is reformulated to be continuous, differentiable, and smooth. Thereby, any differentiable optimization method can be applied to the gradient-based search of the relaxed architecture parameters. Our comprehensive experiments on $12$ NAS tasks of $4$ search spaces across three different domains including computer vision, natural language processing, and resource-constrained NAS fully demonstrate the superior search results and efficiency. Further experiments on low-fidelity scenarios verify the flexibility.
Abstract (translated)
神经架构搜索（NAS）是一个计算上非常昂贵的优化问题，其搜索空间是离散、巨大且具有突变特性的。针对这一领域的一项关键研究努力集中在通过某些代理评估来加速NAS。与流行的基于预测的方法使用替代模型和超网络进行可微架构搜索不同，我们提出了一种优化代理OptiProxy-NAS，它将NAS作为一个端到端的优化框架进行了简化。特别是，利用一个代理表示，我们可以重新定义NAS空间使其成为连续、可微且平滑的空间。因此，任何基于梯度的方法都可以应用于松弛后的架构参数搜索。我们在计算机视觉、自然语言处理和资源受限的NAS三大领域中的12个NAS任务上进行了全面实验，证明了OptiProxy-NAS在搜索效果和效率上的优越性。此外，在低精度场景下的进一步实验验证了该方法的高度灵活性。
URL
https://arxiv.org/abs/2509.05656
PDF
-
Leveraging Transfer Learning and Mobile-enabled Convolutional Neural Networks for Improved Arabic Handwritten Character Recognition
2025-09-05 11:28:53Mohsine El Khayati, Ayyad Maafiri, Yassine Himeur, Hamzah Ali Alkhazaleh, Shadi Atalla, Wathiq MansoorAbstract
The study explores the integration of transfer learning (TL) with mobile-enabled convolutional neural networks (MbNets) to enhance Arabic Handwritten Character Recognition (AHCR). Addressing challenges like extensive computational requirements and dataset scarcity, this research evaluates three TL strategies--full fine-tuning, partial fine-tuning, and training from scratch--using four lightweight MbNets: MobileNet, SqueezeNet, MnasNet, and ShuffleNet. Experiments were conducted on three benchmark datasets: AHCD, HIJJA, and IFHCDB. MobileNet emerged as the top-performing model, consistently achieving superior accuracy, robustness, and efficiency, with ShuffleNet excelling in generalization, particularly under full fine-tuning. The IFHCDB dataset yielded the highest results, with 99% accuracy using MnasNet under full fine-tuning, highlighting its suitability for robust character recognition. The AHCD dataset achieved competitive accuracy (97%) with ShuffleNet, while HIJJA posed significant challenges due to its variability, achieving a peak accuracy of 92% with ShuffleNet. Notably, full fine-tuning demonstrated the best overall performance, balancing accuracy and convergence speed, while partial fine-tuning underperformed across metrics. These findings underscore the potential of combining TL and MbNets for resource-efficient AHCR, paving the way for further optimizations and broader applications. Future work will explore architectural modifications, in-depth dataset feature analysis, data augmentation, and advanced sensitivity analysis to enhance model robustness and generalizability.
Abstract (translated)
这项研究探讨了将迁移学习（TL）与移动设备支持的卷积神经网络（MbNets）结合，以增强阿拉伯手写字符识别（AHCR）。为了解决诸如计算需求量大和数据集稀缺等挑战，本研究评估了三种迁移学习策略——完全微调、部分微调以及从头开始训练，使用四种轻量级的MbNet模型：MobileNet、SqueezeNet、MnasNet 和 ShuffleNet。实验在三个基准数据集上进行：AHCD、HIJJA 和 IFHCDB。 MobileNet 凸显为表现最佳的模型，在精度、鲁棒性和效率方面持续表现出色，而ShuffleNet 在完全微调的情况下，尤其在泛化性能方面表现出色。IFHCDB 数据集取得了最高结果，使用 MnasNet 进行完全微调达到了99% 的准确率，突显其对于字符识别的稳健性。AHCD 数据集利用 ShuffleNet 达到了有竞争力的 97% 准确率，而 HIJJA 数据集由于其变异性带来了重大挑战，在使用 ShuffleNet 的情况下达到最高的 92% 准确率。 值得注意的是，完全微调策略展示了最佳的整体性能，很好地平衡了精度和收敛速度，相比之下部分微调在各项指标中表现较差。这些发现强调了将迁移学习与MbNets 结合应用于资源高效的AHCR的潜力，并为进一步优化及更广泛的应用铺平道路。未来的研究工作将进一步探索架构改进、深入的数据集特征分析、数据增强以及高级敏感性分析，以提高模型的鲁棒性和泛化能力。
URL
https://arxiv.org/abs/2509.05019
PDF
-
STA-Net: A Decoupled Shape and Texture Attention Network for Lightweight Plant Disease Classification
2025-09-03 22:46:20Zongsen QiuAbstract
Responding to rising global food security needs, precision agriculture and deep learning-based plant disease diagnosis have become crucial. Yet, deploying high-precision models on edge devices is challenging. Most lightweight networks use attention mechanisms designed for generic object recognition, which poorly capture subtle pathological features like irregular lesion shapes and complex textures. To overcome this, we propose a twofold solution: first, using a training-free neural architecture search method (DeepMAD) to create an efficient network backbone for edge devices; second, introducing the Shape-Texture Attention Module (STAM). STAM splits attention into two branches -- one using deformable convolutions (DCNv4) for shape awareness and the other using a Gabor filter bank for texture awareness. On the public CCMT plant disease dataset, our STA-Net model (with 401K parameters and 51.1M FLOPs) reached 89.00% accuracy and an F1 score of 88.96%. Ablation studies confirm STAM significantly improves performance over baseline and standard attention models. Integrating domain knowledge via decoupled attention thus presents a promising path for edge-deployed precision agriculture AI. The source code is available at this https URL.
Abstract (translated)
为了应对全球粮食安全需求的增长，精准农业和基于深度学习的植物病害诊断变得至关重要。然而，在边缘设备上部署高精度模型面临挑战。大多数轻量级网络使用的是为通用物体识别设计的关注机制，这些机制难以捕捉到如不规则病变形状和复杂纹理等细微病理特征。为此，我们提出了一种两阶段解决方案：首先，采用无训练的神经架构搜索方法（DeepMAD）来创建一个适用于边缘设备的有效网络骨干；其次，引入形状-纹理注意力模块（STAM）。该模块将注意力分为两个分支——其中一个使用可变形卷积（DCNv4）以增强对形状的关注，另一个则利用Gabor滤波器组关注纹理。 在公开的CCMT植物病害数据集上，我们的STA-Net模型（具有401K参数和51.1M FLOPs）达到了89.00%的准确率以及88.96%的F1分数。消融研究表明，STAM显著提高了性能，在基准和标准注意力模型之上表现出优势。通过解耦注意力整合领域知识，为部署于边缘设备上的精准农业AI提供了有希望的发展路径。 源代码可在以下链接获取：[此URL]（请将括号内的文本替换为实际的网址）。
URL
https://arxiv.org/abs/2509.03754
PDF
-
A Continuous Encoding-Based Representation for Efficient Multi-Fidelity Multi-Objective Neural Architecture Search
2025-09-02 04:31:02Zhao Wei, Chin Chun Ooi, Yew-Soon OngAbstract
Neural architecture search (NAS) is an attractive approach to automate the design of optimized architectures but is constrained by high computational budget, especially when optimizing for multiple, important conflicting objectives. To address this, an adaptive Co-Kriging-assisted multi-fidelity multi-objective NAS algorithm is proposed to further reduce the computational cost of NAS by incorporating a clustering-based local multi-fidelity infill sampling strategy, enabling efficient exploration of the search space for faster convergence. This algorithm is further accelerated by the use of a novel continuous encoding method to represent the connections of nodes in each cell within a generalized cell-based U-Net backbone, thereby decreasing the search dimension (number of variables). Results indicate that the proposed NAS algorithm outperforms previously published state-of-the-art methods under limited computational budget on three numerical benchmarks, a 2D Darcy flow regression problem and a CHASE_DB1 biomedical image segmentation problem. The proposed method is subsequently used to create a wind velocity regression model with application in urban modelling, with the found model able to achieve good prediction with less computational complexity. Further analysis revealed that the NAS algorithm independently identified principles undergirding superior U-Net architectures in other literature, such as the importance of allowing each cell to incorporate information from prior cells.
Abstract (translated)
神经架构搜索（NAS）是一种自动设计优化网络结构的方法，但其计算成本高昂，尤其是在针对多个重要且相互冲突的目标进行优化时。为此，提出了一种基于自适应Co-Kriging的多保真度多目标NAS算法，通过引入一种基于聚类的局部多保真度填充采样策略来进一步降低NAS的计算成本，从而能够有效地探索搜索空间并加速收敛过程。此外，该算法还采用了一种新颖的连续编码方法表示每个单元内的节点连接，在一个通用的基于单元的U-Net骨干网络中减少了搜索维度（变量的数量）。实验结果表明，在三个数值基准、二维达西流回归问题以及CHASE_DB1生物医学图像分割问题上，在有限计算预算下，所提出的NAS算法优于先前公布的最先进方法。此外，该方法还被用于创建一个应用于城市建模的风速回归模型，找到的模型能够在降低计算复杂度的情况下实现良好的预测效果。进一步分析表明，NAS算法独立地识别出了其他文献中优越U-Net架构背后的原则，例如允许每个单元从先前单元中获取信息的重要性。
URL
https://arxiv.org/abs/2509.01943
PDF
-
Multi-Representation Adapter with Neural Architecture Search for Efficient Range-Doppler Radar Object Detection
2025-09-01 09:06:53Zhiwei Lin, Weicheng Zheng, Yongtao WangAbstract
Detecting objects efficiently from radar sensors has recently become a popular trend due to their robustness against adverse lighting and weather conditions compared with cameras. This paper presents an efficient object detection model for Range-Doppler (RD) radar maps. Specifically, we first represent RD radar maps with multi-representation, i.e., heatmaps and grayscale images, to gather high-level object and fine-grained texture features. Then, we design an additional Adapter branch, an Exchanger Module with two modes, and a Primary-Auxiliary Fusion Module to effectively extract, exchange, and fuse features from the multi-representation inputs, respectively. Furthermore, we construct a supernet with various width and fusion operations in the Adapter branch for the proposed model and employ a One-Shot Neural Architecture Search method to further improve the model's efficiency while maintaining high performance. Experimental results demonstrate that our model obtains favorable accuracy and efficiency trade-off. Moreover, we achieve new state-of-the-art performance on RADDet and CARRADA datasets with mAP@50 of 71.9 and 57.1, respectively.
Abstract (translated)
最近，由于雷达传感器在恶劣光照和天气条件下的鲁棒性优于摄像头，从雷达传感器中高效检测物体已成为一种流行趋势。本文提出了一种针对范围-多普勒（RD）雷达地图的高效目标检测模型。具体而言，我们首先使用多种表示方式来表征RD雷达图：热力图和灰度图像，以收集高层次的对象特征及细粒度的纹理特征。接着，设计了一个额外的适配器分支、具有两种模式的交换器模块以及主辅助融合模块，分别有效地从多表示输入中提取、交换和融合特征。此外，我们构建了一个包含不同宽度和融合操作的超网络（supernet）用于所提出的模型，并采用了一次性神经架构搜索方法以进一步提高模型效率同时保持高性能。实验结果表明，我们的模型在准确性和效率之间取得了有利的平衡。另外，在RADDet和CARRADA数据集上，我们分别达到了mAP@50为71.9和57.1的新最先进性能水平。
URL
https://arxiv.org/abs/2509.01280
PDF
-
SAR-NAS: Lightweight SAR Object Detection with Neural Architecture Search
2025-09-01 09:06:13Xinyi Yu, Zhiwei Lin, Yongtao WangAbstract
Synthetic Aperture Radar (SAR) object detection faces significant challenges from speckle noise, small target ambiguities, and on-board computational constraints. While existing approaches predominantly focus on SAR-specific architectural modifications, this paper explores the application of the existing lightweight object detector, i.e., YOLOv10, for SAR object detection and enhances its performance through Neural Architecture Search (NAS). Specifically, we employ NAS to systematically optimize the network structure, especially focusing on the backbone architecture search. By constructing an extensive search space and leveraging evolutionary search, our method identifies a favorable architecture that balances accuracy, parameter efficiency, and computational cost. Notably, this work introduces NAS to SAR object detection for the first time. The experimental results on the large-scale SARDet-100K dataset demonstrate that our optimized model outperforms existing SAR detection methods, achieving superior detection accuracy while maintaining lower computational overhead. We hope this work offers a novel perspective on leveraging NAS for real-world applications.
Abstract (translated)
合成孔径雷达（SAR）目标检测面临的主要挑战包括斑点噪声、小目标模糊性和机载计算限制。虽然现有的方法主要集中在针对 SAR 特性的架构修改上，本文探索了将现有轻量级对象检测器——YOLOv10 应用于 SAR 目标检测，并通过神经结构搜索（NAS）提高其性能。具体来说，我们利用 NAS 系统地优化网络结构，特别是在骨干架构搜索方面尤为关注。通过构建广泛的搜索空间并采用进化搜索方法，我们的方法能够识别出一种在精度、参数效率和计算成本之间取得平衡的有利架构。值得注意的是，本工作首次将 NAS 引入 SAR 目标检测领域。 在大规模 SARDet-100K 数据集上的实验结果表明，优化后的模型优于现有的 SAR 检测方法，在保持较低计算开销的同时实现了更优的目标检测精度。我们希望这项工作能够为利用 NAS 解决实际问题提供一种新的视角。
URL
https://arxiv.org/abs/2509.01279
PDF
-
Principled Approximation Methods for Efficient and Scalable Deep Learning
2025-08-29 18:17:48Pedro SavareseAbstract
Recent progress in deep learning has been driven by increasingly larger models. However, their computational and energy demands have grown proportionally, creating significant barriers to their deployment and to a wider adoption of deep learning technologies. This thesis investigates principled approximation methods for improving the efficiency of deep learning systems, with a particular focus on settings that involve discrete constraints and non-differentiability. We study three main approaches toward improved efficiency: architecture design, model compression, and optimization. For model compression, we propose novel approximations for pruning and quantization that frame the underlying discrete problem as continuous and differentiable, enabling gradient-based training of compression schemes alongside the model's parameters. These approximations allow for fine-grained sparsity and precision configurations, leading to highly compact models without significant fine-tuning. In the context of architecture design, we design an algorithm for neural architecture search that leverages parameter sharing across layers to efficiently explore implicitly recurrent architectures. Finally, we study adaptive optimization, revisiting theoretical properties of widely used methods and proposing an adaptive optimizer that allows for quick hyperparameter tuning. Our contributions center on tackling computationally hard problems via scalable and principled approximations. Experimental results on image classification, language modeling, and generative modeling tasks show that the proposed methods provide significant improvements in terms of training and inference efficiency while maintaining, or even improving, the model's performance.
Abstract (translated)
最近的深度学习进展主要得益于模型规模的不断扩大。然而，这些大型模型在计算和能源需求方面也相应增加，给它们的实际部署以及更广泛地采用深度学习技术带来了显著障碍。本论文探讨了改善深度学习系统效率的原则性近似方法，特别关注包含离散约束和非可微分问题的情境。我们研究了提高效率的三个主要途径：架构设计、模型压缩和优化。 对于模型压缩，我们提出了一种新的修剪和量化近似方法，将底层的离散问题视为连续且可微的问题，使压缩方案可以与模型参数一起进行基于梯度的训练。这些近似方法允许精细调整稀疏性和精度配置，从而创建高度紧凑的模型而无需大量微调。 在架构设计方面，我们提出了一种算法用于神经网络架构搜索，通过利用层间的参数共享来高效地探索隐式递归架构。 最后，我们研究了自适应优化技术，并重新审视广泛使用的现有方法的理论特性，同时提出了一个新的自适应优化器，该优化器能够快速调整超参数。我们的贡献集中在通过可扩展且原则性的近似解决计算难题上。 实验结果表明，在图像分类、语言模型和生成式建模任务中，所提出的方法在保持或甚至提高模型性能的同时，显著提高了训练和推理的效率。
URL
https://arxiv.org/abs/2509.00174
PDF
-
SugarcaneShuffleNet: A Very Fast, Lightweight Convolutional Neural Network for Diagnosis of 15 Sugarcane Leaf Diseases
2025-08-23 18:39:25Shifat E. Arman, Hasan Muhammad Abdullah, Syed Nazmus Sakib, RM Saiem, Shamima Nasrin Asha, Md Mehedi Hasan, Shahrear Bin Amin, S M Mahin AbrarAbstract
Despite progress in AI-based plant diagnostics, sugarcane farmers in low-resource regions remain vulnerable to leaf diseases due to the lack of scalable, efficient, and interpretable tools. Many deep learning models fail to generalize under real-world conditions and require substantial computational resources, limiting their use in resource-constrained regions. In this paper, we present SugarcaneLD-BD, a curated dataset for sugarcane leaf-disease classification; SugarcaneShuffleNet, an optimized lightweight model for rapid on-device diagnosis; and SugarcaneAI, a Progressive Web Application for field deployment. SugarcaneLD-BD contains 638 curated images across five classes, including four major sugarcane diseases, collected in Bangladesh under diverse field conditions and verified by expert pathologists. To enhance diversity, we combined SugarcaneLD-BD with two additional datasets, yielding a larger and more representative corpus. Our optimized model, SugarcaneShuffleNet, offers the best trade-off between speed and accuracy for real-time, on-device diagnosis. This 9.26 MB model achieved 98.02% accuracy, an F1-score of 0.98, and an average inference time of 4.14 ms per image. For comparison, we fine-tuned five other lightweight convolutional neural networks: MnasNet, EdgeNeXt, EfficientNet-Lite, MobileNet, and SqueezeNet via transfer learning and Bayesian optimization. MnasNet and EdgeNeXt achieved comparable accuracy to SugarcaneShuffleNet, but required significantly more parameters, memory, and computation, limiting their suitability for low-resource deployment. We integrate SugarcaneShuffleNet into SugarcaneAI, delivering Grad-CAM-based explanations in the field. Together, these contributions offer a diverse benchmark, efficient models for low-resource environments, and a practical tool for sugarcane disease classification. It spans varied lighting, backgrounds and devices used on-farm
Abstract (translated)
尽管基于人工智能的植物诊断技术取得了进展，但资源匮乏地区甘蔗种植者仍易受到叶片疾病的威胁，因为缺乏可扩展、高效且易于解释的工具。许多深度学习模型在实际条件下的泛化能力较差，并需要大量计算资源，这限制了它们在资源受限地区的应用。 本文介绍了SugarcaneLD-BD（用于甘蔗叶病分类的数据集）、SugarcaneShuffleNet（一种优化后的轻量级模型，可在设备上快速进行诊断）和SugarcaneAI（一款适用于现场部署的渐进式Web应用程序）。SugarcaneLD-BD包含638张经过精心挑选的图像，涵盖五个类别，包括四种主要的甘蔗疾病，在孟加拉国的各种田间条件下收集，并由专家病理学家进行了验证。为了增强多样性，我们结合了两个额外的数据集，形成了一个更大、更具代表性的数据集合。 我们的优化模型SugarcaneShuffleNet在实时设备上诊断的速度和准确性之间提供了最佳的权衡。这款9.26MB大小的模型达到了98.02%的准确率，F1评分为0.98，并且每张图像的平均推理时间为4.14毫秒。作为比较，我们通过迁移学习和贝叶斯优化对另外五种轻量级卷积神经网络（MnasNet、EdgeNeXt、EfficientNet-Lite、MobileNet 和 SqueezeNet）进行了微调。尽管MnasNet和EdgeNeXt在准确率方面与SugarcaneShuffleNet相当，但它们需要更多的参数、内存及计算资源，限制了其在低资源配置环境中的适用性。 我们将SugarcaneShuffleNet集成到SugarcaneAI中，在现场提供了基于Grad-CAM的解释。这些贡献共同提供了一个多样化的基准测试、适用于低资源环境的有效模型以及一个实用的甘蔗疾病分类工具。它涵盖了各种照明条件、背景和田间使用的设备。
URL
https://arxiv.org/abs/2508.17107
PDF
-
Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search
2025-08-21 17:59:08Yuxian Gu, Qinghao Hu, Shang Yang, Haocheng Xi, Junyu Chen, Song Han, Han CaiAbstract
We present Jet-Nemotron, a new family of hybrid-architecture language models, which matches or exceeds the accuracy of leading full-attention models while significantly improving generation throughput. Jet-Nemotron is developed using Post Neural Architecture Search (PostNAS), a novel neural architecture exploration pipeline that enables efficient model design. Unlike prior approaches, PostNAS begins with a pre-trained full-attention model and freezes its MLP weights, allowing efficient exploration of attention block designs. The pipeline includes four key components: (1) learning optimal full-attention layer placement and elimination, (2) linear attention block selection, (3) designing new attention blocks, and (4) performing hardware-aware hyperparameter search. Our Jet-Nemotron-2B model achieves comparable or superior accuracy to Qwen3, Qwen2.5, Gemma3, and Llama3.2 across a comprehensive suite of benchmarks while delivering up to 53.6x generation throughput speedup and 6.1x prefilling speedup. It also achieves higher accuracy on MMLU and MMLU-Pro than recent advanced MoE full-attention models, such as DeepSeek-V3-Small and Moonlight, despite their larger scale with 15B total and 2.2B activated parameters.
Abstract (translated)
我们介绍了Jet-Nemotron，这是一种新的混合架构语言模型系列。它在准确率上与领先的全注意力模型持平或超越它们的同时，显著提高了生成吞吐量。Jet-Nemotron 使用后神经网络结构搜索（Post Neural Architecture Search，简称 PostNAS），这是一种新颖的神经架构探索流水线，能够高效地进行模型设计。不同于以往的方法，PostNAS 从一个预训练好的全注意力模型开始，并冻结其 MLP 权重，从而可以高效地探索注意力模块的设计。该流水线包含四个关键组件：(1) 学习最优全注意力层的放置和消除；(2) 线性注意力块的选择；(3) 设计新的注意力模块；以及 (4) 进行硬件感知的超参数搜索。 我们的 Jet-Nemotron-2B 模型在全面的基准测试套件中，实现了与 Qwen3、Qwen2.5、Gemma3 和 Llama3.2 相当或更好的准确率，同时提供了高达 53.6 倍的生成吞吐量加速和 6.1 倍的预填充速度提升。此外，在 MMLU 和 MMLU-Pro 测试中，Jet-Nemotron-2B 的表现优于最近先进的 MoE 全注意力模型（如 DeepSeek-V3-Small 和 Moonlight），尽管这些模型拥有更大规模的参数总数和激活参数数（分别为 150 亿和 22 亿）。
URL
https://arxiv.org/abs/2508.15884
PDF
