NAS
-
Search Multilayer Perceptron-Based Fusion for Efficient and Accurate Siamese Tracking
2026-03-02 10:30:54Tianqi Shen, Huakao Lin, Ning AnAbstract
Siamese visual trackers have recently advanced through increasingly sophisticated fusion mechanisms built on convolutional or Transformer architectures. However, both struggle to deliver pixel-level interactions efficiently on resource-constrained hardware, leading to a persistent accuracy-efficiency imbalance. Motivated by this limitation, we redesign the Siamese neck with a simple yet effective Multilayer Perception (MLP)-based fusion module that enables pixel-level interaction with minimal structural overhead. Nevertheless, naively stacking MLP blocks introduces a new challenge: computational cost can scale quadratically with channel width. To overcome this, we construct a hierarchical search space of carefully designed MLP modules and introduce a customized relaxation strategy that enables differentiable neural architecture search (DNAS) to decouple channel-width optimization from other architectural choices. This targeted decoupling automatically balances channel width and depth, yielding a low-complexity architecture. The resulting tracker achieves state-of-the-art accuracy-efficiency trade-offs. It ranks among the top performers on four general-purpose and three aerial tracking benchmarks, while maintaining real-time performance on both resource-constrained Graphics Processing Units (GPUs) and Neural Processing Units (NPUs).
Abstract (translated)
最近，暹罗视觉跟踪器通过基于卷积或Transformer架构的日益复杂的融合机制得到了改进。然而，这两种方法都在资源受限的硬件上难以高效地实现像素级别的交互，导致了持久存在的精度与效率之间的不平衡问题。鉴于这一局限性，我们重新设计了暹罗颈部，采用了一种简单而有效的多层感知器（MLP）融合模块，该模块能够以最少的结构开销实现像素级互动。然而，直接堆叠MLP块会带来一个新的挑战：计算成本可能会随着通道宽度呈二次方增长。为了解决这个问题，我们构建了一个精心设计的MLP模块层次搜索空间，并引入了一种定制化的松弛策略，使可微神经架构搜索（DNAS）能够将通道宽度优化与其他架构选择解耦。这种有目标的解耦可以自动平衡通道宽度与深度，从而产生低复杂度的架构。由此产生的跟踪器在精度和效率之间的权衡上达到了最先进的水平，在四个通用目的和三个空中追踪基准测试中表现出顶级性能，并且能够在资源受限的图形处理器（GPU）和神经处理单元（NPU）上保持实时性能。
URL
https://arxiv.org/abs/2603.01706
PDF
-
Model Agreement via Anchoring
2026-02-26 18:59:32Eric Eaton, Surbhi Goel, Marcel Hussing, Michael Kearns, Aaron Roth, Sikata Bela Sengupta, Jessica SorrellAbstract
Numerous lines of aim to control $\textit{model disagreement}$ -- the extent to which two machine learning models disagree in their predictions. We adopt a simple and standard notion of model disagreement in real-valued prediction problems, namely the expected squared difference in predictions between two models trained on independent samples, without any coordination of the training processes. We would like to be able to drive disagreement to zero with some natural parameter(s) of the training procedure using analyses that can be applied to existing training methodologies. We develop a simple general technique for proving bounds on independent model disagreement based on $\textit{anchoring}$ to the average of two models within the analysis. We then apply this technique to prove disagreement bounds for four commonly used machine learning algorithms: (1) stacked aggregation over an arbitrary model class (where disagreement is driven to 0 with the number of models $k$ being stacked) (2) gradient boosting (where disagreement is driven to 0 with the number of iterations $k$) (3) neural network training with architecture search (where disagreement is driven to 0 with the size $n$ of the architecture being optimized over) and (4) regression tree training over all regression trees of fixed depth (where disagreement is driven to 0 with the depth $d$ of the tree architecture). For clarity, we work out our initial bounds in the setting of one-dimensional regression with squared error loss -- but then show that all of our results generalize to multi-dimensional regression with any strongly convex loss.
Abstract (translated)
许多研究工作旨在控制**模型分歧**，即两个机器学习模型在预测中不一致的程度。我们采用了一种关于实值预测问题中的模型分歧的简单且标准的概念：这是指训练过程中，在独立样本上训练的两个模型之间预测结果的期望平方差（无需协调各自的训练过程）。 我们的目标是能够通过调整一些自然参数来将这种分歧降至零，这些参数可以应用于现有的训练方法。为此，我们开发了一种基于**锚定到两个模型平均值**来进行独立模型分歧界限证明的基本技术。然后，我们将此技术应用到四种常用的机器学习算法中以证明分歧界限： 1. **堆叠聚合（stacked aggregation）**：针对任意模型类别的堆叠聚合方法，在该方法中，随着被堆叠的模型数量 $k$ 的增加，分歧趋向于零。 2. **梯度提升（gradient boosting）**：在梯度提升算法中，通过迭代次数 $k$ 的增加来使分歧减少直至趋近于零。 3. **神经网络训练与架构搜索**：对于使用架构搜索进行的神经网络训练，在优化目标为特定大小 $n$ 架构的情况下，随着架构规模的增长，分歧也趋向于减小直到接近于零。 4. **回归树训练（regression tree training）**：当对所有固定深度 $d$ 的回归树进行训练时，随着树的深度增加，模型之间的分歧将减少至趋于零。 为了解释清晰，我们首先在单变量回归和平方误差损失的情况下推导出了初始界限，然后展示了我们的结果可以推广到具有任意强凸损失函数的多维回归问题中。
URL
https://arxiv.org/abs/2602.23360
PDF
-
Optimizing Neural Network Architecture for Medical Image Segmentation Using Monte Carlo Tree Search
2026-02-25 19:40:41Liping Meng, Fan Nie, Yunyun Zhang, Chao HanAbstract
This paper proposes a novel medical image segmentation framework, MNAS-Unet, which combines Monte Carlo Tree Search (MCTS) and Neural Architecture Search (NAS). MNAS-Unet dynamically explores promising network architectures through MCTS, significantly enhancing the efficiency and accuracy of architecture search. It also optimizes the DownSC and UpSC unit structures, enabling fast and precise model adjustments. Experimental results demonstrate that MNAS-Unet outperforms NAS-Unet and other state-of-the-art models in segmentation accuracy on several medical image datasets, including PROMISE12, Ultrasound Nerve, and CHAOS. Furthermore, compared with NAS-Unet, MNAS-Unet reduces the architecture search budget by 54% (early stopping at 139 epochs versus 300 epochs under the same search setting), while achieving a lightweight model with only 0.6M parameters and lower GPU memory consumption, which further improves its practical applicability. These results suggest that MNAS-Unet can improve search efficiency while maintaining competitive segmentation accuracy under practical resource constraints.
Abstract (translated)
本文提出了一种新颖的医学图像分割框架MNAS-Unet，该框架结合了蒙特卡洛树搜索（MCTS）和神经架构搜索（NAS）。MNAS-Unet通过MCTS动态探索有前景的网络结构，从而显著提高架构搜索的效率和准确性。此外，它还优化了DownSC和UpSC单元结构，使其能够快速准确地调整模型。 实验结果表明，在多个医学图像数据集上（包括PROMISE12、超声神经和CHAOS），MNAS-Unet在分割精度方面超越了NAS-Unet及其他最先进的模型。与NAS-Unet相比，在相同的搜索设置下，MNAS-Unet将架构搜索预算减少了54%（提前停止于139个时期而其他情况是300个时期），同时获得了仅包含0.6M参数的轻量级模型，并且降低了GPU内存消耗，从而进一步提升了其实用性。 这些结果表明，在实际资源约束条件下，MNAS-Unet能够在提高搜索效率的同时保持竞争力的分割精度。
URL
https://arxiv.org/abs/2602.22361
PDF
-
DGPO: RL-Steered Graph Diffusion for Neural Architecture Generation
2026-02-22 16:23:42Aleksei Liuliakov, Luca Hermes, Barbara HammerAbstract
Reinforcement learning fine-tuning has proven effective for steering generative diffusion models toward desired properties in image and molecular domains. Graph diffusion models have similarly been applied to combinatorial structure generation, including neural architecture search (NAS). However, neural architectures are directed acyclic graphs (DAGs) where edge direction encodes functional semantics such as data flow-information that existing graph diffusion methods, designed for undirected structures, discard. We propose Directed Graph Policy Optimization (DGPO), which extends reinforcement learning fine-tuning of discrete graph diffusion models to DAGs via topological node ordering and positional encoding. Validated on NAS-Bench-101 and NAS-Bench-201, DGPO matches the benchmark optimum on all three NAS-Bench-201 tasks (91.61%, 73.49%, 46.77%). The central finding is that the model learns transferable structural priors: pretrained on only 7% of the search space, it generates near-oracle architectures after fine-tuning, within 0.32 percentage points of the full-data model and extrapolating 7.3 percentage points beyond its training ceiling. Bidirectional control experiments confirm genuine reward-driven steering, with inverse optimization reaching near random-chance accuracy (9.5%). These results demonstrate that reinforcement learning-steered discrete diffusion, once extended to handle directionality, provides a controllable generative framework for directed combinatorial structures.
Abstract (translated)
强化学习微调已被证明在引导生成扩散模型朝向图像和分子领域所需的特性方面是有效的。图扩散模型也已经被应用于组合结构生成，包括神经架构搜索（NAS）。然而，神经架构是由有向无环图（DAG）组成的，其中边的方向编码了功能语义，如数据流信息——这种信息现有的针对非定向结构设计的图扩散方法会忽略掉。 我们提出了有向图策略优化（DGPO），这是一种通过拓扑节点排序和位置编码来将离散图扩散模型的强化学习微调扩展到DAG的方法。在NAS-Bench-101和NAS-Bench-201上的验证显示，DGPO在所有三个NAS-Bench-201任务上达到了基准最优值（91.61%，73.49%，46.77%）。主要发现是模型学习到了可迁移的结构先验：仅在搜索空间的7%进行预训练后，在微调阶段就生成了接近Oracle架构，其性能几乎与完全数据模型相当，并且还在训练上限之上额外提升了7.3个百分点。双向控制实验确认了真正的奖励驱动引导，逆向优化达到了接近随机机会准确度（9.5%）。这些结果表明，一旦强化学习引导的离散扩散方法能够处理方向性问题，它就能为有向组合结构提供一个可控的生成框架。
URL
https://arxiv.org/abs/2602.19261
PDF
-
SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search
2026-02-17 15:02:02Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, Hung CaoAbstract
Neural architecture search (NAS) automates the discovery of neural networks that meet specified criteria, yet its evaluation procedures are often hardcoded, limiting the ability to introduce new metrics. This issue is especially pronounced in hardware-aware NAS, where objectives depend on target devices such as edge hardware. To address this limitation, we propose SEval-NAS, a metric-evaluation mechanism that converts architectures to strings, embeds them as vectors, and predicts performance metrics. Using NATS-Bench and HW-NAS-Bench, we evaluated accuracy, latency, and memory. Kendall's $\tau$ correlations showed stronger latency and memory predictions than accuracy, indicating the suitability of SEval-NAS as a hardware cost predictor. We further integrated SEval-NAS into FreeREA to evaluate metrics not originally included. The method successfully ranked FreeREA-generated architectures, maintained search time, and required minimal algorithmic changes. Our implementation is available at: this https URL
Abstract (translated)
神经架构搜索（NAS）自动发现了符合特定标准的神经网络，但其评估过程通常是硬编码的，限制了引入新度量指标的能力。在硬件感知型NAS中，目标设备如边缘计算硬件的具体需求使得这一问题更加突出。 为了解决这个问题，我们提出了SEval-NAS，这是一种将架构转换为字符串、嵌入向量并预测性能指标的度量评估机制。使用NATS-Bench和HW-NAS-Bench进行了准确性、延迟和内存的评估。根据肯德尔的$\tau$相关性分析，对于延迟和内存的预测结果比准确性的预测更强，这表明SEval-NAS可以作为硬件成本预测的有效工具。 我们进一步将SEval-NAS集成到了FreeREA中，以评估原本未包含在内的度量指标。该方法成功地对FreeREA生成的架构进行了排序，并保持了搜索时间的同时只需要进行少量算法修改。 我们的实现代码可在以下链接获取：this https URL
URL
https://arxiv.org/abs/2603.00099
PDF
-
Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback
2026-02-13 04:38:32Sein Kim, Sangwu Park, Hongseok Kang, Wonjoong Kim, Jimin Seo, Yeonjun In, Kanghoon Yoon, Chanyoung ParkAbstract
Traditional methods for automating recommender system design, such as Neural Architecture Search (NAS), are often constrained by a fixed search space defined by human priors, limiting innovation to pre-defined operators. While recent LLM-driven code evolution frameworks shift fixed search space target to open-ended program spaces, they primarily rely on scalar metrics (e.g., NDCG, Hit Ratio) that fail to provide qualitative insights into model failures or directional guidance for improvement. To address this, we propose Self-EvolveRec, a novel framework that establishes a directional feedback loop by integrating a User Simulator for qualitative critiques and a Model Diagnosis Tool for quantitative internal verification. Furthermore, we introduce a Diagnosis Tool - Model Co-Evolution strategy to ensure that evaluation criteria dynamically adapt as the recommendation architecture evolves. Extensive experiments demonstrate that Self-EvolveRec significantly outperforms state-of-the-art NAS and LLM-driven code evolution baselines in both recommendation performance and user satisfaction. Our code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.12612
PDF
-
Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs
2026-02-10 23:51:00Luoyang Sun, Jiwen Jiang, Yifeng Ding, Fengfa Li, Yan Song, Haifeng Zhang, Jian Ying, Lei Ren, Kun Zhan, Wei Chen, Yan Xie, Cheng DengAbstract
Vision-Language-Action Models (VLAs) have emerged as a key paradigm of Physical AI and are increasingly deployed in autonomous vehicles, robots, and smart spaces. In these resource-constrained on-device settings, selecting an appropriate large language model (LLM) backbone is a critical challenge: models must balance accuracy with strict inference latency and hardware efficiency constraints. This makes hardware-software co-design a game-changing requirement for on-device LLM deployment, where each hardware platform demands a tailored architectural solution. We propose a hardware co-design law that jointly captures model accuracy and inference performance. Specifically, we model training loss as an explicit function of architectural hyperparameters and characterise inference latency via roofline modelling. We empirically evaluate 1,942 candidate architectures on NVIDIA Jetson Orin, training 170 selected models for 10B tokens each to fit a scaling law relating architecture to training loss. By coupling this scaling law with latency modelling, we establish a direct accuracy-latency correspondence and identify the Pareto frontier for hardware co-designed LLMs. We further formulate architecture search as a joint optimisation over precision and performance, deriving feasible design regions under industrial hardware and application budgets. Our approach reduces architecture selection from months to days. At the same latency as Qwen2.5-0.5B on the target hardware, our co-designed architecture achieves 19.42% lower perplexity on WikiText-2. To our knowledge, this is the first principled and operational framework for hardware co-design scaling laws in on-device LLM deployment. We will make the code and related checkpoints publicly available.
Abstract (translated)
视觉语言行动模型（VLAs）已作为物理人工智能的关键范式而出现，并且越来越多地被部署在自主车辆、机器人和智能空间中。在这种资源受限的设备环境中，选择合适的大型语言模型（LLM）骨干是一个关键挑战：模型必须在严格的推理延迟和硬件效率限制下平衡准确性和性能。这使得软硬件协同设计成为设备上部署LLM的关键需求，每个硬件平台都要求定制的架构解决方案。我们提出了一项软硬件协同设计法则，该法则同时捕捉模型精度和推理性能。具体而言，我们将训练损失建模为架构超参数的显式函数，并通过顶点模型来表征推理延迟。我们在NVIDIA Jetson Orin上对1,942个候选架构进行了实证评估，每个选择了170种模型进行10B代币的训练，以拟合从架构到训练损失的关系。通过将这种扩展法则与延迟建模相结合，我们建立了一个直接的精度-延迟对应关系，并识别出软硬件协同设计LLM的帕累托前沿。进一步地，我们将架构搜索形式化为对精度和性能的同时优化，在工业硬件和应用预算下导出了可行的设计区域。我们的方法将架构选择的时间从数月缩短到了几天。在目标硬件上的相同延迟条件下，我们所提出的协同设计架构相比Qwen2.5-0.5B模型在WikiText-2上实现了19.42%更低的困惑度。据我们所知，这是首个系统且可操作的软硬件协同设计扩展法则框架，用于设备上LLM部署。我们将公开发布代码及相关检查点。
URL
https://arxiv.org/abs/2602.10377
PDF
-
From Lightweight CNNs to SpikeNets: Benchmarking Accuracy-Energy Tradeoffs with Pruned Spiking SqueezeNet
2026-02-10 12:20:11Radib Bin Kabir, Tawsif Tashwar Dipto, Mehedi Ahamed, Sabbir Ahmed, Md Hasanul KabirAbstract
Spiking Neural Networks (SNNs) are increasingly studied as energy-efficient alternatives to Convolutional Neural Networks (CNNs), particularly for edge intelligence. However, prior work has largely emphasized large-scale models, leaving the design and evaluation of lightweight CNN-to-SNN pipelines underexplored. In this paper, we present the first systematic benchmark of lightweight SNNs obtained by converting compact CNN architectures into spiking networks, where activations are modeled with Leaky-Integrate-and-Fire (LIF) neurons and trained using surrogate gradient descent under a unified setup. We construct spiking variants of ShuffleNet, SqueezeNet, MnasNet, and MixNet, and evaluate them on CIFAR-10, CIFAR-100, and TinyImageNet, measuring accuracy, F1-score, parameter count, computational complexity, and energy consumption. Our results show that SNNs can achieve up to 15.7x higher energy efficiency than their CNN counterparts while retaining competitive accuracy. Among these, the SNN variant of SqueezeNet consistently outperforms other lightweight SNNs. To further optimize this model, we apply a structured pruning strategy that removes entire redundant modules, yielding a pruned architecture, SNN-SqueezeNet-P. This pruned model improves CIFAR-10 accuracy by 6% and reduces parameters by 19% compared to the original SNN-SqueezeNet. Crucially, it narrows the gap with CNN-SqueezeNet, achieving nearly the same accuracy (only 1% lower) but with an 88.1% reduction in energy consumption due to sparse spike-driven computations. Together, these findings establish lightweight SNNs as practical, low-power alternatives for edge deployment, highlighting a viable path toward deploying high-performance, low-power intelligence on the edge.
Abstract (translated)
脉冲神经网络（SNN）作为卷积神经网络（CNN）的节能替代方案，特别是在边缘智能领域中得到了越来越多的关注。然而，先前的研究主要集中在大规模模型上，忽略了轻量级 CNN 到 SNN 转换管道的设计和评估。在这篇论文中，我们介绍了首次系统性地对通过将紧凑型 CNN 架构转换为脉冲网络获得的轻量级 SNN 进行基准测试的结果。在这些转换后的 SNN 中，激活被建模为使用泄露积分放电（LIF）神经元，并采用统一设置下的替代梯度下降法进行训练。 我们构建了 ShuffleNet、SqueezeNet、MnasNet 和 MixNet 的脉冲版本，并针对 CIFAR-10、CIFAR-100 和 TinyImageNet 数据集进行了评估，包括准确率、F1 分数、参数数量、计算复杂性以及能耗等指标。我们的结果显示，在保持竞争力的准确性的前提下，SNN 可以比其 CNN 对应模型高 15.7 倍的能量效率。在这其中，SqueezeNet 的 SNN 变体在所有轻量级 SNN 中表现最优。 为了进一步优化这个模型，我们应用了一种结构化剪枝策略，删除了整个冗余模块，从而得到了一个剪枝架构：SNN-SqueezeNet-P。相比原始的 SNN-SqueezeNet，此剪枝模型将 CIFAR-10 准确率提高了 6%，参数减少了 19%。尤为重要的是，它缩小了与 CNN-SqueezeNet 的差距，在几乎相同的准确度（低仅 1%）下，由于稀疏脉冲驱动的计算，能量消耗降低了 88.1%。 这些发现共同证明轻量级 SNN 在边缘部署中是实用且低能耗的选择，并为在边缘部署高性能、低功耗智能提供了一条可行路径。
URL
https://arxiv.org/abs/2602.09717
PDF
-
MIDAS: Mosaic Input-Specific Differentiable Architecture Search
Abstract
Differentiable Neural Architecture Search (NAS) provides efficient, gradient-based methods for automatically designing neural networks, yet its adoption remains limited in practice. We present MIDAS, a novel approach that modernizes DARTS by replacing static architecture parameters with dynamic, input-specific parameters computed via self-attention. To improve robustness, MIDAS (i) localizes the architecture selection by computing it separately for each spatial patch of the activation map, and (ii) introduces a parameter-free, topology-aware search space that models node connectivity and simplifies selecting the two incoming edges per node. We evaluate MIDAS on the DARTS, NAS-Bench-201, and RDARTS search spaces. In DARTS, it reaches 97.42% top-1 on CIFAR-10 and 83.38% on CIFAR-100. In NAS-Bench-201, it consistently finds globally optimal architectures. In RDARTS, it sets the state of the art on two of four search spaces on CIFAR-10. We further analyze why MIDAS works, showing that patchwise attention improves discrimination among candidate operations, and the resulting input-specific parameter distributions are class-aware and predominantly unimodal, providing reliable guidance for decoding.
Abstract (translated)
可微神经架构搜索（NAS）提供了一种自动设计神经网络的高效、基于梯度的方法，但其在实践中应用仍然有限。我们提出了MIDAS，这是一种通过用自注意力计算出的动态且特定于输入的参数来替换静态架构参数，从而对DARTS进行了现代化改进的新方法。为了提高鲁棒性，MIDAS（i）通过对激活图中的每个空间补丁分别进行架构选择来进行局部化，并（ii）引入了一种无参数、拓扑感知的搜索空间，该空间建模节点连接并简化了每个节点选择两个输入边的过程。 我们在DARTS、NAS-Bench-201和RDARTS搜索空间上评估了MIDAS。在DARTS中，它在CIFAR-10数据集上的顶级准确率为97.42%，在CIFAR-100数据集上的准确率为83.38%。在NAS-Bench-201中，它始终能找到全局最优架构。在RDARTS上，它在四个搜索空间中的两个中为CIFAR-10设定了最新的技术水平。 我们进一步分析了MIDAS为何有效：补丁级注意力提高了对候选操作的区分能力，并且生成的特定于输入的参数分布是类别感知的并且主要是单峰的，这为解码提供了可靠的指导。
URL
https://arxiv.org/abs/2602.17700
PDF
-
TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation
2026-02-02 22:49:39Prajna G. Malettira, Manish Nagaraj, Arjun Roy, Shubham Negi, Kaushik RoyAbstract
Structured pruning is essential for efficient deployment of Large Language Models (LLMs). The varying sensitivity of LLM sub-blocks to pruning necessitates the identification of optimal non-uniformly pruned models. Existing methods evaluate the importance of layers, attention heads, or weight channels in isolation. Such localized focus ignores the complex global structural dependencies that exist across the model. Training-aware structured pruning addresses global dependencies, but its computational cost can be just as expensive as post-pruning training. To alleviate the computational burden of training-aware pruning and capture global structural dependencies, we propose TraceNAS, a training-free Neural Architecture Search (NAS) framework that jointly explores structured pruning of LLM depth and width. TraceNAS identifies pruned models that maintain a high degree of loss landscape alignment with the pretrained model using a scale-invariant zero-shot proxy, effectively selecting models that exhibit maximal performance potential during post-pruning training. TraceNAS is highly efficient, enabling high-fidelity discovery of pruned models on a single GPU in 8.5 hours, yielding a 10$\times$ reduction in GPU-hours compared to training-aware methods. Evaluations on the Llama and Qwen families demonstrate that TraceNAS is competitive with training-aware baselines across commonsense and reasoning benchmarks.
Abstract (translated)
结构化剪枝对于大型语言模型（LLM）的高效部署至关重要。由于LLM子模块对剪枝敏感度不同，因此需要识别最优的非均匀剪枝模型。现有的方法分别评估层、注意力头或权重通道的重要性，这种局部关注忽略了模型内部复杂的全局结构性依赖关系。训练感知结构化剪枝可以解决这一问题，但其计算成本可能与后剪枝训练一样昂贵。为了减轻训练感知剪枝的计算负担并捕获全局结构性依赖关系，我们提出了TraceNAS，这是一种无需训练的神经架构搜索（NAS）框架，它同时探索LLM深度和宽度的结构化剪枝。TraceNAS通过使用比例不变的零样本代理来识别与预训练模型损失景观高度对齐的剪枝模型，有效地选择在后剪枝训练中表现出最大性能潜力的模型。TraceNAS效率极高，在单个GPU上仅需8.5小时即可发现高质量的剪枝模型，相比训练感知方法减少了10倍的GPU时间消耗。在Llama和Qwen家族上的评估表明，TraceNAS在常识和推理基准测试方面与训练感知基线具有竞争力。
URL
https://arxiv.org/abs/2602.02891
PDF
-
UL-UNAS: Ultra-Lightweight U-Nets for Real-Time Speech Enhancement via Network Architecture Search
2026-02-01 05:44:16Xiaobin Rong, Leyan Yang, Dahan Wang, Yuxiang Hu, Changbao Zhu, Kai Chen, Jing LuAbstract
Lightweight models are essential for real-time speech enhancement applications. In recent years, there has been a growing trend toward developing increasingly compact models for speech enhancement. In this paper, we propose an Ultra-Lightweight U-net optimized by Network Architecture Search (UL-UNAS), which is suitable for implementation in low-footprint devices. Firstly, we explore the application of various efficient convolutional blocks within the U-Net framework to identify the most promising candidates. Secondly, we introduce two boosting components to enhance the capacity of these convolutional blocks: a novel activation function named affine PReLU and a causal time-frequency attention module. Furthermore, we leverage neural architecture search to discover an optimal architecture within our carefully designed search space. By integrating the above strategies, UL-UNAS not only significantly outperforms the latest ultra-lightweight models with the same or lower computational complexity, but also delivers competitive performance compared to recent baseline models that require substantially higher computational resources. Source code and audio demos are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2503.00340
PDF
-
SQUAD: Scalable Quorum Adaptive Decisions via ensemble of early exit neural networks
2026-01-30 08:32:33Matteo Gambella, Fabrizio Pittorino, Giuliano Casale, Manuel RoveriAbstract
Early-exit neural networks have become popular for reducing inference latency by allowing intermediate predictions when sufficient confidence is achieved. However, standard approaches typically rely on single-model confidence thresholds, which are frequently unreliable due to inherent calibration issues. To address this, we introduce SQUAD (Scalable Quorum Adaptive Decisions), the first inference scheme that integrates early-exit mechanisms with distributed ensemble learning, improving uncertainty estimation while reducing the inference time. Unlike traditional methods that depend on individual confidence scores, SQUAD employs a quorum-based stopping criterion on early-exit learners by collecting intermediate predictions incrementally in order of computational complexity until a consensus is reached and halting the computation at that exit if the consensus is statistically significant. To maximize the efficacy of this voting mechanism, we also introduce QUEST (Quorum Search Technique), a Neural Architecture Search method to select early-exit learners with optimized hierarchical diversity, ensuring learners are complementary at every intermediate layer. This consensus-driven approach yields statistically robust early exits, improving the test accuracy up to 5.95% compared to state-of-the-art dynamic solutions with a comparable computational cost and reducing the inference latency up to 70.60% compared to static ensembles while maintaining a good accuracy.
Abstract (translated)
URL
https://arxiv.org/abs/2601.22711
PDF
-
Reinforcement Learning for Quantum Technology
2026-01-26 20:47:48Marin Bukov, Florian MarquardtAbstract
Many challenges arising in Quantum Technology can be successfully addressed using a set of machine learning algorithms collectively known as reinforcement learning (RL), based on adaptive decision-making through interaction with the quantum device. After a concise and intuitive introduction to RL aimed at a broad physics readership, we discuss the key ideas and core concepts in reinforcement learning with a particular focus on quantum systems. We then survey recent progress in RL in all relevant areas. We discuss state preparation in few- and many-body quantum systems, the design and optimization of high-fidelity quantum gates, and the automated construction of quantum circuits, including applications to variational quantum eigensolvers and architecture search. We further highlight the interactive capabilities of RL agents, emphasizing recent progress in quantum feedback control and quantum error correction, and briefly discuss quantum reinforcement learning as well as applications to quantum metrology. The review concludes with a discussion of open challenges -- such as scalability, interpretability, and integration with experimental platforms -- and outlines promising directions for future research. Throughout, we highlight experimental implementations that exemplify the increasing role of reinforcement learning in shaping the development of quantum technologies.
Abstract (translated)
URL
https://arxiv.org/abs/2601.18953
PDF
-
Differentiable Architecture Search for Adversarially Robust Quantum Computer Vision
2026-01-26 01:21:44Mohamed Afane, Quanjiang Long, Haoting Shen, Ying Mao, Junaid Farooq, Ying Wang, Juntao ChenAbstract
Current quantum neural networks suffer from extreme sensitivity to both adversarial perturbations and hardware noise, creating a significant barrier to real-world deployment. Existing robustness techniques typically sacrifice clean accuracy or require prohibitive computational resources. We propose a hybrid quantum-classical Differentiable Quantum Architecture Search (DQAS) framework that addresses these limitations by jointly optimizing circuit structure and robustness through gradient-based methods. Our approach enhances traditional DQAS with a lightweight Classical Noise Layer applied before quantum processing, enabling simultaneous optimization of gate selection and noise parameters. This design preserves the quantum circuit's integrity while introducing trainable perturbations that enhance robustness without compromising standard performance. Experimental validation on MNIST, FashionMNIST, and CIFAR datasets shows consistent improvements in both clean and adversarial accuracy compared to existing quantum architecture search methods. Under various attack scenarios, including Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), Basic Iterative Method (BIM), and Momentum Iterative Method (MIM), and under realistic quantum noise conditions, our hybrid framework maintains superior performance. Testing on actual quantum hardware confirms the practical viability of discovered architectures. These results demonstrate that strategic classical preprocessing combined with differentiable quantum architecture optimization can significantly enhance quantum neural network robustness while maintaining computational efficiency.
Abstract (translated)
URL
https://arxiv.org/abs/2601.18058
PDF
-
Dynamic Meta-Ensemble Framework for Efficient and Accurate Deep Learning in Plant Leaf Disease Detection on Resource-Constrained Edge Devices
2026-01-24 03:57:49Weloday Fikadu Moges, Jianmei Su, Amin WaqasAbstract
Deploying deep learning models for plant disease detection on edge devices such as IoT sensors, smartphones, and embedded systems is severely constrained by limited computational resources and energy budgets. To address this challenge, we introduce a novel Dynamic Meta-Ensemble Framework (DMEF) for high-accuracy plant disease diagnosis under resource constraints. DMEF employs an adaptive weighting mechanism that dynamically combines the predictions of three lightweight convolutional neural networks (MobileNetV2, NASNetMobile, and InceptionV3) by optimizing a trade-off between accuracy improvements (DeltaAcc) and computational efficiency (model size). During training, the ensemble weights are updated iteratively, favoring models exhibiting high performance and low complexity. Extensive experiments on benchmark datasets for potato and maize diseases demonstrate state-of-the-art classification accuracies of 99.53% and 96.61%, respectively, surpassing standalone models and static ensembles by 2.1% and 6.3%. With computationally efficient inference latency (<75ms) and a compact footprint (<1 million parameters), DMEF shows strong potential for edge-based agricultural monitoring, suggesting viability for scalable crop disease management. This bridges the gap between high-accuracy AI and practical field applications.
Abstract (translated)
URL
https://arxiv.org/abs/2601.17290
PDF
-
DeepFedNAS: A Unified Framework for Principled, Hardware-Aware, and Predictor-Free Federated Neural Architecture Search
Abstract
Federated Neural Architecture Search (FedNAS) aims to automate model design for privacy-preserving Federated Learning (FL) but currently faces two critical bottlenecks: unguided supernet training that yields suboptimal models, and costly multi-hour pipelines for post-training subnet discovery. We introduce DeepFedNAS, a novel, two-phase framework underpinned by a principled, multi-objective fitness function that synthesizes mathematical network design with architectural heuristics. Enabled by a re-engineered supernet, DeepFedNAS introduces Federated Pareto Optimal Supernet Training, which leverages a pre-computed Pareto-optimal cache of high-fitness architectures as an intelligent curriculum to optimize shared supernet weights. Subsequently, its Predictor-Free Search Method eliminates the need for costly accuracy surrogates by utilizing this fitness function as a direct, zero-cost proxy for accuracy, enabling on-demand subnet discovery in mere seconds. DeepFedNAS achieves state-of-the-art accuracy (e.g., up to 1.21% absolute improvement on CIFAR-100), superior parameter and communication efficiency, and a substantial ~61x speedup in total post-training search pipeline time. By reducing the pipeline from over 20 hours to approximately 20 minutes (including initial cache generation) and enabling 20-second individual subnet searches, DeepFedNAS makes hardware-aware FL deployments instantaneous and practical. The complete source code and experimental scripts are available at: this https URL
Abstract (translated)
联邦神经架构搜索（FedNAS）旨在为隐私保护的联邦学习（FL）自动化模型设计，但目前面临两个关键瓶颈：缺乏指导的超网训练导致次优模型产生，以及耗时数小时的后训练子网络发现流程。我们提出了DeepFedNAS，这是一个新颖的两阶段框架，基于一个原则性的多目标适应性函数，将数学网络设计与架构启发式相结合。借助重新设计的超网，DeepFedNAS引入了联邦帕累托最优超网训练方法，该方法利用预计算的高适应性架构帕累托缓存作为智能课程表来优化共享超网权重。随后，其无需预测器搜索方法通过使用此适应性函数直接替代精度代理（零成本），从而消除昂贵的精度代理需求，实现按需子网络发现仅需几秒即可完成。 DeepFedNAS实现了最先进的准确性（例如，在CIFAR-100上高达1.21%绝对改进）、卓越的参数和通信效率，并且在总后训练搜索流水线时间上提高了约61倍的速度。通过将流程从超过20小时缩短至大约20分钟（包括初始缓存生成）并实现每个子网络仅需20秒的单独搜索，DeepFedNAS使硬件感知FL部署即时和实用化。 完整的源代码和实验脚本可在以下网址获取：[此URL]
URL
https://arxiv.org/abs/2601.15127
PDF
-
Automating Parameter Selection in Deep Image Prior for Fluorescence Microscopy Image Denoising via Similarity-Based Parameter Transfer
2026-01-17 13:47:41Lina Meyer, Felix Wissel, Tobias Knopp, Susanne Pfefferle, Ralf Fliegert, Maximilian Sandmann, Liana Uebler, Franziska M\"ockl, Bj\"orn-Philipp Diercks, David Lohr, Ren\'e WernerAbstract
Unsupervised deep image prior (DIP) addresses shortcomings of training data requirements and limited generalization associated with supervised deep learning. The performance of DIP depends on the network architecture and the stopping point of its iterative process. Optimizing these parameters for a new image requires time, restricting DIP application in domains where many images need to be processed. Focusing on fluorescence microscopy data, we hypothesize that similar images share comparable optimal parameter configurations for DIP-based denoising, potentially enabling optimization-free DIP for fluorescence microscopy. We generated a calibration (n=110) and validation set (n=55) of semantically different images from an open-source dataset for a network architecture search targeted towards ideal U-net architectures and stopping points. The calibration set represented our transfer basis. The validation set enabled the assessment of which image similarity criterion yields the best results. We then implemented AUTO-DIP, a pipeline for automatic parameter transfer, and compared it to the originally published DIP configuration (baseline) and a state-of-the-art image-specific variational denoising approach. We show that a parameter transfer from the calibration dataset to a test image based on only image metadata similarity (e.g., microscope type, imaged specimen) leads to similar and better performance than a transfer based on quantitative image similarity measures. AUTO-DIP outperforms the baseline DIP (DIP with original DIP parameters) as well as the variational denoising approaches for several open-source test datasets of varying complexity, particularly for very noisy inputs. Applications to locally acquired fluorescence microscopy images further proved superiority of AUTO-DIP.
Abstract (translated)
无监督深度图像先验（DIP）解决了有监督深度学习中训练数据需求和泛化能力有限的问题。DIP的性能取决于网络架构及其迭代过程的停止点。为了一个新的图像优化这些参数需要时间，这限制了在处理大量图像领域使用DIP的应用范围。针对荧光显微镜数据，我们假设相似的图像具有类似的最佳参数配置，这有可能实现无需优化的DIP应用。为此，我们在一个开源的数据集中生成了一个校准集（n=110）和验证集（n=55），这些集合包含语义上不同的图像，并针对理想的U-net架构和停止点进行了网络结构搜索。校准集代表了我们的迁移基础，而验证集则用于评估哪种图像相似性标准能产生最佳结果。 我们随后实施了一种自动参数转移管道AUTO-DIP，并将其与最初发布的DIP配置（基准线）以及一种最先进的特定图像变分去噪方法进行比较。结果显示，仅基于图像元数据相似性（如显微镜类型、成像标本等），从校准数据集向测试图像传输参数可以实现类似甚至更好的性能，相比基于定量图像相似度测量的转移方法而言。 对于几个不同复杂程度的开源测试数据集，AUTO-DIP优于基准DIP（使用原始DIP参数）以及变分去噪方法，特别是在非常嘈杂的数据输入方面。应用到本地采集的荧光显微镜图像中进一步证明了AUTO-DIP的优势。
URL
https://arxiv.org/abs/2601.12055
PDF
-
Building Production-Ready Probes For Gemini
2026-01-16 18:54:29J\'anos Kram\'ar, Joshua Engels, Zheng Wang, Bilal Chughtai, Rohin Shah, Neel Nanda, Arthur ConmyAbstract
Frontier language model capabilities are improving rapidly. We thus need stronger mitigations against bad actors misusing increasingly powerful systems. Prior work has shown that activation probes may be a promising misuse mitigation technique, but we identify a key remaining challenge: probes fail to generalize under important production distribution shifts. In particular, we find that the shift from short-context to long-context inputs is difficult for existing probe architectures. We propose several new probe architecture that handle this long-context distribution shift. We evaluate these probes in the cyber-offensive domain, testing their robustness against various production-relevant shifts, including multi-turn conversations, static jailbreaks, and adaptive red teaming. Our results demonstrate that while multimax addresses context length, a combination of architecture choice and training on diverse distributions is required for broad generalization. Additionally, we show that pairing probes with prompted classifiers achieves optimal accuracy at a low cost due to the computational efficiency of probes. These findings have informed the successful deployment of misuse mitigation probes in user-facing instances of Gemini, Google's frontier language model. Finally, we find early positive results using AlphaEvolve to automate improvements in both probe architecture search and adaptive red teaming, showing that automating some AI safety research is already possible.
Abstract (translated)
前沿语言模型的能力正在迅速提升，因此我们需要更强的措施来防止恶意用户滥用日益强大的系统。先前的研究表明，激活探测器可能是一种有前景的误用缓解技术，但我们识别到一个关键挑战：现有的探测器在面对重要的生产分布变化时无法泛化。特别是我们发现，从短上下文输入向长上下文输入的变化对现有探测器架构来说是一个难题。 为此，我们提出了一些新的探测器架构来处理这种长上下文的分布偏移。我们在网络攻击领域测试了这些探测器的鲁棒性，评估它们在各种生产相关变化中的表现，包括多轮对话、静态破解和适应性的红队行动（模拟对手）。我们的结果显示，虽然多重最大值方法可以解决上下文长度的问题，但要实现广泛泛化，则需要结合架构选择与训练多样化分布。此外，我们还发现将探测器与提示分类器配对可以在低计算成本下达到最佳准确率。 这些研究成果已经指导了谷歌前沿语言模型Gemini中误用缓解探测器的成功部署。最后，我们通过AlphaEvolve早期获得了积极的结果，用于自动化改进探测器架构搜索和适应性红队行动，表明部分AI安全性研究的自动化已经成为可能。
URL
https://arxiv.org/abs/2601.11516
PDF
-
Mobile-friendly Image de-noising: Hardware Conscious Optimization for Edge Application
2026-01-16 09:39:01Srinivas Miriyala, Sowmya Vajrala, Hitesh Kumar, Sravanth Kodavanti, Vikram RajendiranAbstract
Image enhancement is a critical task in computer vision and photography that is often entangled with noise. This renders the traditional Image Signal Processing (ISP) ineffective compared to the advances in deep learning. However, the success of such methods is increasingly associated with the ease of their deployment on edge devices, such as smartphones. This work presents a novel mobile-friendly network for image de-noising obtained with Entropy-Regularized differentiable Neural Architecture Search (NAS) on a hardware-aware search space for a U-Net architecture, which is first-of-its-kind. The designed model has 12% less parameters, with ~2-fold improvement in ondevice latency and 1.5-fold improvement in the memory footprint for a 0.7% drop in PSNR, when deployed and profiled on Samsung Galaxy S24 Ultra. Compared to the SOTA Swin-Transformer for Image Restoration, the proposed network had competitive accuracy with ~18-fold reduction in GMACs. Further, the network was tested successfully for Gaussian de-noising with 3 intensities on 4 benchmarks and real-world de-noising on 1 benchmark demonstrating its generalization ability.
Abstract (translated)
图像增强是计算机视觉和摄影中的一个关键任务，通常会受到噪声的影响。这使得传统的图像信号处理（ISP）在与深度学习的进展相比时显得不够有效。然而，此类方法的成功越来越依赖于其在边缘设备（如智能手机）上部署的简便性。本工作提出了一种新型的手机友好型网络用于图像去噪，该网络通过基于硬件感知搜索空间上的熵正则化可微神经架构搜索(NAS)获得，这是首次实现这一概念的设计。 设计的模型参数减少了12%，在三星Galaxy S24 Ultra上部署和评估时，设备端延迟提高了约两倍，内存占用减少了一半，而PSNR仅下降了0.7%。与最先进的Swin-Transformer图像恢复方法相比，该提议的网络具有竞争性的准确性，并且GMACs减少了大约18倍。 此外，在四个基准测试上成功对三个强度级别的高斯去噪进行了测试，并在一个真实世界基准测试中验证了其去噪能力，从而证明了该模型具备良好的泛化能力。
URL
https://arxiv.org/abs/2601.11684
PDF
-
Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction
2026-01-16 02:01:40Xiaojie Xia, Huigang Zhang, Chaoliang Zhong, Jun Sun, Yusuke OishiAbstract
Transformer architectures deliver state-of-the-art accuracy via dense full-attention, but their quadratic time and memory complexity with respect to sequence length limits practical deployment. Linear attention mechanisms offer linear or near-linear scaling yet often incur performance degradation. Hybrid models that integrate full and linear attention layers promise a balance between efficiency and expressiveness, but face two major challenges: training such hybrid models from scratch is computationally expensive, and manually designing the optimal placement of attention types is highly nontrivial. We address both issues by first transferring weights from the pretrained full-attention modules to its linear attention counterparts through blockwise local distillation, and second, introducing a greedy layer replacement strategy that iteratively substitutes full attention blocks with linear ones while monitoring validation performance on the target task. This yields a task-specific hybrid model in a single efficient pass, without costly re-training or neural architecture search, and can be applied to any pretrained full-attention backbone for diverse downstream tasks.
Abstract (translated)
翻译如下： 基于Transformer的架构通过密集全注意力机制提供了最先进的准确性，但它们的时间和内存复杂度与序列长度呈二次关系，这限制了其实用部署。线性注意力机制则提供线性的或接近线性的缩放比例，尽管如此，往往会导致性能下降。混合模型将全注意力层和线性注意力层结合在一起，旨在平衡效率和表达能力，但面临着两大挑战：从头开始训练此类混合模型计算成本高昂，而手动设计最佳注意力类型放置方案极其复杂且不切实际。 为了解决这两个问题，我们首先通过分块本地蒸馏的方式，将预训练的全注意力模块权重转移到其线性注意力对应模块中。其次，我们引入了一种贪婪层替换策略：该策略在监测目标任务验证性能的同时，迭代地用线性注意力模块替代全注意力模块。这可以在一次高效的遍历过程中生成特定任务的混合模型，而无需耗时的重新训练或神经架构搜索，并且可以应用于任何预训练的全注意力骨干网络以应对各种下游任务。
URL
https://arxiv.org/abs/2601.11667
PDF
