Gradient_Descent
-
Robust Causal Discovery under Imperfect Structural Constraints
2025-11-10 07:27:08Zidong Wang, Xi Lin, Chuchao He, Xiaoguang GaoAbstract
Robust causal discovery from observational data under imperfect prior knowledge remains a significant and largely unresolved challenge. Existing methods typically presuppose perfect priors or can only handle specific, pre-identified error types. And their performance degrades substantially when confronted with flawed constraints of unknown location and type. This decline arises because most of them rely on inflexible and biased thresholding strategies that may conflict with the data distribution. To overcome these limitations, we propose to harmonizes knowledge and data through prior alignment and conflict resolution. First, we assess the credibility of imperfect structural constraints through a surrogate model, which then guides a sparse penalization term measuring the loss between the learned and constrained adjacency matrices. We theoretically prove that, under ideal assumption, the knowledge-driven objective aligns with the data-driven objective. Furthermore, to resolve conflicts when this assumption is violated, we introduce a multi-task learning framework optimized via multi-gradient descent, jointly minimizing both objectives. Our proposed method is robust to both linear and nonlinear settings. Extensive experiments, conducted under diverse noise conditions and structural equation model types, demonstrate the effectiveness and efficiency of our method under imperfect structural constraints.
Abstract (translated)
URL
https://arxiv.org/abs/2511.06790
PDF
-
Census-Based Population Autonomy For Distributed Robotic Teaming
2025-11-04 00:32:23Tyler M. Paine, Anastasia Bizyaeva, Michael R. BenjaminAbstract
Collaborating teams of robots show promise due in their ability to complete missions more efficiently and with improved robustness, attributes that are particularly useful for systems operating in marine environments. A key issue is how to model, analyze, and design these multi-robot systems to realize the full benefits of collaboration, a challenging task since the domain of multi-robot autonomy encompasses both collective and individual behaviors. This paper introduces a layered model of multi-robot autonomy that uses the principle of census, or a weighted count of the inputs from neighbors, for collective decision-making about teaming, coupled with multi-objective behavior optimization for individual decision-making about actions. The census component is expressed as a nonlinear opinion dynamics model and the multi-objective behavior optimization is accomplished using interval programming. This model can be reduced to recover foundational algorithms in distributed optimization and control, while the full model enables new types of collective behaviors that are useful in real-world scenarios. To illustrate these points, a new method for distributed optimization of subgroup allocation is introduced where robots use a gradient descent algorithm to minimize portions of the cost functions that are locally known, while being influenced by the opinion states from neighbors to account for the unobserved costs. With this method the group can collectively use the information contained in the Hessian matrix of the total global cost. The utility of this model is experimentally validated in three categorically different experiments with fleets of autonomous surface vehicles: an adaptive sampling scenario, a high value unit protection scenario, and a competitive game of capture the flag.
Abstract (translated)
URL
https://arxiv.org/abs/2511.02147
PDF
-
Stein-based Optimization of Sampling Distributions in Model Predictive Path Integral Control
2025-11-03 19:41:51Jace Aldrich, Odest Chadwicke JenkinsAbstract
This paper presents a novel method for Model Predictive Path Integral (MPPI) control that optimizes sample generation towards an optimal trajectory through Stein Variational Gradient Descent (SVGD). MPPI is traditionally reliant on randomly sampled trajectories, often by a Gaussian distribution. The result can lead to sample deprivation, under-representing the space of possible trajectories, and yield suboptimal results. Through introducing SVGD updates in between MPPI environment steps, we present Stein-Optimized Path-Integral Inference (SOPPI), an MPPI/SVGD algorithm that can dynamically update noise distributions at runtime to shape a more optimal representation without an excessive increase in computational requirements. We demonstrate the efficacy of our method systems ranging from a Cart-Pole to a two-dimensional bipedal walking task, indicating improved performance above standard MPPI across a range of hyper-parameters and demonstrate feasibility at lower particle counts. We discuss the applicability of this MPPI/SVGD method to higher degree-of-freedom systems, as well as its potential to new developments in state-of-the-art differentiable simulators.
Abstract (translated)
URL
https://arxiv.org/abs/2511.02015
PDF
-
Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering
2025-11-03 16:00:45Hossein Abdi, Mingfei Sun, Wei PanAbstract
Vision-language pre-trained models, such as CLIP, have established new benchmarks in multimodal data mining. In such models, few-shot fine-tuning is a major challenge to achieve optimal performance on both in-distribution (ID) and out-of-distribution (OOD) datasets, especially when labeled data is scarce. Most existing fine-tuning approaches rely on first-order gradient-based optimizers, which typically suffer from slow convergence, sensitivity to step-size hyperparameters, and poor generalization in OOD settings. In contrast, second-order methods utilize local curvature information of the loss landscape to adjust the update step size. This is particularly beneficial for CLIP models, whose non-convex loss functions often contain sharp critical points. In such cases, natural gradient direction can offer more substantial and efficient per-iteration updates when fine-tuning with limited data. Natural Gradient Descent (NGD) is obtained by preconditioning the standard gradient with the inverse Fisher Information Matrix (FIM), which is computationally expensive for large models. To address this, we propose a Bayesian approximation of NGD using a Kalman filter for CLIP models. Our method combines the benefits of second-order optimization with Bayesian inference, which enhances generalization while providing uncertainty quantification. Extensive experiments conducted on diverse image classification datasets demonstrate that our algorithm consistently achieves superior--or comparable--ID performance and improved OOD robustness compared to state-of-the-art baselines. To the best of our knowledge, this work represents the first successful application of Kalman filtering to fine-tuning CLIP-based models, which enables more robust and efficient learning in vision-language tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2511.01694
PDF
-
Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition
2025-10-31 17:24:58Shuyan Lyu, Zhanzimo Wu, Junliang DuAbstract
Modern deep neural networks (DNNs) are typically trained with a global cross-entropy loss in a supervised end-to-end manner: neurons need to store their outgoing weights; training alternates between a forward pass (computation) and a top-down backward pass (learning) which is biologically implausible. Alternatively, greedy layer-wise training eliminates the need for cross-entropy loss and backpropagation. By avoiding the computation of intermediate gradients and the storage of intermediate outputs, it reduces memory usage and helps mitigate issues such as vanishing or exploding gradients. However, most existing layer-wise training approaches have been evaluated only on relatively small datasets with simple deep architectures. In this paper, we first systematically analyze the training dynamics of popular convolutional neural networks (CNNs) trained by stochastic gradient descent (SGD) through an information-theoretic lens. Our findings reveal that networks converge layer-by-layer from bottom to top and that the flow of information adheres to a Markov information bottleneck principle. Building on these observations, we propose a novel layer-wise training approach based on the recently developed deterministic information bottleneck (DIB) and the matrix-based Rényi's $\alpha$-order entropy functional. Specifically, each layer is trained jointly with an auxiliary classifier that connects directly to the output layer, enabling the learning of minimal sufficient task-relevant representations. We empirically validate the effectiveness of our training procedure on CIFAR-10 and CIFAR-100 using modern deep CNNs and further demonstrate its applicability to a practical task involving traffic sign recognition. Our approach not only outperforms existing layer-wise training baselines but also achieves performance comparable to SGD.
Abstract (translated)
URL
https://arxiv.org/abs/2510.27651
PDF
-
Spiking Neural Networks: The Future of Brain-Inspired Computing
2025-10-31 11:14:59Sales G. Aribe JrAbstract
Spiking Neural Networks (SNNs) represent the latest generation of neural computation, offering a brain-inspired alternative to conventional Artificial Neural Networks (ANNs). Unlike ANNs, which depend on continuous-valued signals, SNNs operate using distinct spike events, making them inherently more energy-efficient and temporally dynamic. This study presents a comprehensive analysis of SNN design models, training algorithms, and multi-dimensional performance metrics, including accuracy, energy consumption, latency, spike count, and convergence behavior. Key neuron models such as the Leaky Integrate-and-Fire (LIF) and training strategies, including surrogate gradient descent, ANN-to-SNN conversion, and Spike-Timing Dependent Plasticity (STDP), are examined in depth. Results show that surrogate gradient-trained SNNs closely approximate ANN accuracy (within 1-2%), with faster convergence by the 20th epoch and latency as low as 10 milliseconds. Converted SNNs also achieve competitive performance but require higher spike counts and longer simulation windows. STDP-based SNNs, though slower to converge, exhibit the lowest spike counts and energy consumption (as low as 5 millijoules per inference), making them optimal for unsupervised and low-power tasks. These findings reinforce the suitability of SNNs for energy-constrained, latency-sensitive, and adaptive applications such as robotics, neuromorphic vision, and edge AI systems. While promising, challenges persist in hardware standardization and scalable training. This study concludes that SNNs, with further refinement, are poised to propel the next phase of neuromorphic computing.
Abstract (translated)
URL
https://arxiv.org/abs/2510.27379
PDF
-
Non-Convex Over-the-Air Heterogeneous Federated Learning: A Bias-Variance Trade-off
2025-10-30 17:22:57Muhammad Faraz Ul Abrar, Nicol\`o MichelusiAbstract
Over-the-air (OTA) federated learning (FL) has been well recognized as a scalable paradigm that exploits the waveform superposition of the wireless multiple-access channel to aggregate model updates in a single use. Existing OTA-FL designs largely enforce zero-bias model updates by either assuming \emph{homogeneous} wireless conditions (equal path loss across devices) or forcing zero-bias updates to guarantee convergence. Under \emph{heterogeneous} wireless scenarios, however, such designs are constrained by the weakest device and inflate the update variance. Moreover, prior analyses of biased OTA-FL largely address convex objectives, while most modern AI models are highly non-convex. Motivated by these gaps, we study OTA-FL with stochastic gradient descent (SGD) for general smooth non-convex objectives under wireless heterogeneity. We develop novel OTA-FL SGD updates that allow a structured, time-invariant model bias while facilitating reduced variance updates. We derive a finite-time stationarity bound (expected time average squared gradient norm) that explicitly reveals a bias-variance trade-off. To optimize this trade-off, we pose a non-convex joint OTA power-control design and develop an efficient successive convex approximation (SCA) algorithm that requires only statistical CSI at the base station. Experiments on a non-convex image classification task validate the approach: the SCA-based design accelerates convergence via an optimized bias and improves generalization over prior OTA-FL baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2510.26722
PDF
-
VISAT: Benchmarking Adversarial and Distribution Shift Robustness in Traffic Sign Recognition with Visual Attributes
2025-10-29 18:31:21Simon Yu, Peilin Yu, Hongbo Zheng, Huajie Shao, Han Zhao, Lui ShaAbstract
We present VISAT, a novel open dataset and benchmarking suite for evaluating model robustness in the task of traffic sign recognition with the presence of visual attributes. Built upon the Mapillary Traffic Sign Dataset (MTSD), our dataset introduces two benchmarks that respectively emphasize robustness against adversarial attacks and distribution shifts. For our adversarial attack benchmark, we employ the state-of-the-art Projected Gradient Descent (PGD) method to generate adversarial inputs and evaluate their impact on popular models. Additionally, we investigate the effect of adversarial attacks on attribute-specific multi-task learning (MTL) networks, revealing spurious correlations among MTL tasks. The MTL networks leverage visual attributes (color, shape, symbol, and text) that we have created for each traffic sign in our dataset. For our distribution shift benchmark, we utilize ImageNet-C's realistic data corruption and natural variation techniques to perform evaluations on the robustness of both base and MTL models. Moreover, we further explore spurious correlations among MTL tasks through synthetic alterations of traffic sign colors using color quantization techniques. Our experiments focus on two major backbones, ResNet-152 and ViT-B/32, and compare the performance between base and MTL models. The VISAT dataset and benchmarking framework contribute to the understanding of model robustness for traffic sign recognition, shedding light on the challenges posed by adversarial attacks and distribution shifts. We believe this work will facilitate advancements in developing more robust models for real-world applications in autonomous driving and cyber-physical systems.
Abstract (translated)
URL
https://arxiv.org/abs/2510.26833
PDF
-
The Neural Differential Manifold: An Architecture with Explicit Geometric Structure
2025-10-29 02:24:27Di ZhangAbstract
This paper introduces the Neural Differential Manifold (NDM), a novel neural network architecture that explicitly incorporates geometric structure into its fundamental design. Departing from conventional Euclidean parameter spaces, the NDM re-conceptualizes a neural network as a differentiable manifold where each layer functions as a local coordinate chart, and the network parameters directly parameterize a Riemannian metric tensor at every point. The architecture is organized into three synergistic layers: a Coordinate Layer implementing smooth chart transitions via invertible transformations inspired by normalizing flows, a Geometric Layer that dynamically generates the manifold's metric through auxiliary sub-networks, and an Evolution Layer that optimizes both task performance and geometric simplicity through a dual-objective loss function. This geometric regularization penalizes excessive curvature and volume distortion, providing intrinsic regularization that enhances generalization and robustness. The framework enables natural gradient descent optimization aligned with the learned manifold geometry and offers unprecedented interpretability by endowing internal representations with clear geometric meaning. We analyze the theoretical advantages of this approach, including its potential for more efficient optimization, enhanced continual learning, and applications in scientific discovery and controllable generative modeling. While significant computational challenges remain, the Neural Differential Manifold represents a fundamental shift towards geometrically structured, interpretable, and efficient deep learning systems.
Abstract (translated)
URL
https://arxiv.org/abs/2510.25113
PDF
-
An Analytic Theory of Quantum Imaginary Time Evolution
2025-10-26 01:43:55Min Chen, Bingzhi Zhang, Quntao Zhuang, Junyu LiuAbstract
Quantum imaginary time evolution (QITE) algorithm is one of the most promising variational quantum algorithms (VQAs), bridging the current era of Noisy Intermediate-Scale Quantum devices and the future of fully fault-tolerant quantum computing. Although practical demonstrations of QITE and its potential advantages over the general VQA trained with vanilla gradient descent (GD) in certain tasks have been reported, a first-principle, theoretical understanding of QITE remains limited. Here, we aim to develop an analytic theory for the dynamics of QITE. First, we show that QITE can be interpreted as a form of a general VQA trained with Quantum Natural Gradient Descent (QNGD), where the inverse quantum Fisher information matrix serves as the learning-rate tensor. This equivalence is established not only at the level of gradient update rules, but also through the action principle: the variational principle can be directly connected to the geometric geodesic distance in the quantum Fisher information metric, up to an integration constant. Second, for wide quantum neural networks, we employ the quantum neural tangent kernel framework to construct an analytic model for QITE. We prove that QITE always converges faster than GD-based VQA, though this advantage is suppressed by the exponential growth of Hilbert space dimension. This helps explain certain experimental results in quantum computational chemistry. Our theory encompasses linear, quadratic, and more general loss functions. We validate the analytic results through numerical simulations. Our findings establish a theoretical foundation for QITE dynamics and provide analytic insights for the first-principle design of variational quantum algorithms.
Abstract (translated)
URL
https://arxiv.org/abs/2510.22481
PDF
-
Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations
Abstract
In this paper, we explore the two-point zeroth-order gradient estimator and identify the distribution of random perturbations that minimizes the estimator's asymptotic variance as the perturbation stepsize tends to zero. We formulate it as a constrained functional optimization problem over the space of perturbation distributions. Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length. While existing research has largely focused on fixed-length perturbations, the potential advantages of directional alignment have been overlooked. To address this gap, we delve into the theoretical and empirical properties of the directionally aligned perturbation (DAP) scheme, which adaptively offers higher accuracy along critical directions. Additionally, we provide a convergence analysis for stochastic gradient descent using $\delta$-unbiased random perturbations, extending existing complexity bounds to a wider range of perturbations. Through empirical evaluations on both synthetic problems and practical tasks, we demonstrate that DAPs outperform traditional methods under specific conditions.
Abstract (translated)
在这篇论文中，我们探讨了两点零阶梯度估计器，并确定了随机扰动分布，该分布可以在扰动步长趋向于零时使估计量的渐近方差最小化。我们将这个问题表述为在扰动分布空间上的约束泛函优化问题。我们的研究结果表明，理想的扰动可以与真实梯度的方向对齐，而不仅仅是保持固定长度。虽然现有研究主要集中在固定长度的扰动上，但方向对齐的潜在优势却被忽视了。 为了填补这一空白，我们深入探讨了方向对齐扰动（DAP）方案的理论和实证特性，该方案可以自适应地在关键方向上提供更高的准确性。此外，我们还为使用$\delta$无偏随机扰动进行随机梯度下降提供了收敛性分析，并将现有的复杂度界限扩展到了更广泛的扰动范围内。 通过合成问题和实际任务上的经验评估，我们在特定条件下展示了DAP优于传统方法的性能。
URL
https://arxiv.org/abs/2510.19975
PDF
-
SOCIA-Nabla: Textual Gradient Meets Multi-Agent Orchestration for Automated Simulator Generation
2025-10-21 12:00:00Yuncheng Hua, Sion Weatherhead, Mehdi Jafari, Hao Xue, Flora D. SalimAbstract
In this paper, we present SOCIA-Nabla, an end-to-end, agentic framework that treats simulator construction asinstance optimization over code within a textual computation graph. Specialized LLM-driven agents are embedded as graph nodes, and a workflow manager executes a loss-driven loop: code synthesis -> execution -> evaluation -> code repair. The optimizer performs Textual-Gradient Descent (TGD), while human-in-the-loop interaction is reserved for task-spec confirmation, minimizing expert effort and keeping the code itself as the trainable object. Across three CPS tasks, i.e., User Modeling, Mask Adoption, and Personal Mobility, SOCIA-Nabla attains state-of-the-art overall accuracy. By unifying multi-agent orchestration with a loss-aligned optimization view, SOCIA-Nabla converts brittle prompt pipelines into reproducible, constraint-aware simulator code generation that scales across domains and simulation granularities. This work is under review, and we will release the code soon.
Abstract (translated)
在这篇论文中，我们介绍了SOCIA-Nabla，这是一种端到端、具备代理能力的框架，它将模拟器构造视为文本计算图中的代码实例优化问题。该框架中嵌入了专门由大型语言模型驱动的代理作为图节点，并且一个工作流管理器执行了一个以损失为导向的循环：代码合成 -> 执行 -> 评估 -> 代码修复。优化器采用文本梯度下降（Textual-Gradient Descent, TGD），而人机交互则保留用于任务具体要求确认，从而将专家的努力最小化并将代码本身作为可训练的对象。在三个CPS（Cyber-Physical Systems）任务中，即用户建模、口罩采用和个人移动性方面，SOCIA-Nabla 达到了最先进的总体准确性。通过将多代理编排与损失对齐的优化视角统一起来，SOCIA-Nabla 将脆弱的提示管道转化为可重复且具有约束意识的模拟器代码生成过程，并且该方法可以跨领域和模拟粒度进行扩展。这项工作正在审稿中，我们很快会发布代码。
URL
https://arxiv.org/abs/2510.18551
PDF
-
Learning under Quantization for High-Dimensional Linear Regression
2025-10-21 03:30:11Dechen Zhang, Junwei Su, Difan ZouAbstract
The use of low-bit quantization has emerged as an indispensable technique for enabling the efficient training of large-scale models. Despite its widespread empirical success, a rigorous theoretical understanding of its impact on learning performance remains notably absent, even in the simplest linear regression setting. We present the first systematic theoretical study of this fundamental question, analyzing finite-step stochastic gradient descent (SGD) for high-dimensional linear regression under a comprehensive range of quantization targets: data, labels, parameters, activations, and gradients. Our novel analytical framework establishes precise algorithm-dependent and data-dependent excess risk bounds that characterize how different quantization affects learning: parameter, activation, and gradient quantization amplify noise during training; data quantization distorts the data spectrum; and data and label quantization introduce additional approximation and quantized error. Crucially, we prove that for multiplicative quantization (with input-dependent quantization step), this spectral distortion can be eliminated, and for additive quantization (with constant quantization step), a beneficial scaling effect with batch size emerges. Furthermore, for common polynomial-decay data spectra, we quantitatively compare the risks of multiplicative and additive quantization, drawing a parallel to the comparison between FP and integer quantization methods. Our theory provides a powerful lens to characterize how quantization shapes the learning dynamics of optimization algorithms, paving the way to further explore learning theory under practical hardware constraints.
Abstract (translated)
低比特量化作为一种技术，已成为实现大规模模型高效训练不可或缺的方法。尽管它在经验上取得了广泛的成功，但在最简单的线性回归设置中，对其对学习性能影响的严格理论理解仍然明显缺乏。我们首次系统地研究了这一基本问题，分析了高维线性回归下的有限步随机梯度下降（SGD），涵盖了数据、标签、参数、激活和梯度等全面范围内的量化目标。我们的新型分析框架建立了一套精确的算法依赖性和数据依赖性的额外风险界限，这些界限描述了不同类型的量化如何影响学习过程：参数量化、激活量化和梯度量化在训练过程中放大噪声；数据量化扭曲数据谱；而数据和标签量化则引入额外的近似误差和量化误差。尤为重要的是，我们证明对于乘法量化（具有输入依赖性的量化步长），这种光谱失真可以被消除，并且对于加性量化（具有恒定的量化步长），随着批量大小的增加会形成有益的缩放效应。此外，在常见的多项式衰减数据谱中，我们从定量上比较了乘法和加性量化的风险，将其与浮点数和整数量化方法之间的比较进行了类比。我们的理论为如何通过量化塑造优化算法的学习动态提供了有力的研究视角，并为进一步在实际硬件约束下的学习理论研究铺平了道路。
URL
https://arxiv.org/abs/2510.18259
PDF
-
DDBot: Differentiable Physics-based Digging Robot for Unknown Granular Materials
2025-10-20 09:27:24Xintong Yang, Minglun Wei, Ze Ji, Yu-Kun LaiAbstract
Automating the manipulation of granular materials poses significant challenges due to complex contact dynamics, unpredictable material properties, and intricate system states. Existing approaches often fail to achieve efficiency and accuracy in such tasks. To fill the research gap, this paper studies the small-scale and high-precision granular material digging task with unknown physical properties. A new framework, named differentiable digging robot (DDBot), is proposed to manipulate granular materials, including sand and soil. Specifically, we equip DDBot with a differentiable physics-based simulator, tailored for granular material manipulation, powered by GPU-accelerated parallel computing and automatic differentiation. DDBot can perform efficient differentiable system identification and high-precision digging skill optimisation for unknown granular materials, which is enabled by a differentiable skill-to-action mapping, a task-oriented demonstration method, gradient clipping and line search-based gradient descent. Experimental results show that DDBot can efficiently (converge within 5 to 20 minutes) identify unknown granular material dynamics and optimise digging skills, with high-precision results in zero-shot real-world deployments, highlighting its practicality. Benchmark results against state-of-the-art baselines also confirm the robustness and efficiency of DDBot in such digging tasks.
Abstract (translated)
对颗粒材料进行自动化操作面临着复杂的接触动力学、不可预测的材料特性和复杂系统状态等重大挑战。现有方法通常无法在这些任务中实现高效和准确的操作。为了填补这一研究空白，本文探讨了具有未知物理属性的小规模高精度颗粒材料挖掘任务，并提出了一种新的框架——可微挖土机器人（DDBot），用于操作包括沙子和土壤在内的颗粒材料。具体而言，我们为 DDBot 配备了一个基于物理学的可微模拟器，该模拟器专门针对颗粒材料的操作设计，采用 GPU 加速并行计算与自动微分技术。 通过这一框架，DDBot 能够进行有效的可微系统识别和未知颗粒材料的高精度挖掘技能优化。这得益于一种可微技能到动作映射、面向任务的演示方法、梯度裁剪以及基于线性搜索的梯度下降法。实验结果表明，DDBot 可以在 5 至 20 分钟内高效地识别未知颗粒材料的动力学并优化挖掘技能，在零样本真实世界部署中实现高精度的结果，突显了其实用性。对比最新的基准方法，DDBot 在此类挖掘任务中的鲁棒性和效率也得到了证实。
URL
https://arxiv.org/abs/2510.17335
PDF
-
SAMOSA: Sharpness Aware Minimization for Open Set Active learning
2025-10-19 08:58:15Young In Kim, Andrea Agiollo, Rajiv KhannaAbstract
Modern machine learning solutions require extensive data collection where labeling remains costly. To reduce this burden, open set active learning approaches aim to select informative samples from a large pool of unlabeled data that includes irrelevant or unknown classes. In this context, we propose Sharpness Aware Minimization for Open Set Active Learning (SAMOSA) as an effective querying algorithm. Building on theoretical findings concerning the impact of data typicality on the generalization properties of traditional stochastic gradient descent (SGD) and sharpness-aware minimization (SAM), SAMOSA actively queries samples based on their typicality. SAMOSA effectively identifies atypical samples that belong to regions of the embedding manifold close to the model decision boundaries. Therefore, SAMOSA prioritizes the samples that are (i) highly informative for the targeted classes, and (ii) useful for distinguishing between targeted and unwanted classes. Extensive experiments show that SAMOSA achieves up to 3% accuracy improvement over the state of the art across several datasets, while not introducing computational overhead. The source code of our experiments is available at: this https URL
Abstract (translated)
现代机器学习解决方案需要广泛的数据收集，而数据标注的成本非常高。为了减轻这一负担，开放集主动学习方法旨在从包含不相关或未知类别的大量未标记数据池中选择具有信息性的样本。在此背景下，我们提出了一种名为Sharpness Aware Minimization for Open Set Active Learning (SAMOSA)的有效查询算法。该算法基于理论发现，这些发现在数据典型性对传统随机梯度下降(SGD)和尖锐度感知最小化(SAM)泛化属性的影响方面提供了重要见解。SAMOSA根据样本的典型性主动进行样本选择。 具体来说，SAMOSA能够识别位于嵌入流形中接近模型决策边界的非典型样本。因此，它优先选择对目标类别具有高度信息性的样本，并且这些样本对于区分目标类和未期望类同样有用。广泛的实验表明，在多个数据集上，与现有技术相比，SAMOSA可以实现高达3%的准确率改进，同时不引入计算开销。 我们的实验源代码可在此 URL 获取：[请在这里插入实际链接]
URL
https://arxiv.org/abs/2510.16757
PDF
-
Asymptotically Stable Quaternion-valued Hopfield-structured Neural Network with Periodic Projection-based Supervised Learning Rules
Abstract
Motivated by the geometric advantages of quaternions in representing rotations and postures, we propose a quaternion-valued supervised learning Hopfield-structured neural network (QSHNN) with a fully connected structure inspired by the classic Hopfield neural network (HNN). Starting from a continuous-time dynamical model of HNNs, we extend the formulation to the quaternionic domain and establish the existence and uniqueness of fixed points with asymptotic stability. For the learning rules, we introduce a periodic projection strategy that modifies standard gradient descent by periodically projecting each 4*4 block of the weight matrix onto the closest quaternionic structure in the least-squares sense. This approach preserves both convergence and quaternionic consistency throughout training. Benefiting from this rigorous mathematical foundation, the experimental model implementation achieves high accuracy, fast convergence, and strong reliability across randomly generated target sets. Moreover, the evolution trajectories of the QSHNN exhibit well-bounded curvature, i.e., sufficient smoothness, which is crucial for applications such as control systems or path planning modules in robotic arms, where joint postures are parameterized by quaternion neurons. Beyond these application scenarios, the proposed model offers a practical implementation framework and a general mathematical methodology for designing neural networks under hypercomplex or non-commutative algebraic structures.
Abstract (translated)
受四元数在表示旋转和姿态方面的几何优势启发，我们提出了一种基于经典霍普菲尔德神经网络（HNN）的全连接结构，并结合监督学习方法设计的一种四元数值霍普菲尔德结构神经网络（QSHNN）。从连续时间动力学模型出发，我们将该模型的形式扩展到四元数域，并建立了固定点的存在性、唯一性和渐近稳定性。在学习规则方面，我们引入了一种周期性投影策略，通过周期性地将权重矩阵中的每个4*4块投影到最小二乘意义上的最近的四元数值结构上来修改标准梯度下降法。这种策略在整个训练过程中保持了收敛性和四元数的一致性。 得益于这一严格的数学基础，实验模型实现了高精度、快速收敛和强鲁棒性的目标，在随机生成的目标集合上表现良好。此外，QSHNN的演化轨迹表现出良好的曲率边界，即足够的平滑度，这对于像控制系统或机器人手臂中的路径规划模块这样的应用场景至关重要，其中关节姿态由四元数神经元参数化。 除了这些应用场景之外，所提出的模型还提供了一个实用的实施框架和一种用于在超复数或非交换代数结构下设计神经网络的一般数学方法。
URL
https://arxiv.org/abs/2510.16607
PDF
-
Bridging Accuracy and Interpretability: Deep Learning with XAI for Breast Cancer Detection
2025-10-18 07:47:26Bishal Chhetri, B. V. Rathish KumarAbstract
In this study, we present an interpretable deep learning framework for the early detection of breast cancer using quantitative features extracted from digitized fine needle aspirate (FNA) images of breast masses. Our deep neural network, using ReLU activations, the Adam optimizer, and a binary cross-entropy loss, delivers state-of-the-art classification performance, achieving an accuracy of 0.992, precision of 1.000, recall of 0.977, and an F1 score of 0.988. These results substantially exceed the benchmarks reported in the literature. We evaluated the model under identical protocols against a suite of well-established algorithms (logistic regression, decision trees, random forests, stochastic gradient descent, K-nearest neighbors, and XGBoost) and found the deep model consistently superior on the same metrics. Recognizing that high predictive accuracy alone is insufficient for clinical adoption due to the black-box nature of deep learning models, we incorporated model-agnostic Explainable AI techniques such as SHAP and LIME to produce feature-level attributions and human-readable visualizations. These explanations quantify the contribution of each feature to individual predictions, support error analysis, and increase clinician trust, thus bridging the gap between performance and interpretability for real-world clinical use. The concave points feature of the cell nuclei is found to be the most influential feature positively impacting the classification task. This insight can be very helpful in improving the diagnosis and treatment of breast cancer by highlighting the key characteristics of breast tumor.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21780
PDF
-
Structured Universal Adversarial Attacks on Object Detection for Video Sequences
2025-10-16 09:00:41Sven Jacob, Weijia Shao, Gjergji KasneciAbstract
Video-based object detection plays a vital role in safety-critical applications. While deep learning-based object detectors have achieved impressive performance, they remain vulnerable to adversarial attacks, particularly those involving universal perturbations. In this work, we propose a minimally distorted universal adversarial attack tailored for video object detection, which leverages nuclear norm regularization to promote structured perturbations concentrated in the background. To optimize this formulation efficiently, we employ an adaptive, optimistic exponentiated gradient method that enhances both scalability and convergence. Our results demonstrate that the proposed attack outperforms both low-rank projected gradient descent and Frank-Wolfe based attacks in effectiveness while maintaining high stealthiness. All code and data are publicly available at this https URL.
Abstract (translated)
基于视频的对象检测在安全关键应用中扮演着重要角色。虽然基于深度学习的对象探测器已经取得了令人印象深刻的性能，但它们仍然容易受到对抗性攻击的影响，特别是那些涉及普遍扰动的攻击。在这项工作中，我们提出了一种针对视频对象检测的、最小化结构变形的通用对抗攻击方法，该方法利用核范数正则化来促进集中在背景中的结构化扰动。为了高效地优化这一公式，我们采用了一种自适应的乐观指数梯度法，这种方法增强了可扩展性和收敛性。我们的研究结果表明，在保持高度隐蔽性的前提下，所提出的攻击方法在有效性上超过了低秩投影梯度下降和基于Frank-Wolfe的方法。所有代码和数据都可以通过以下链接公开获取：[此URL]。 请确保替换上述提示中的“this https URL”为实际的链接地址以供访问。
URL
https://arxiv.org/abs/2510.14460
PDF
-
Feature Selection and Regularization in Multi-Class Classification: An Empirical Study of One-vs-Rest Logistic Regression with Gradient Descent Optimization and L1 Sparsity Constraints
2025-10-16 08:47:05Jahidul Arafat, Fariha Tasmin, Md Kaosar Uddin, Sanjaya Poudel, Eftakhar Ahmed ArnobAbstract
Multi-class wine classification presents fundamental trade-offs between model accuracy, feature dimensionality, and interpretability - critical factors for production deployment in analytical chemistry. This paper presents a comprehensive empirical study of One-vs-Rest logistic regression on the UCI Wine dataset (178 samples, 3 cultivars, 13 chemical features), comparing from-scratch gradient descent implementation against scikit-learn's optimized solvers and quantifying L1 regularization effects on feature sparsity. Manual gradient descent achieves 92.59 percent mean test accuracy with smooth convergence, validating theoretical foundations, though scikit-learn provides 24x training speedup and 98.15 percent accuracy. Class-specific analysis reveals distinct chemical signatures with heterogeneous patterns where color intensity varies dramatically (0.31 to 16.50) across cultivars. L1 regularization produces 54-69 percent feature reduction with only 4.63 percent accuracy decrease, demonstrating favorable interpretability-performance trade-offs. We propose an optimal 5-feature subset achieving 62 percent complexity reduction with estimated 92-94 percent accuracy, enabling cost-effective deployment with 80 dollars savings per sample and 56 percent time reduction. Statistical validation confirms robust generalization with sub-2ms prediction latency suitable for real-time quality control. Our findings provide actionable guidelines for practitioners balancing comprehensive chemical analysis against targeted feature measurement in resource-constrained environments.
Abstract (translated)
多类葡萄酒分类在模型准确性、特征维度和可解释性之间存在基本的权衡，这些因素对于分析化学中的生产部署至关重要。本文对UCI葡萄酒数据集（178个样本，3种葡萄品种，13个化学特征）上的一对多逻辑回归进行了一项全面的经验研究，比较了从头开始实现的梯度下降和scikit-learn优化求解器，并量化了L1正则化对特征稀疏性的影响。手动梯度下降实现了平均测试准确率为92.59%，并且具有平滑收敛的特点，验证了理论基础的有效性；然而，scikit-learn提供了比前者快24倍的训练速度和高达98.15%的准确性。针对不同类别的分析揭示了不同的化学特征模式，在这些模式中，颜色强度在品种间差异显著（0.31至16.50）。L1正则化能够减少约54-69%的特征数量，并且仅导致4.63%的准确率下降，这表明了较好的可解释性和性能之间的权衡。我们提出了一种理想的5个特征子集方案，它在保持估计为92-94%准确性的同时实现了复杂度减少62%，从而能够以低成本部署（每样本节省80美元）和时间减少56%的方式进行实施。统计验证确认了其具有鲁棒的泛化能力，并且预测延迟小于2毫秒，这使其适用于实时质量控制。我们的研究为在资源受限环境中平衡全面化学分析与目标特征测量提供了实用指导。
URL
https://arxiv.org/abs/2510.14449
PDF
-
Time-Varying Optimization for Streaming Data Via Temporal Weighting
2025-10-15 00:18:17Muhammad Faraz Ul Abrar, Nicol\`o Michelusi, Erik G. LarssonAbstract
Classical optimization theory deals with fixed, time-invariant objective functions. However, time-varying optimization has emerged as an important subject for decision-making in dynamic environments. In this work, we study the problem of learning from streaming data through a time-varying optimization lens. Unlike prior works that focus on generic formulations, we introduce a structured, \emph{weight-based} formulation that explicitly captures the streaming-data origin of the time-varying objective, where at each time step, an agent aims to minimize a weighted average loss over all the past data samples. We focus on two specific weighting strategies: (1) uniform weights, which treat all samples equally, and (2) discounted weights, which geometrically decay the influence of older data. For both schemes, we derive tight bounds on the ``tracking error'' (TE), defined as the deviation between the model parameter and the time-varying optimum at a given time step, under gradient descent (GD) updates. We show that under uniform weighting, the TE vanishes asymptotically with a $\mathcal{O}(1/t)$ decay rate, whereas discounted weighting incurs a nonzero error floor controlled by the discount factor and the number of gradient updates performed at each time step. Our theoretical findings are validated through numerical simulations.
Abstract (translated)
经典优化理论主要处理固定且不随时间变化的目标函数。然而，在动态环境中，时变优化已经成为决策的重要课题。在本工作中，我们通过时变优化的视角研究从流数据中学习的问题。与之前侧重于通用公式化的工作不同，我们引入了一种结构化的、基于权重（weight-based）的方法，这种方法明确地捕捉到时间变化的目标函数源自流数据的本质，在每一时刻，一个代理试图最小化过去所有数据样本的加权平均损失。 本文重点关注两种特定的加权策略：(1) 均匀权重，即平等对待所有的样本；(2) 折扣权重，它使较旧数据的影响几何级数地衰减。对于这两种方案，在使用梯度下降（GD）更新时，我们推导出了“跟踪误差”（TE）的紧致界。“跟踪误差”定义为某一时刻模型参数与时间变化最优解之间的偏差。 我们的理论研究表明，在均匀加权的情况下，随着迭代次数增加，“跟踪误差”以$\mathcal{O}(1/t)$的速度渐近消失。而在使用折扣权重时，由于折扣因子和每个时间步执行的梯度更新数量的影响，会产生一个非零的误差下限。 这些理论结果通过数值模拟得到了验证。
URL
https://arxiv.org/abs/2510.13052
PDF
