Gradient_Descent
-
Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging
2025-05-21 16:34:37Shenghe Zheng, Hongzhi Wang, Chenyu Huang, Xiaohui Wang, Tao Chen, Jiayuan Fan, Shuyue Hu, Peng YeAbstract
With more open-source models available for diverse tasks, model merging has gained attention by combining models into one, reducing training, storage, and inference costs. Current research mainly focuses on model merging for full fine-tuning, overlooking the popular LoRA. However, our empirical analysis reveals that: a) existing merging methods designed for full fine-tuning perform poorly on LoRA; b) LoRA modules show much larger parameter magnitude variance than full fine-tuned weights; c) greater parameter magnitude variance correlates with worse merging performance. Considering that large magnitude variances cause deviations in the distribution of the merged parameters, resulting in information loss and performance degradation, we propose a Decoupled and Orthogonal merging approach(DO-Merging). By separating parameters into magnitude and direction components and merging them independently, we reduce the impact of magnitude differences on the directional alignment of the merged models, thereby preserving task information. Furthermore, we introduce a data-free, layer-wise gradient descent method with orthogonal constraints to mitigate interference during the merging of direction components. We provide theoretical guarantees for both the decoupling and orthogonal components. And we validate through extensive experiments across vision, language, and multi-modal domains that our proposed DO-Merging can achieve significantly higher performance than existing merging methods at a minimal cost. Notably, each component can be flexibly integrated with existing methods, offering near free-lunch improvements across tasks.
Abstract (translated)
随着开源模型的增多，能够为各种任务服务，模型合并（将多个模型整合成一个）越来越受到关注。通过这种方法可以减少训练、存储和推理的成本。然而，当前的研究主要集中在全面微调（full fine-tuning）上的模型合并上，而忽略了流行的LoRA方法。我们的实证分析表明： a) 为全面微调设计的现有合并方法在处理LoRA时表现不佳； b) LoRA模块显示出比完全微调权重更大的参数幅度变化； c) 更大的参数幅度变化与较差的合并性能相关联。 考虑到大范围的幅度差异会导致合并参数分布中的偏差，从而导致信息丢失和性能下降，我们提出了一种解耦正交合并方法（Decoupled and Orthogonal merging, DO-Merging）。通过将参数分解为幅度和方向分量，并独立地对它们进行合并，我们可以减少幅度差异对合并模型方向一致性的负面影响，从而保留任务相关信息。 此外，为了在方向分量的合并过程中减轻干扰，我们引入了一种无数据依赖、逐层梯度下降的方法，该方法带有正交约束。我们为解耦和正交部分都提供了理论保证。通过广泛的实验验证（涵盖视觉、语言以及跨模态领域），我们的DO-Merging方法可以在最低成本下实现比现有合并方法显著更高的性能。 值得注意的是，每个组件都可以灵活地与现有的方法集成，从而在各种任务中提供近乎无代价的改进效果。
URL
https://arxiv.org/abs/2505.15875
PDF
-
Second-Order Convergence in Private Stochastic Non-Convex Optimization
2025-05-21 15:25:23Youming Tao, Zuyuan Zhang, Dongxiao Yu, Xiuzhen Cheng, Falko Dressler, Di WangAbstract
We investigate the problem of finding second-order stationary points (SOSP) in differentially private (DP) stochastic non-convex optimization. Existing methods suffer from two key limitations: (i) inaccurate convergence error rate due to overlooking gradient variance in the saddle point escape analysis, and (ii) dependence on auxiliary private model selection procedures for identifying DP-SOSP, which can significantly impair utility, particularly in distributed settings. To address these issues, we propose a generic perturbed stochastic gradient descent (PSGD) framework built upon Gaussian noise injection and general gradient oracles. A core innovation of our framework is using model drift distance to determine whether PSGD escapes saddle points, ensuring convergence to approximate local minima without relying on second-order information or additional DP-SOSP identification. By leveraging the adaptive DP-SPIDER estimator as a specific gradient oracle, we develop a new DP algorithm that rectifies the convergence error rates reported in prior work. We further extend this algorithm to distributed learning with arbitrarily heterogeneous data, providing the first formal guarantees for finding DP-SOSP in such settings. Our analysis also highlights the detrimental impacts of private selection procedures in distributed learning under high-dimensional models, underscoring the practical benefits of our design. Numerical experiments on real-world datasets validate the efficacy of our approach.
Abstract (translated)
我们研究了在差异隐私（DP）随机非凸优化中寻找二阶平稳点（SOSP）的问题。现有方法存在两个关键局限性：(i) 由于忽视了鞍点逃脱分析中的梯度方差，导致收敛误差率不准确；(ii) 需要依赖于辅助的私有模型选择程序来识别DP-SOSP，这在分布式环境中可能会显著损害效用。 为解决这些问题，我们提出了一种基于高斯噪声注入和一般梯度预测器构建的泛化扰动随机梯度下降（PSGD）框架。我们框架的一个核心创新是使用模型漂移距离来确定PSGD是否逃脱了鞍点，并确保在不依赖于二阶信息或额外DP-SOSP识别的情况下收敛到近似局部最小值。 通过利用自适应DP-SPIDER估计算法作为特定的梯度预测器，我们开发了一种新的DP算法，该算法纠正了先前工作中报告的收敛误差率。我们将此算法进一步扩展至任意异构数据分布式的环境中，并为在这种设置下寻找DP-SOSP提供了第一个正式保证。 我们的分析还强调，在高维模型下的分布式学习中私有选择程序带来的负面影响，突显了我们设计的实际应用价值。在现实世界的数据集上的数值实验验证了我们方法的有效性。
URL
https://arxiv.org/abs/2505.15647
PDF
-
One-Layer Transformers are Provably Optimal for In-context Reasoning and Distributional Association Learning in Next-Token Prediction Tasks
2025-05-21 01:26:44Quan Nguyen, Thanh Nguyen-TangAbstract
We study the approximation capabilities and on-convergence behaviors of one-layer transformers on the noiseless and noisy in-context reasoning of next-token prediction. Existing theoretical results focus on understanding the in-context reasoning behaviors for either the first gradient step or when the number of samples is infinite. Furthermore, no convergence rates nor generalization abilities were known. Our work addresses these gaps by showing that there exists a class of one-layer transformers that are provably Bayes-optimal with both linear and ReLU attention. When being trained with gradient descent, we show via a finite-sample analysis that the expected loss of these transformers converges at linear rate to the Bayes risk. Moreover, we prove that the trained models generalize to unseen samples as well as exhibit learning behaviors that were empirically observed in previous works. Our theoretical findings are further supported by extensive empirical validations.
Abstract (translated)
我们研究了一层变压器在无噪声和有噪声的上下文推理中的下-token预测时的近似能力和在线收敛行为。现有的理论结果主要集中在理解要么第一次梯度步骤，要么样本数量无限时的上下文推理行为上。此外，没有已知的收敛速率或泛化能力。我们的工作通过展示一类一层变压器，在使用线性注意力和ReLU注意力时都是贝叶斯最优的，填补了这些空白。当用梯度下降法训练时，我们通过对有限样本分析显示，这些变压器的预期损失以线性的速度收敛到贝叶斯风险。此外，我们还证明了经过训练的模型能够泛化至未见过的数据，并且表现出与之前工作中观察到的学习行为一致的行为模式。我们的理论发现进一步通过广泛的实证验证得到了支持。
URL
https://arxiv.org/abs/2505.15009
PDF
-
When majority rules, minority loses: bias amplification of gradient descent
2025-05-19 13:51:49Fran\c{c}ois Bachoc (IMT), J\'er\^ome Bolte (TSE-R), Ryan Boustany (TSE-R), Jean-Michel Loubes (IMT)Abstract
Despite growing empirical evidence of bias amplification in machine learning, its theoretical foundations remain poorly understood. We develop a formal framework for majority-minority learning tasks, showing how standard training can favor majority groups and produce stereotypical predictors that neglect minority-specific features. Assuming population and variance imbalance, our analysis reveals three key findings: (i) the close proximity between ``full-data'' and stereotypical predictors, (ii) the dominance of a region where training the entire model tends to merely learn the majority traits, and (iii) a lower bound on the additional training required. Our results are illustrated through experiments in deep learning for tabular and image classification tasks.
Abstract (translated)
尽管有关机器学习中偏见放大的实证证据越来越多，但其理论基础仍然不甚明了。我们建立了一个用于多数群体和少数群体学习任务的形式框架，展示了标准训练如何倾向于支持多数群体，并产生忽视少数群体特定特征的刻板印象预测模型。假设存在人口不平衡和方差差异的情况下，我们的分析揭示了三个关键发现：(i) “全数据”与刻板印象预测之间的紧密关联；(ii) 训练整个模型往往只是学习多数群体特性的主导区域；以及 (iii) 额外训练所需的最低限度。通过深度学习在表格和图像分类任务中的实验，我们展示了这些结果。
URL
https://arxiv.org/abs/2505.13122
PDF
-
FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
2025-05-19 07:32:56Seanie Lee, Sangwoo Park, Dong Bok Lee, Dominik Wagner, Haebin Seong, Tobias Bocklet, Juho Lee, Sung Ju HwangAbstract
Low-Rank Adaptation (LoRA), which introduces a product of two trainable low-rank matrices into frozen pre-trained weights, is widely used for efficient fine-tuning of language models in federated learning (FL). However, when combined with differentially private stochastic gradient descent (DP-SGD), LoRA faces substantial noise amplification: DP-SGD perturbs per-sample gradients, and the matrix multiplication of the LoRA update ($BA$) intensifies this effect. Freezing one matrix (e.g., $A$) reduces the noise but restricts model expressiveness, often resulting in suboptimal adaptation. To address this, we propose FedSVD, a simple yet effective method that introduces a global reparameterization based on singular value decomposition (SVD). In our approach, each client optimizes only the $B$ matrix and transmits it to the server. The server aggregates the $B$ matrices, computes the product $BA$ using the previous $A$, and refactorizes the result via SVD. This yields a new adaptive $A$ composed of the orthonormal right singular vectors of $BA$, and an updated $B$ containing the remaining SVD components. This reparameterization avoids quadratic noise amplification, while allowing $A$ to better capture the principal directions of the aggregate updates. Moreover, the orthonormal structure of $A$ bounds the gradient norms of $B$ and preserves more signal under DP-SGD, as confirmed by our theoretical analysis. As a result, FedSVD consistently improves stability and performance across a variety of privacy settings and benchmarks, outperforming relevant baselines under both private and non-private regimes.
Abstract (translated)
低秩适应（LoRA）通过将两个可训练的低秩矩阵的乘积引入冻结的预训练权重中，被广泛用于联邦学习（FL）中的语言模型高效微调。然而，在与差异隐私随机梯度下降（DP-SGD）结合时，LoRA面临着显著的噪声放大问题：DP-SGD会扰动每个样本的梯度，而LoRA更新的矩阵乘法($BA$)则增强了这一影响。冻结一个矩阵（例如 $A$）虽然可以减少噪音但限制了模型的表现力，往往导致次优适应。 为解决上述问题，我们提出了一种简单且有效的方法FedSVD，该方法基于奇异值分解（SVD）引入了一个全局重参数化。在我们的方法中，每个客户端仅优化矩阵 $B$ 并将其传输到服务器。服务器聚合了所有的 $B$ 矩阵，并使用之前的 $A$ 计算乘积 $BA$ ，然后通过 SVD 重新因子化该结果。这生成了一个新的适应性 $A$，由 $BA$ 的正交右奇异向量组成，并且更新后的 $B$ 包含剩余的 SVD 组件。这种重参数化避免了二次噪声放大，同时允许 $A$ 更好地捕捉聚合更新的主要方向。此外，由于 $A$ 的正交结构，它限制了 $B$ 的梯度范数并在 DP-SGD 下保留更多信号，这一点已被我们的理论分析所证实。 因此，在各种隐私设置和基准测试中，FedSVD 一致提高了稳定性和性能，并且在私人和非私人模式下均超越了相关的基线方法。
URL
https://arxiv.org/abs/2505.12805
PDF
-
EnvInjection: Environmental Prompt Injection Attack to Multi-modal Web Agents
2025-05-16 22:00:26Xilong Wang, John Bloch, Zedian Shao, Yuepeng Hu, Shuyan Zhou, Neil Zhenqiang GongAbstract
Multi-modal large language model (MLLM)-based web agents interact with webpage environments by generating actions based on screenshots of the webpages. Environmental prompt injection attacks manipulate the environment to induce the web agent to perform a specific, attacker-chosen action--referred to as the target action. However, existing attacks suffer from limited effectiveness or stealthiness, or are impractical in real-world settings. In this work, we propose EnvInjection, a new attack that addresses these limitations. Our attack adds a perturbation to the raw pixel values of the rendered webpage, which can be implemented by modifying the webpage's source code. After these perturbed pixels are mapped into a screenshot, the perturbation induces the web agent to perform the target action. We formulate the task of finding the perturbation as an optimization problem. A key challenge in solving this problem is that the mapping between raw pixel values and screenshot is non-differentiable, making it difficult to backpropagate gradients to the perturbation. To overcome this, we train a neural network to approximate the mapping and apply projected gradient descent to solve the reformulated optimization problem. Extensive evaluation on multiple webpage datasets shows that EnvInjection is highly effective and significantly outperforms existing baselines.
Abstract (translated)
基于多模态大型语言模型（MLLM）的网页代理通过根据网页截图生成动作来与网页环境互动。环境提示注入攻击通过操控环境，诱使网页代理执行特定的、由攻击者选定的动作——称为目标动作。然而，现有的攻击方法在有效性或隐蔽性方面存在局限，或者在现实世界场景中难以实施。在这项工作中，我们提出了一种新的攻击方式EnvInjection，旨在克服这些限制。 我们的攻击通过向渲染后的网页的原始像素值添加干扰来实现，这可以通过修改网页源代码完成。当这些被干扰的像素映射到截图后，干扰会诱使网页代理执行目标动作。我们将寻找这一干扰的任务形式化为一个优化问题。解决这个问题的关键挑战在于，原始像素值与截图之间的映射是非可微分的（non-differentiable），这使得难以将梯度反向传播到干扰上。为了克服这一点，我们训练了一个神经网络来近似该映射，并应用投影梯度下降法来求解重新表述后的优化问题。 在多个网页数据集上的广泛评估表明，EnvInjection具有高度有效性，并且显著优于现有的基准方法。
URL
https://arxiv.org/abs/2505.11717
PDF
-
Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis
2025-05-16 16:28:34Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. StanleyAbstract
Much of the excitement in modern AI is driven by the observation that scaling up existing systems leads to better performance. But does better performance necessarily imply better internal representations? While the representational optimist assumes it must, this position paper challenges that view. We compare neural networks evolved through an open-ended search process to networks trained via conventional stochastic gradient descent (SGD) on the simple task of generating a single image. This minimal setup offers a unique advantage: each hidden neuron's full functional behavior can be easily visualized as an image, thus revealing how the network's output behavior is internally constructed neuron by neuron. The result is striking: while both networks produce the same output behavior, their internal representations differ dramatically. The SGD-trained networks exhibit a form of disorganization that we term fractured entangled representation (FER). Interestingly, the evolved networks largely lack FER, even approaching a unified factored representation (UFR). In large models, FER may be degrading core model capacities like generalization, creativity, and (continual) learning. Therefore, understanding and mitigating FER could be critical to the future of representation learning.
Abstract (translated)
现代人工智能领域的许多兴奋之处在于，扩大现有系统的规模会导致性能提升。然而，更好的性能是否必然意味着内部表示也更好呢？虽然代表乐观主义者认为答案是肯定的，但本文对此观点提出了挑战。我们比较了通过开放式演化过程进化的神经网络与使用传统随机梯度下降（SGD）训练的神经网络在生成单个图像这一简单任务上的表现。这种最小设置提供了一个独特的优势：可以轻松地将每个隐藏神经元的完整功能行为可视化为一个图像，从而揭示出网络输出行为是如何通过一个个神经元构建起来的。 结果令人震惊：尽管两种类型的网络都产生了相同的行为输出，但它们的内部表示却大相径庭。SGD训练的网络表现出一种我们称之为“碎片化纠缠表示”（FER）的形式。有趣的是，演化出来的网络在很大程度上缺乏这种“碎片化纠缠表示”，甚至接近于统一因子表示（UFR）。在大规模模型中，“碎片化纠缠表示”可能会削弱核心功能能力，如泛化、创造力和持续学习等。 因此，理解和缓解“碎片化纠缠表示”的问题对于未来代表学习的发展可能至关重要。
URL
https://arxiv.org/abs/2505.11581
PDF
-
Adaptive Diffusion Policy Optimization for Robotic Manipulation
2025-05-13 09:21:45Huiyun Jiang, Zhuang YangAbstract
Recent studies have shown the great potential of diffusion models in improving reinforcement learning (RL) by modeling complex policies, expressing a high degree of multi-modality, and efficiently handling high-dimensional continuous control tasks. However, there is currently limited research on how to optimize diffusion-based polices (e.g., Diffusion Policy) fast and stably. In this paper, we propose an Adam-based Diffusion Policy Optimization (ADPO), a fast algorithmic framework containing best practices for fine-tuning diffusion-based polices in robotic control tasks using the adaptive gradient descent method in RL. Adaptive gradient method is less studied in training RL, let alone diffusion-based policies. We confirm that ADPO outperforms other diffusion-based RL methods in terms of overall effectiveness for fine-tuning on standard robotic tasks. Concretely, we conduct extensive experiments on standard robotic control tasks to test ADPO, where, particularly, six popular diffusion-based RL methods are provided as benchmark methods. Experimental results show that ADPO acquires better or comparable performance than the baseline methods. Finally, we systematically analyze the sensitivity of multiple hyperparameters in standard robotics tasks, providing guidance for subsequent practical applications. Our video demonstrations are released in this https URL.
Abstract (translated)
最近的研究表明，扩散模型在通过建模复杂的策略、表达高多模态性以及高效处理高维连续控制任务方面，具有显著提升强化学习（RL）的潜力。然而，目前关于如何快速且稳定地优化基于扩散的策略（如Diffusion Policy）的研究还很有限。在这篇论文中，我们提出了一种基于Adam的方法——扩散策略优化(ADPO)，这是一个快速的算法框架，使用自适应梯度下降方法在强化学习中对机器人控制任务中的基于扩散的策略进行微调。自适应梯度法在训练RL中研究较少，更不用说针对基于扩散的政策了。我们确认，在标准机器人任务上的微调总体效果上，ADPO优于其他基于扩散的方法。 具体来说，我们在标准的机器人控制任务上进行了广泛的实验以测试ADPO的有效性，并提供了六种流行的基于扩散的强化学习方法作为基准方法进行对比。实验证明，ADPO在性能上至少达到了与这些基准方法相同的水平，或者更好。最后，我们系统地分析了标准机器人任务中多个超参数的敏感度，为后续的实际应用提供了指导。我们的视频演示可在[此处](https://this.https.URL)查看。
URL
https://arxiv.org/abs/2505.08376
PDF
-
IRNN: Innovation-driven Recurrent Neural Network for Time-Series Data Modeling and Prediction
2025-05-09 09:43:40Yifan Zhou, Yibo Wang, Chao ShangAbstract
Many real-world datasets are time series that are sequentially collected and contain rich temporal information. Thus, a common interest in practice is to capture dynamics of time series and predict their future evolutions. To this end, the recurrent neural network (RNN) has been a prevalent and effective machine learning option, which admits a nonlinear state-space model representation. Motivated by the resemblance between RNN and Kalman filter (KF) for linear state-space models, we propose in this paper Innovation-driven RNN (IRNN), a novel RNN architecture tailored to time-series data modeling and prediction tasks. By adapting the concept of "innovation" from KF to RNN, past prediction errors are adopted as additional input signals to update hidden states of RNN and boost prediction performance. Since innovation data depend on network parameters, existing training algorithms for RNN do not apply to IRNN straightforwardly. Thus, a tailored training algorithm dubbed input updating-based back-propagation through time (IU-BPTT) is further proposed, which alternates between updating innovations and optimizing network parameters via gradient descent. Experiments on real-world benchmark datasets show that the integration of innovations into various forms of RNN leads to remarkably improved prediction accuracy of IRNN without increasing the training cost substantially.
Abstract (translated)
许多现实世界的数据集都是按时间顺序收集的时间序列数据，包含丰富的时序信息。因此，在实践中一个共同的兴趣是捕捉时间序列的动力学特性并预测它们的未来演变趋势。为此，递归神经网络（RNN）已成为一种普遍且有效的机器学习选项，它可以表示非线性状态空间模型。受RNN与线性状态空间模型中的卡尔曼滤波器（KF）之间的相似性的启发，本文提出了创新驱动式RNN（IRNN），这是一种新型的针对时间序列数据建模和预测任务定制化的RNN架构。通过将“创新”这一概念从KF引入到RNN中，我们采用了过去预测误差作为额外输入信号来更新RNN的状态，并提高了预测性能。由于创新数据依赖于网络参数，现有的适用于RNN的训练算法不能直接应用于IRNN。因此，进一步提出了一种定制化的训练算法——基于输入更新的反向传播通过时间（IU-BPTT），该算法交替进行创新更新和利用梯度下降优化网络参数。在现实世界的基准数据集上的实验表明，在不显著增加训练成本的情况下，将创新整合到各种形式的RNN中可以大大提高IRNN的预测准确性。
URL
https://arxiv.org/abs/2505.05916
PDF
-
ADMM-Based Training for Spiking Neural Networks
2025-05-08 10:20:33Giovanni Perin, Cesare Bidini, Riccardo Mazzieri, Michele RossiAbstract
In recent years, spiking neural networks (SNNs) have gained momentum due to their high potential in time-series processing combined with minimal energy consumption. However, they still lack a dedicated and efficient training algorithm. The popular backpropagation with surrogate gradients, adapted from stochastic gradient descent (SGD)-derived algorithms, has several drawbacks when used as an optimizer for SNNs. Specifically, it suffers from low scalability and numerical imprecision. In this paper, we propose a novel SNN training method based on the alternating direction method of multipliers (ADMM). Our ADMM-based training aims to solve the problem of the SNN step function's non-differentiability. We formulate the problem, derive closed-form updates, and empirically show the optimizer's convergence properties, great potential, and possible new research directions to improve the method in a simulated proof-of-concept.
Abstract (translated)
近年来，由于脉冲神经网络（SNN）在时间序列处理方面具有巨大潜力，并且能耗极低，因此它们得到了迅速的发展。然而，目前这些网络仍缺乏一个专门和高效的训练算法。从随机梯度下降（SGD）衍生出来的流行反向传播方法与替代梯度，在作为SNN优化器使用时存在一些缺点，特别是可扩展性和数值精度方面的问题。在本文中，我们提出了一种基于交替方向乘子法（ADMM）的新颖的SNN训练方法。我们的基于ADMM的训练旨在解决SNN步函数不可微分的问题。我们对问题进行了建模，推导了封闭形式的更新公式，并通过模拟的概念验证实验展示了优化器的收敛特性、巨大的潜力以及可能的研究新方向来改进这种方法。
URL
https://arxiv.org/abs/2505.05527
PDF
-
Precise gradient descent training dynamics for finite-width multi-layer neural networks
Abstract
In this paper, we provide the first precise distributional characterization of gradient descent iterates for general multi-layer neural networks under the canonical single-index regression model, in the `finite-width proportional regime' where the sample size and feature dimension grow proportionally while the network width and depth remain bounded. Our non-asymptotic state evolution theory captures Gaussian fluctuations in first-layer weights and concentration in deeper-layer weights, and remains valid for non-Gaussian features. Our theory differs from existing neural tangent kernel (NTK), mean-field (MF) theories and tensor program (TP) in several key aspects. First, our theory operates in the finite-width regime whereas these existing theories are fundamentally infinite-width. Second, our theory allows weights to evolve from individual initializations beyond the lazy training regime, whereas NTK and MF are either frozen at or only weakly sensitive to initialization, and TP relies on special initialization schemes. Third, our theory characterizes both training and generalization errors for general multi-layer neural networks beyond the uniform convergence regime, whereas existing theories study generalization almost exclusively in two-layer settings. As a statistical application, we show that vanilla gradient descent can be augmented to yield consistent estimates of the generalization error at each iteration, which can be used to guide early stopping and hyperparameter tuning. As a further theoretical implication, we show that despite model misspecification, the model learned by gradient descent retains the structure of a single-index function with an effective signal determined by a linear combination of the true signal and the initialization.
Abstract (translated)
在这篇论文中，我们首次提供了多层神经网络在标准单索引回归模型下的梯度下降迭代的精确分布特征，在“有限宽度比例制度”下进行研究，即样本数量和特征维度按比例增长的同时，网络的宽度和深度保持不变。我们的非渐近状态演化理论捕捉了第一层权重中的高斯波动和更深层级权重中的集中特性，并且对非高斯特征依然有效。 该理论在几个关键方面与现有的神经切片核（NTK）、均值场（MF）理论以及张量程序（TP）有所不同。首先，我们的理论适用于有限宽度的情形，而这些现有理论本质上是无限宽度的。其次，我们的理论允许权重从个体初始化演变超出懒惰训练阶段，而NTK和MF要么在初始化时冻结不变，要么对初始化仅有微弱敏感性，并且TP依赖于特殊的初始化方案。第三，我们的理论描述了广泛多层神经网络的训练误差和泛化误差，超出了均匀收敛范围，而现有的理论几乎完全专注于两层设置下的泛化研究。 作为统计学的应用实例，我们展示了普通的梯度下降可以被增强以产生一致性的估计值，这些估计值在每次迭代时反映了泛化误差，并可用于指导提前停止和超参数调整。此外，从理论上讲，即使存在模型规格不当的情况，由梯度下降学习到的模型仍保留了单索引函数的形式，其有效信号由真实信号与初始化的线性组合决定。
URL
https://arxiv.org/abs/2505.04898
PDF
-
Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation
2025-05-05 18:54:41Anjila Budathoki, Manish DhakalAbstract
Adversarial attacks have been fairly explored for computer vision and vision-language models. However, the avenue of adversarial attack for the vision language segmentation models (VLSMs) is still under-explored, especially for medical image analysis. Thus, we have investigated the robustness of VLSMs against adversarial attacks for 2D medical images with different modalities with radiology, photography, and endoscopy. The main idea of this project was to assess the robustness of the fine-tuned VLSMs specially in the medical domain setting to address the high risk scenario. First, we have fine-tuned pre-trained VLSMs for medical image segmentation with adapters. Then, we have employed adversarial attacks -- projected gradient descent (PGD) and fast gradient sign method (FGSM) -- on that fine-tuned model to determine its robustness against adversaries. We have reported models' performance decline to analyze the adversaries' impact. The results exhibit significant drops in the DSC and IoU scores after the introduction of these adversaries. Furthermore, we also explored universal perturbation but were not able to find for the medical images. \footnote{this https URL}
Abstract (translated)
对抗攻击在计算机视觉和视觉-语言模型领域已得到了相当程度的研究。然而，针对视觉语言分割模型（VLSMs）的对抗攻击研究仍然较少，特别是在医学图像分析方面。因此，我们调查了不同模态2D医学图像上经过微调的VLSMs面对对抗攻击时的鲁棒性，尤其是在放射学、摄影和内窥镜检查领域。该项目的主要目的是评估在特定医疗场景下针对高风险情况下的微调VLSM的稳健性。 首先，我们在医学图像分割任务中使用适配器对预训练的VLSMs进行了微调。然后，我们对该模型应用了对抗攻击——包括投影梯度下降（PGD）和快速梯度符号方法（FGSM），以评估其在面对对手时的鲁棒性。我们报告了模型性能的下降情况来分析这些对抗样本的影响。实验结果显示，在引入这些对抗样本后，Dice相似系数（DSC）和交并比（IoU）得分有显著下降。 此外，我们也探索了一种通用扰动方法但未能在医学图像上找到有效应用。\footnote{此链接指向的是原始研究或相关资源的URL，但由于没有提供具体的URL地址，这里仅以说明性文本形式表达。} 这段翻译详细描述了项目的目标、方法和结果，并指出对抗攻击对VLSMs在医疗图像分析中的影响显著，但通用扰动法的应用尚未成功实现。
URL
https://arxiv.org/abs/2505.02971
PDF
-
Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oracles
Abstract
This study explores the performance of a random Gaussian smoothing zeroth-order (ZO) scheme for minimising quasar-convex (QC) and strongly quasar-convex (SQC) functions in both unconstrained and constrained settings. For the unconstrained problem, we establish the ZO algorithm's convergence to a global minimum along with its complexity when applied to both QC and SQC functions. For the constrained problem, we introduce the new notion of proximal-quasar-convexity and prove analogous results to the unconstrained case. Specifically, we show the complexity bounds and the convergence of the algorithm to a neighbourhood of a global minimum whose size can be controlled under a variance reduction scheme. Theoretical findings are illustrated through investigating the performance of the algorithm applied to a range of problems in machine learning and optimisation. Specifically, we observe scenarios where the ZO method outperforms gradient descent. We provide a possible explanation for this phenomenon.
Abstract (translated)
这项研究探讨了随机高斯平滑零阶(ZO)方案在无约束和有约束条件下最小化类夸克凸(QC)和强类夸克凸(SQC)函数时的性能。对于无约束问题，我们建立了ZO算法应用于QC和SQC函数时收敛于全局最优解及其复杂度的理论基础。对于有约束问题，我们引入了新的近似-类夸克凸性概念，并证明了与无约束情况类似的结果。具体来说，我们展示了算法在方差减少方案下的复杂度界线以及该算法收敛到一个可以控制大小的全局最优点邻域。 通过研究ZO方法应用于机器学习和优化问题中的性能表现，本文理论发现得到了验证。特别是，在一些场景中，ZO方法的表现优于梯度下降法。对此现象，我们提供了一种可能的解释。
URL
https://arxiv.org/abs/2505.02281
PDF
-
Focus on the Likely: Test-time Instance-based Uncertainty Removal
Abstract
We propose two novel test-time fine-tuning methods to improve uncertain model predictions. Our methods require no auxiliary data and use the given test instance only. Instead of performing a greedy selection of the most likely class to make a prediction, we introduce an additional focus on the likely classes step during inference. By applying a single-step gradient descent, we refine predictions when an initial forward pass indicates high uncertainty. This aligns predictions more closely with the ideal of assigning zero probability to less plausible outcomes. Our theoretical discussion provides a deeper understanding highlighting the impact on shared and non-shared features among (focus) classes. The experimental evaluation highlights accuracy gains on samples exhibiting high decision uncertainty for a diverse set of models from both the text and image domain using the same hyperparameters.
Abstract (translated)
我们提出了两种新颖的测试时间微调方法，旨在改进模型在不确定情况下的预测。这些方法不需要额外的数据支持，并且仅使用给定的测试实例。不同于执行贪婪选择最可能类别的策略以进行预测，我们在推理阶段引入了一个额外关注可能类别步骤的方法。通过应用单步梯度下降法，在初始前向传递表明高不确定性时，我们可以优化预测结果。这种方法使预测更加接近理想状态：即对不太可能的结果赋予零概率。 理论讨论深入探讨了共享和非共享特征在各类别中的影响，并提供了更深层次的理解。实验评估显示，使用相同的超参数设置，对于来自文本和图像领域多种模型中表现出高度决策不确定性的样本，我们的方法能够显著提高准确性。
URL
https://arxiv.org/abs/2505.03819
PDF
-
Towards the Resistance of Neural Network Watermarking to Fine-tuning
2025-05-02 05:11:17Ling Tang, Yuefeng Chen, Hui Xue, Quanshi ZhangAbstract
This paper proves a new watermarking method to embed the ownership information into a deep neural network (DNN), which is robust to fine-tuning. Specifically, we prove that when the input feature of a convolutional layer only contains low-frequency components, specific frequency components of the convolutional filter will not be changed by gradient descent during the fine-tuning process, where we propose a revised Fourier transform to extract frequency components from the convolutional filter. Additionally, we also prove that these frequency components are equivariant to weight scaling and weight permutations. In this way, we design a watermark module to encode the watermark information to specific frequency components in a convolutional filter. Preliminary experiments demonstrate the effectiveness of our method.
Abstract (translated)
本文提出了一种新的水印技术，可以将所有权信息嵌入到深度神经网络（DNN）中，并且这种水印在微调过程中是鲁棒的。具体而言，我们证明了当卷积层的输入特征仅包含低频分量时，在梯度下降过程中的微调阶段，该层的卷积滤波器的特定频率成分不会发生变化。为此，我们提出了一种修订版的傅里叶变换方法来从卷积滤波器中提取这些频率成分。此外，我们也证明了这些频率成分对于权重缩放和权重置换是等变（equivariant）的。通过这种方式，我们设计了一个水印模块，可以将水印信息编码到卷积滤波器的特定频率分量中。初步实验表明我们的方法的有效性。
URL
https://arxiv.org/abs/2505.01007
PDF
-
How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias
2025-05-02 00:07:35Ruiquan Huang, Yingbin Liang, Jing YangAbstract
Language recognition tasks are fundamental in natural language processing (NLP) and have been widely used to benchmark the performance of large language models (LLMs). These tasks also play a crucial role in explaining the working mechanisms of transformers. In this work, we focus on two representative tasks in the category of regular language recognition, known as `even pairs' and `parity check', the aim of which is to determine whether the occurrences of certain subsequences in a given sequence are even. Our goal is to explore how a one-layer transformer, consisting of an attention layer followed by a linear layer, learns to solve these tasks by theoretically analyzing its training dynamics under gradient descent. While even pairs can be solved directly by a one-layer transformer, parity check need to be solved by integrating Chain-of-Thought (CoT), either into the inference stage of a transformer well-trained for the even pairs task, or into the training of a one-layer transformer. For both problems, our analysis shows that the joint training of attention and linear layers exhibits two distinct phases. In the first phase, the attention layer grows rapidly, mapping data sequences into separable vectors. In the second phase, the attention layer becomes stable, while the linear layer grows logarithmically and approaches in direction to a max-margin hyperplane that correctly separates the attention layer outputs into positive and negative samples, and the loss decreases at a rate of $O(1/t)$. Our experiments validate those theoretical results.
Abstract (translated)
语言识别任务是自然语言处理（NLP）中的基本任务，广泛用于评估大型语言模型（LLMs）的性能。这些任务在解释变压器的工作机制方面也扮演着重要角色。在这项工作中，我们专注于两类常规语言识别任务的代表性任务：即“偶数对”和“奇偶校验”，其目的是确定给定序列中某些子序列出现次数是否为偶数。我们的目标是通过理论分析单层变压器（由注意力层和线性层组成）在梯度下降训练过程中的学习动态，来探索它如何解决这些任务。“偶数对”可以直接使用单层变压器解决，“奇偶校验”则需要结合“链式思维”(Chain-of-Thought, CoT) 方法来解决。对于已经经过“偶数对”任务良好训练的变压器推理阶段中集成CoT，或是在单层变压器的训练过程中直接整合。 针对这两个问题，我们的分析表明注意力和线性层的联合训练展示了两个不同的阶段。在第一阶段，注意力层迅速增长，并将数据序列映射为可分向量。而在第二阶段，注意力层变得稳定，而线性层则以对数方式增长，在方向上逐渐接近于能够正确分离注意力层输出（将样本分为正负两类）的最大间隔超平面，并且损失函数的减少速度是$O(1/t)$。 实验结果验证了这些理论成果。
URL
https://arxiv.org/abs/2505.00926
PDF
-
The Dark Side of Digital Twins: Adversarial Attacks on AI-Driven Water Forecasting
2025-04-28 22:34:11Mohammadhossein Homaei, Victor Gonzalez Morales, Oscar Mogollon-Gutierrez, Andres CaroAbstract
Digital twins (DTs) are improving water distribution systems by using real-time data, analytics, and prediction models to optimize operations. This paper presents a DT platform designed for a Spanish water supply network, utilizing Long Short-Term Memory (LSTM) networks to predict water consumption. However, machine learning models are vulnerable to adversarial attacks, such as the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD). These attacks manipulate critical model parameters, injecting subtle distortions that degrade forecasting accuracy. To further exploit these vulnerabilities, we introduce a Learning Automata (LA) and Random LA-based approach that dynamically adjusts perturbations, making adversarial attacks more difficult to detect. Experimental results show that this approach significantly impacts prediction reliability, causing the Mean Absolute Percentage Error (MAPE) to rise from 26% to over 35%. Moreover, adaptive attack strategies amplify this effect, highlighting cybersecurity risks in AI-driven DTs. These findings emphasize the urgent need for robust defenses, including adversarial training, anomaly detection, and secure data pipelines.
Abstract (translated)
数字孪生（DT）通过使用实时数据、分析和预测模型来优化运营，正在改善水分配系统。本文介绍了一个专为西班牙供水网络设计的数字孪生平台，并利用长短期记忆（LSTM）网络来预测用水量。然而，机器学习模型容易受到诸如快速梯度符号法（FGSM）和投影梯度下降（PGD）等对抗性攻击的影响。这些攻击操纵关键的模型参数，在不影响数据外观的情况下注入细微扰动以降低预测准确性。为了进一步利用这些漏洞，我们引入了一种基于学习自动机（LA）和随机学习自动机的方法，该方法能够动态调整扰动，使对抗性攻击更难被检测到。实验结果表明，这种方法显著影响了预测的可靠性，导致平均绝对百分比误差（MAPE）从26%上升至超过35%。此外，适应性的攻击策略放大了这种效果，突显了AI驱动的数字孪生中的网络安全风险。这些发现强调了建立强大的防御措施的迫切需要，包括对抗性训练、异常检测和安全的数据管道。
URL
https://arxiv.org/abs/2504.20295
PDF
-
FedAvgen: Metadata for Model Aggregation In Communication Systems
2025-04-28 13:11:32Anthony Kiggundu, Dennis Krummacker, Hans D. SchottenAbstract
To improve business efficiency and minimize costs, Artificial Intelligence (AI) practitioners have adopted a shift from formulating models from scratch towards sharing pretrained models. The pretrained models are then aggregated into a global model with higher generalization capabilities, which is afterwards distributed to the client devices. This approach is known as federated learning and inherently utilizes different techniques to select the candidate client models averaged to obtain the global model. This approach, in the case of communication systems, faces challenges arising from the existential diversity in device profiles. The multiplicity in profiles motivates our conceptual assessment of a metaheuristic algorithm (FedAvgen), which relates each pretrained model with its weight space as metadata, to a phenotype and genotype, respectively. This parent-child genetic evolution characterizes the global averaging step in federated learning. We then compare the results of our approach to two widely adopted baseline federated learning algorithms like Federated Averaging (FedAvg) and Federated Stochastic Gradient Descent (FedSGD).
Abstract (translated)
为了提高业务效率并减少成本，人工智能（AI）从业者已从零开始构建模型转向共享预训练模型。这些预训练模型会被聚合为一个具有更强泛化能力的全局模型，然后分发到客户端设备上。这种方法被称为联邦学习，并且内在地利用不同的技术来选择用于获得全局模型的候选客户端模型的平均值。在通信系统中，该方法面临着由于设备配置文件存在多样性而产生的挑战。这种配置文件的多样化促使我们对一种元启发式算法（FedAvgen）进行概念评估，该算法将每个预训练模型与其权重空间分别视为元数据中的表型和基因型。这种父代-子代遗传进化特征化了联邦学习中的全局平均步骤。然后，我们将我们的方法的结果与两个广泛采用的基线联邦学习算法，如联邦平均（FedAvg）和联邦随机梯度下降（FedSGD），进行比较。
URL
https://arxiv.org/abs/2505.05486
PDF
-
Smart Placement, Faster Robots -- A Comparison of Algorithms for Robot Base-Pose Optimization
2025-04-28 08:36:17Matthias Mayer, Matthias AlthoffAbstract
Robotic automation is a key technology that increases the efficiency and flexibility of manufacturing processes. However, one of the challenges in deploying robots in novel environments is finding the optimal base pose for the robot, which affects its reachability and deployment cost. Yet, the existing research for automatically optimizing the base pose of robots has not been compared. We address this problem by optimizing the base pose of industrial robots with Bayesian optimization, exhaustive search, genetic algorithms, and stochastic gradient descent and find that all algorithms can reduce the cycle time for various evaluated tasks in synthetic and real-world environments. Stochastic gradient descent shows superior performance with regard to success rate solving over 90% of our real-world tasks, while genetic algorithms show the lowest final costs. All benchmarks and implemented methods are available as baselines against which novel approaches can be compared.
Abstract (translated)
机器人自动化是提高制造过程效率和灵活性的关键技术。然而，在新的环境中部署机器人时，一个挑战在于找到机器人的最优基座姿态（base pose），这会影响其可达性和部署成本。不过，现有的关于自动优化机器人基座姿态的研究尚未进行过比较研究。我们通过使用贝叶斯优化、穷举搜索、遗传算法和随机梯度下降来优化工业机器人的基座姿态，并发现所有这些算法都能在合成环境和真实世界环境中减少各种任务的循环时间。随机梯度下降法表现出色，成功解决了超过90%的真实世界任务；而遗传算法则显示出最低的最终成本。所有的基准测试和实现方法都可以作为新方法进行比较的基础。
URL
https://arxiv.org/abs/2504.19577
PDF
-
Sharp higher order convergence rates for the Adam optimizer
2025-04-28 02:17:50Steffen Dereich, Arnulf Jentzen, Adrian RiekertAbstract
Gradient descent based optimization methods are the methods of choice to train deep neural networks in machine learning. Beyond the standard gradient descent method, also suitable modified variants of standard gradient descent involving acceleration techniques such as the momentum method and/or adaptivity techniques such as the RMSprop method are frequently considered optimization methods. These days the most popular of such sophisticated optimization schemes is presumably the Adam optimizer that has been proposed in 2014 by Kingma and Ba. A highly relevant topic of research is to investigate the speed of convergence of such optimization methods. In particular, in 1964 Polyak showed that the standard gradient descent method converges in a neighborhood of a strict local minimizer with rate (x - 1)(x + 1)^{-1} while momentum achieves the (optimal) strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} where x \in (1,\infty) is the condition number (the ratio of the largest and the smallest eigenvalue) of the Hessian of the objective function at the local minimizer. It is the key contribution of this work to reveal that Adam also converges with the strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} while RMSprop only converges with the convergence rate (x - 1)(x + 1)^{-1}.
Abstract (translated)
基于梯度下降的优化方法是训练深度神经网络在机器学习中的首选方法。除了标准的梯度下降方法外，还包括适合加速技术（如动量法）和适应性技术（如RMSprop方法）的标准梯度下降的各种变体，这些技术也常被考虑作为优化方法。近年来，最流行的此类复杂优化方案可能是Kingma和Ba在2014年提出的Adam优化器。研究这样一个优化方法的收敛速度是一个非常重要的课题。特别地，在1964年Polyak证明了标准梯度下降方法在一个严格局部极小值附近的收敛率为(x - 1)(x + 1)^{-1}，而动量法则实现了最优且更快速的收敛率(\sqrt{x} - 1)(\sqrt{x} + 1)^{-1}，其中x \in (1,\infty)是目标函数在局部极小值点处Hessian矩阵的最大特征值与最小特征值比（即条件数）。这项工作的关键贡献在于揭示了Adam优化器也以严格更快的收敛率(\sqrt{x} - 1)(\sqrt{x} + 1)^{-1}进行收敛，而RMSprop仅以(x - 1)(x + 1)^{-1}的收敛速率进行收敛。
URL
https://arxiv.org/abs/2504.19426
PDF
