Gradient_Descent
-
A Proposal for Networks Capable of Continual Learning
Abstract
We analyze the ability of computational units to retain past responses after parameter updates, a key property for system-wide continual learning. Neural networks trained with gradient descent lack this capability, prompting us to propose Modelleyen, an alternative approach with inherent response preservation. We demonstrate through experiments on modeling the dynamics of a simple environment and on MNIST that, despite increased computational complexity and some representational limitations at its current stage, Modelleyen achieves continual learning without relying on sample replay or predefined task boundaries.
Abstract (translated)
我们分析了计算单元在参数更新后保留过去响应的能力，这是系统级持续学习的关键属性。通过梯度下降训练的神经网络缺乏这种能力，因此我们提出了Modelleyen这一替代方法，该方法具备固有的响应保持功能。通过实验模拟简单环境的动力学以及在MNIST数据集上的实验表明，尽管计算复杂性增加且目前阶段存在一些表示限制，但Modelleyen能够在不依赖样本重播或预定义任务边界的情况下实现持续学习。
URL
https://arxiv.org/abs/2503.22068
PDF
-
Quality-focused Active Adversarial Policy for Safe Grasping in Human-Robot Interaction
2025-03-25 07:09:31Chenghao Li, Razvan Beuran, Nak Young ChongAbstract
Vision-guided robot grasping methods based on Deep Neural Networks (DNNs) have achieved remarkable success in handling unknown objects, attributable to their powerful generalizability. However, these methods with this generalizability tend to recognize the human hand and its adjacent objects as graspable targets, compromising safety during Human-Robot Interaction (HRI). In this work, we propose the Quality-focused Active Adversarial Policy (QFAAP) to solve this problem. Specifically, the first part is the Adversarial Quality Patch (AQP), wherein we design the adversarial quality patch loss and leverage the grasp dataset to optimize a patch with high quality scores. Next, we construct the Projected Quality Gradient Descent (PQGD) and integrate it with the AQP, which contains only the hand region within each real-time frame, endowing the AQP with fast adaptability to the human hand shape. Through AQP and PQGD, the hand can be actively adversarial with the surrounding objects, lowering their quality scores. Therefore, further setting the quality score of the hand to zero will reduce the grasping priority of both the hand and its adjacent objects, enabling the robot to grasp other objects away from the hand without emergency stops. We conduct extensive experiments on the benchmark datasets and a cobot, showing the effectiveness of QFAAP. Our code and demo videos are available here: this https URL.
Abstract (translated)
基于深度神经网络（DNN）的视觉引导机器人抓取方法在处理未知物体时取得了显著的成功，这主要归功于其强大的泛化能力。然而，具有这种泛化的策略往往将人的手及其周围的物体识别为可抓取目标，从而在人机交互(HRI)中降低了安全性。为此，在这项工作中我们提出了以质量为重点的主动对抗策略（QFAAP）来解决这一问题。 具体来说，该方法的第一部分是敌对质量补丁(AQP)，其中我们设计了敌对质量补丁损失，并利用抓取数据集优化具有高评分的质量补丁。接下来，我们构建了投影质量梯度下降(PQGD)并将其与AQP集成在一起，PQGD中仅包含每个实时帧中的手部区域，从而使AQP能够快速适应手形的变化。通过使用AQP和PQGD，可以让手主动对抗周围的物体，降低它们的质量分数。因此，进一步将手的质量分数设为零，可以降低手及其周围物体的抓取优先级，使得机器人可以在不进行紧急停止的情况下避开对手部的抓取。 我们在基准数据集和协作机器人的实验中展示了QFAAP的有效性。我们的代码和演示视频可在以下链接获取：[提供链接]。
URL
https://arxiv.org/abs/2503.19397
PDF
-
Stop Walking in Circles! Bailing Out Early in Projected Gradient Descent
2025-03-25 04:51:44Philip Doldo, Derek Everett, Amol Khanna, Andre T Nguyen, Edward RaffAbstract
Projected Gradient Descent (PGD) under the $L_\infty$ ball has become one of the defacto methods used in adversarial robustness evaluation for computer vision (CV) due to its reliability and efficacy, making a strong and easy-to-implement iterative baseline. However, PGD is computationally demanding to apply, especially when using thousands of iterations is the current best-practice recommendation to generate an adversarial example for a single image. In this work, we introduce a simple novel method for early termination of PGD based on cycle detection by exploiting the geometry of how PGD is implemented in practice and show that it can produce large speedup factors while providing the \emph{exact} same estimate of model robustness as standard PGD. This method substantially speeds up PGD without sacrificing any attack strength, enabling evaluations of robustness that were previously computationally intractable.
Abstract (translated)
基于$L_\infty$球的投影梯度下降（Projected Gradient Descent，PGD）方法由于其可靠性和有效性，在计算机视觉（CV）领域的对抗鲁棒性评估中已经成为了一种事实上的标准方法。然而，这种方法在计算上非常耗费资源，尤其是在当前最佳实践建议使用数千次迭代来为单张图像生成一个对抗样本时更是如此。在这项工作中，我们提出了一种简单的新方法，通过利用实际实现PGD的几何特性来进行循环检测以提前终止PGD，并展示了该方法能够在不牺牲模型鲁棒性评估准确性的前提下产生显著的速度提升。具体来说，它能够提供与标准PGD完全相同的模型鲁棒性估计结果，但速度却快得多。这种方法极大地加快了PGD的执行速度，同时没有削弱攻击力度，使得之前由于计算资源限制而无法进行的对抗鲁棒性评估成为可能。
URL
https://arxiv.org/abs/2503.19347
PDF
-
Leveraging Perturbation Robustness to Enhance Out-of-Distribution Detection
2025-03-24 15:32:33Wenxi Chen, Raymond A. Yeh, Shaoshuai Mou, Yan GuAbstract
Out-of-distribution (OOD) detection is the task of identifying inputs that deviate from the training data distribution. This capability is essential for safely deploying deep computer vision models in open-world environments. In this work, we propose a post-hoc method, Perturbation-Rectified OOD detection (PRO), based on the insight that prediction confidence for OOD inputs is more susceptible to reduction under perturbation than in-distribution (IND) inputs. Based on the observation, we propose an adversarial score function that searches for the local minimum scores near the original inputs by applying gradient descent. This procedure enhances the separability between IND and OOD samples. Importantly, the approach improves OOD detection performance without complex modifications to the underlying model architectures. We conduct extensive experiments using the OpenOOD benchmark~\cite{yang2022openood}. Our approach further pushes the limit of softmax-based OOD detection and is the leading post-hoc method for small-scale models. On a CIFAR-10 model with adversarial training, PRO effectively detects near-OOD inputs, achieving a reduction of more than 10\% on FPR@95 compared to state-of-the-art methods.
Abstract (translated)
异常分布（OOD，Out-of-Distribution）检测的任务是识别那些与训练数据分布不同的输入。这一能力对于在开放世界环境中安全部署深度计算机视觉模型至关重要。在这项工作中，我们提出了一种事后方法——扰动矫正异常检测（PRO），基于这样的见解：对于异常分布的输入，其预测置信度更易受到扰动的影响而降低，相比之下，在分布内（IND）的输入则不然。根据这一观察结果，我们提出了一个对抗性评分函数，通过应用梯度下降来寻找原始输入附近的局部最低分数。这种方法增强了在分布内和异常分布样本之间的可分离性。重要的是，此方法能够在不进行复杂的模型架构修改的情况下提升OOD检测性能。 我们在OpenOOD基准测试~\cite{yang2022openood}上进行了广泛的实验。我们的方法进一步推动了基于softmax的OOD检测的极限，并且是小规模模型中领先的事后方法之一。在经过对抗性训练的CIFAR-10模型上，PRO能够有效地检测近似异常分布的数据输入，在FPR@95指标上的表现比当前最佳的方法减少了超过10%。 这种方法不仅增强了深度学习模型处理未见过数据的能力，还为实际应用中的安全性提供了重要的保障。
URL
https://arxiv.org/abs/2503.18784
PDF
-
Vision-Language Gradient Descent-driven All-in-One Deep Unfolding Networks
2025-03-21 08:02:48Haijin Zeng, Xiangming Wang, Yongyong Chen, Jingyong Su, Jie LiuAbstract
Dynamic image degradations, including noise, blur and lighting inconsistencies, pose significant challenges in image restoration, often due to sensor limitations or adverse environmental conditions. Existing Deep Unfolding Networks (DUNs) offer stable restoration performance but require manual selection of degradation matrices for each degradation type, limiting their adaptability across diverse scenarios. To address this issue, we propose the Vision-Language-guided Unfolding Network (VLU-Net), a unified DUN framework for handling multiple degradation types simultaneously. VLU-Net leverages a Vision-Language Model (VLM) refined on degraded image-text pairs to align image features with degradation descriptions, selecting the appropriate transform for target degradation. By integrating an automatic VLM-based gradient estimation strategy into the Proximal Gradient Descent (PGD) algorithm, VLU-Net effectively tackles complex multi-degradation restoration tasks while maintaining interpretability. Furthermore, we design a hierarchical feature unfolding structure to enhance VLU-Net framework, efficiently synthesizing degradation patterns across various levels. VLU-Net is the first all-in-one DUN framework and outperforms current leading one-by-one and all-in-one end-to-end methods by 3.74 dB on the SOTS dehazing dataset and 1.70 dB on the Rain100L deraining dataset.
Abstract (translated)
动态图像退化，包括噪声、模糊和光照不一致等问题，在图像恢复领域提出了重大挑战，这些问题通常由传感器限制或不利的环境条件引起。现有的深度解卷网络（DUNs）提供了稳定的恢复性能，但需要为每种降质类型手动选择降质矩阵，这限制了它们在各种场景中的适应性。为了解决这个问题，我们提出了一种基于视觉-语言引导的解卷网络（VLU-Net），这是一种统一的DUN框架，能够同时处理多种退化类型。 VLU-Net利用一个经过退化图像-文本对训练的视觉-语言模型（VLM）来将图像特征与降质描述进行对齐，并选择适当的转换以适应目标降质。通过在近似梯度下降算法中集成一种基于VLM的自动梯度估计策略，VLU-Net能够有效处理复杂的多退化恢复任务，同时保持可解释性。 此外，我们设计了一种分层特征解卷结构，增强了VLU-Net框架，在不同层次上高效地合成退化模式。VLU-Net是首个全功能的DUN框架，并且在SOTS去雾数据集和Rain100L除雨数据集中分别优于当前领先的逐一处理方法以及端到端一体化方法3.74 dB和1.70 dB。
URL
https://arxiv.org/abs/2503.16930
PDF
-
Restoring Forgotten Knowledge in Non-Exemplar Class Incremental Learning through Test-Time Semantic Evolution
2025-03-21 02:02:35Haori Lu, Xusheng Cao, Linlan Huang, Enguang Wang, Fei Yang, Xialei LiuAbstract
Continual learning aims to accumulate knowledge over a data stream while mitigating catastrophic forgetting. In Non-exemplar Class Incremental Learning (NECIL), forgetting arises during incremental optimization because old classes are inaccessible, hindering the retention of prior knowledge. To solve this, previous methods struggle in achieving the stability-plasticity balance in the training stages. However, we note that the testing stage is rarely considered among them, but is promising to be a solution to forgetting. Therefore, we propose RoSE, which is a simple yet effective method that \textbf{R}est\textbf{o}res forgotten knowledge through test-time \textbf{S}emantic \textbf{E}volution. Specifically designed for minimizing forgetting, RoSE is a test-time semantic drift compensation framework that enables more accurate drift estimation in a self-supervised manner. Moreover, to avoid incomplete optimization during online testing, we derive an analytical solution as an alternative to gradient descent. We evaluate RoSE on CIFAR-100, TinyImageNet, and ImageNet100 datasets, under both cold-start and warm-start settings. Our method consistently outperforms most state-of-the-art (SOTA) methods across various scenarios, validating the potential and feasibility of test-time evolution in NECIL.
Abstract (translated)
持续学习旨在通过数据流积累知识，同时减轻灾难性遗忘。在非示例类增量学习（NECIL）中，由于旧类别不可访问，所以在增量优化过程中会引发遗忘，从而阻碍先前知识的保留。为了解决这个问题，以往的方法在训练阶段难以实现稳定性和可塑性的平衡。然而，我们注意到，在这些方法中很少考虑测试阶段，但该阶段有可能成为解决遗忘问题的一种解决方案。因此，我们提出了一种简单而有效的方法——RoSE（通过测试时间语义进化恢复遗忘知识）。具体而言，为最小化遗忘设计的RoSE是一个测试时语义漂移补偿框架，它以自监督的方式使更准确的漂移估计得以实现。此外，为了在在线测试过程中避免不完全优化，我们推导出了一种替代梯度下降的方法作为解析解。 我们在冷启动和温启动设置下对CIFAR-100、TinyImageNet 和 ImageNet100 数据集上评估了RoSE。我们的方法在各种场景中均超过了大多数最先进的（SOTA）方法，验证了测试时进化在NECIL中的潜力和可行性。
URL
https://arxiv.org/abs/2503.16793
PDF
-
Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiency
2025-03-18 09:53:48Jiangxuan Long, Zhao Song, Chiwun YangAbstract
Recent studies suggest utilizing generative models instead of traditional auto-regressive algorithms for time series forecasting (TSF) tasks. These non-auto-regressive approaches involving different generative methods, including GAN, Diffusion, and Flow Matching for time series, have empirically demonstrated high-quality generation capability and accuracy. However, we still lack an appropriate understanding of how it processes approximation and generalization. This paper presents the first theoretical framework from the perspective of flow-based generative models to relieve the knowledge of limitations. In particular, we provide our insights with strict guarantees from three perspectives: $\textbf{Approximation}$, $\textbf{Generalization}$ and $\textbf{Efficiency}$. In detail, our analysis achieves the contributions as follows: $\bullet$ By assuming a general data model, the fitting of the flow-based generative models is confirmed to converge to arbitrary error under the universal approximation of Diffusion Transformer (DiT). $\bullet$ Introducing a polynomial-based regularization for flow matching, the generalization error thus be bounded since the generalization of polynomial approximation. $\bullet$ The sampling for generation is considered as an optimization process, we demonstrate its fast convergence with updating standard first-order gradient descent of some objective.
Abstract (translated)
最近的研究表明，在时间序列预测（TSF）任务中，使用生成模型替代传统的自回归算法可以带来更好的效果。这些非自回归方法包括GAN、扩散和流匹配等不同的生成技术，并且在实践中展示了高质量的生成能力和准确性。然而，我们仍缺乏对它们如何进行近似处理以及如何泛化理解的知识。本文首次从基于流的生成模型的角度提出了一个理论框架来澄清其局限性知识。特别地，从以下三个角度提供了严格保证并给出了我们的见解：**近似**、**泛化** 和 **效率**。 具体来说，我们分析的主要贡献如下： - 假设了一个一般数据模型，基于流的生成模型在扩散变换器（DiT）通用逼近下被确认其拟合可以收敛到任意误差。 - 通过引入基于多项式的正则化方法用于流匹配，由于多项式近似的泛化特性，因此可以将泛化误差限制在一个范围内。 - 将生成过程中的采样视为一个优化过程，并展示了通过更新标准的一阶梯度下降目标函数能够实现快速收敛。 这些贡献共同提供了一个全面的理解框架，帮助我们更好地理解基于流的生成模型在时间序列预测任务中的表现及其潜在的应用前景。
URL
https://arxiv.org/abs/2503.14076
PDF
-
Analysis of human visual field information using machine learning methods and assessment of their accuracy
2025-03-18 07:39:41A. I. Medvedeva, V. V. BakutkinAbstract
Subject of research: is the study of methods for analyzing perimetric images for the diagnosis and control of glaucoma diseases. Objects of research: is a dataset collected on the ophthalmological perimeter with the results of various patient pathologies, since the ophthalmological community is acutely aware of the issue of disease control and import substitution. [5]. Purpose of research: is to consider various machine learning methods that can classify glaucoma. This is possible thanks to the classifier built after labeling the dataset. It is able to determine from the image whether the visual fields depicted on it are the results of the impact of glaucoma on the eyes or other visual diseases. Earlier in the work [3], a dataset was described that was collected on the Tomey perimeter. The average age of the examined patients ranged from 30 to 85 years. Methods of research: machine learning methods for classifying image results (stochastic gradient descent, logistic regression, random forest, naive Bayes). Main results of research: the result of the study is computer modeling that can determine from the image whether the result is glaucoma or another disease (binary classification).
Abstract (translated)
研究主题：该研究旨在探讨分析周边视野图像以诊断和监控青光眼疾病的方法。 研究对象：是一组在眼科视野计上收集的数据集，包含各种患者病状的结果。眼科界对于疾病的控制和进口替代问题十分关注。 研究目的：评估能够分类青光眼的各种机器学习方法。通过给数据集贴标签后建立的分类器，可以从图像中判断出描绘的视觉场是否受到青光眼的影响或其它视网膜疾病影响。之前的工作[3]描述了一个在Tomey视野计上收集的数据集。接受检查患者的平均年龄范围是30至85岁。 研究方法：使用机器学习方法对图像结果进行分类（随机梯度下降、逻辑回归、随机森林、朴素贝叶斯）。 主要研究成果：该研究的结果是一种计算机模拟模型，能够从图像中判断出视觉场是否为青光眼所致或其他疾病（二元分类）。
URL
https://arxiv.org/abs/2503.14562
PDF
-
Optimizing ML Training with Metagradient Descent
2025-03-17 22:18:24Logan Engstrom, Andrew Ilyas, Benjamin Chen, Axel Feldmann, William Moses, Aleksander MadryAbstract
A major challenge in training large-scale machine learning models is configuring the training process to maximize model performance, i.e., finding the best training setup from a vast design space. In this work, we unlock a gradient-based approach to this problem. We first introduce an algorithm for efficiently calculating metagradients -- gradients through model training -- at scale. We then introduce a "smooth model training" framework that enables effective optimization using metagradients. With metagradient descent (MGD), we greatly improve on existing dataset selection methods, outperform accuracy-degrading data poisoning attacks by an order of magnitude, and automatically find competitive learning rate schedules.
Abstract (translated)
训练大规模机器学习模型的一个主要挑战是配置训练过程以最大化模型性能，即从庞大的设计空间中找到最佳的训练设置。在这项工作中，我们解锁了一种基于梯度的方法来解决这个问题。首先，我们引入了一种高效计算元梯度（通过模型训练的梯度）的大规模算法。然后，我们介绍了一个“平滑模型训练”框架，该框架能够利用元梯度进行有效优化。使用元梯度下降（MGD），我们在现有数据集选择方法上取得了显著改进，以一个数量级的优势超越了降低准确性的数据投毒攻击，并且可以自动找到竞争力强的学习率调度。
URL
https://arxiv.org/abs/2503.13751
PDF
-
Beyond Propagation of Chaos: A Stochastic Algorithm for Mean Field Optimization
2025-03-17 12:37:53Chandan Tankala, Dheeraj M. Nagaraj, Anant RajAbstract
Gradient flow in the 2-Wasserstein space is widely used to optimize functionals over probability distributions and is typically implemented using an interacting particle system with $n$ particles. Analyzing these algorithms requires showing (a) that the finite-particle system converges and/or (b) that the resultant empirical distribution of the particles closely approximates the optimal distribution (i.e., propagation of chaos). However, establishing efficient sufficient conditions can be challenging, as the finite particle system may produce heavily dependent random variables. In this work, we study the virtual particle stochastic approximation, originally introduced for Stein Variational Gradient Descent. This method can be viewed as a form of stochastic gradient descent in the Wasserstein space and can be implemented efficiently. In popular settings, we demonstrate that our algorithm's output converges to the optimal distribution under conditions similar to those for the infinite particle limit, and it produces i.i.d. samples without the need to explicitly establish propagation of chaos bounds.
Abstract (translated)
在2-Wasserstein空间中的梯度流被广泛用于优化概率分布上的泛函，并通常通过使用具有$n$个粒子的相互作用粒子系统来实现。分析这类算法需要证明（a）有限粒子系统的收敛性，以及/或(b) 粒子的最终经验分布能够接近最优分布（即传播混沌）。然而，建立有效的充分条件可能颇具挑战性，因为有限粒子系统可能会产生高度相关的随机变量。 在这项工作中，我们研究了虚拟粒子随机逼近方法，该方法最初是为Stein变分梯度下降所引入。这种方法可以被视为Wasserstein空间中的某种形式的随机梯度下降，并且可以高效地实现。在流行的应用场景中，我们证明了我们的算法输出可以在与无限粒子极限类似的条件下收敛到最优分布，并且它能够产生独立同分布（i.i.d.）样本，而无需显式建立传播混沌界。 这段文本描述了一种新的方法，即虚拟粒子随机逼近法，用于解决优化概率分布上的泛函问题。这种方法不仅理论上有效，而且在实际应用中也能高效地生成接近最优解的独立同分布样本，避免了直接处理复杂的相关性问题。
URL
https://arxiv.org/abs/2503.13115
PDF
-
LAPIG: Language Guided Projector Image Generation with Surface Adaptation and Stylization
2025-03-15 15:31:04Yuchen Deng, Haibin Ling, Bingyao HuangAbstract
We propose LAPIG, a language guided projector image generation method with surface adaptation and stylization. LAPIG consists of a projector-camera system and a target textured projection surface. LAPIG takes the user text prompt as input and aims to transform the surface style using the projector. LAPIG's key challenge is that due to the projector's physical brightness limitation and the surface texture, the viewer's perceived projection may suffer from color saturation and artifacts in both dark and bright regions, such that even with the state-of-the-art projector compensation techniques, the viewer may see clear surface texture-related artifacts. Therefore, how to generate a projector image that follows the user's instruction while also displaying minimum surface artifacts is an open problem. To address this issue, we propose projection surface adaptation (PSA) that can generate compensable surface stylization. We first train two networks to simulate the projector compensation and project-and-capture processes, this allows us to find a satisfactory projector image without real project-and-capture and utilize gradient descent for fast convergence. Then, we design content and saturation losses to guide the projector image generation, such that the generated image shows no clearly perceivable artifacts when projected. Finally, the generated image is projected for visually pleasing surface style morphing effects. The source code and video are available on the project page: this https URL.
Abstract (translated)
我们提出了一种名为LAPIG的语言引导投影仪图像生成方法，该方法具备表面适应和风格化功能。LAPIG系统由一个投影仪-摄像机系统和一个目标纹理投影表面组成。LAPIG接收用户文本提示作为输入，并旨在使用投影仪改变表面样式。LAPIG面临的关键挑战是由于投影仪物理亮度限制以及表面纹理的存在，观众所感知到的投影图像在暗区与亮区可能会遭受颜色饱和度降低和失真现象，即使采用最先进的投影补偿技术也无法完全消除这些问题，使得观者仍能察觉到明显的表面纹理相关瑕疵。因此，在遵循用户指令的同时生成一个能够最小化表面瑕疵的投影仪图像成为了亟待解决的问题。 为了解决这个问题，我们提出了一种名为“投影表面适应”（PSA）的技术，该技术可以生成可补偿的表面风格化效果。首先，我们训练了两个网络来模拟投影补偿和投射捕捉过程，这使我们在不进行实际投射捕捉的情况下也能找到满意的投影仪图像，并利用梯度下降法实现快速收敛。随后，我们设计了内容损失和饱和度损失以指导投影仪图像的生成，从而使得生成的图像在被投射时不会产生明显的可感知瑕疵。最终，所生成的图像将被用于视觉上吸引人的表面样式变形效果。 项目的源代码和演示视频可在以下链接获取：[请在此处插入实际URL]
URL
https://arxiv.org/abs/2503.12173
PDF
-
Adaptive Stochastic Gradient Descents on Manifolds with an Application on Weighted Low-Rank Approximation
Abstract
We prove a convergence theorem for stochastic gradient descents on manifolds with adaptive learning rate and apply it to the weighted low-rank approximation problem.
Abstract (translated)
我们证明了具有自适应学习率的流形上随机梯度下降的收敛性定理，并将其应用于加权低秩近似问题。
URL
https://arxiv.org/abs/2503.11833
PDF
-
Multiplicative Learning
2025-03-13 08:14:00Han Kim, Hyungjoon Soh, Vipul Periwal, Junghyo JoAbstract
Efficient training of artificial neural networks remains a key challenge in deep learning. Backpropagation (BP), the standard learning algorithm, relies on gradient descent and typically requires numerous iterations for convergence. In this study, we introduce Expectation Reflection (ER), a novel learning approach that updates weights multiplicatively based on the ratio of observed to predicted outputs. Unlike traditional methods, ER maintains consistency without requiring ad hoc loss functions or learning rate hyperparameters. We extend ER to multilayer networks and demonstrate its effectiveness in performing image classification tasks. Notably, ER achieves optimal weight updates in a single iteration. Additionally, we reinterpret ER as a modified form of gradient descent incorporating the inverse mapping of target propagation. These findings suggest that ER provides an efficient and scalable alternative for training neural networks.
Abstract (translated)
人工神经网络的高效训练仍然是深度学习中的一个关键挑战。反向传播（BP）作为标准的学习算法，依赖于梯度下降，并且通常需要多次迭代才能收敛。在本研究中，我们引入了期望反射（ER），这是一种新的学习方法，它根据观察到的输出与预测输出的比例来乘法更新权重。不同于传统的方法，ER能够在不使用临时损失函数或学习率超参数的情况下保持一致性。我们将ER扩展到了多层网络，并展示了其在执行图像分类任务方面的有效性。特别地，ER可以在一次迭代中实现最优的权重更新。此外，我们重新诠释了ER为一种结合目标传播逆映射的修改版梯度下降法。这些发现表明，ER为训练神经网络提供了一种高效且可扩展的替代方案。
URL
https://arxiv.org/abs/2503.10144
PDF
-
Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $mu$P Parametrization
2025-03-12 17:33:13Zixiang Chen, Greg Yang, Qingyue Zhao, Quanquan GuAbstract
Despite deep neural networks' powerful representation learning capabilities, theoretical understanding of how networks can simultaneously achieve meaningful feature learning and global convergence remains elusive. Existing approaches like the neural tangent kernel (NTK) are limited because features stay close to their initialization in this parametrization, leaving open questions about feature properties during substantial evolution. In this paper, we investigate the training dynamics of infinitely wide, $L$-layer neural networks using the tensor program (TP) framework. Specifically, we show that, when trained with stochastic gradient descent (SGD) under the Maximal Update parametrization ($\mu$P) and mild conditions on the activation function, SGD enables these networks to learn linearly independent features that substantially deviate from their initial values. This rich feature space captures relevant data information and ensures that any convergent point of the training process is a global minimum. Our analysis leverages both the interactions among features across layers and the properties of Gaussian random variables, providing new insights into deep representation learning. We further validate our theoretical findings through experiments on real-world datasets.
Abstract (translated)
尽管深度神经网络具有强大的表示学习能力，但关于如何同时实现有意义的特征学习和全局收敛的理论理解仍然不明确。现有的方法，如神经切片核（NTK），由于在这种参数化中特征保持接近初始化状态而受到限制，从而留下了有关特征在显著演变过程中属性的问题。在这篇论文中，我们使用张量程序（TP）框架研究了无限宽、$L$层神经网络的训练动力学。具体来说，我们展示了当使用随机梯度下降法（SGD）和激活函数的一些温和条件下，在最大更新参数化（$\mu$P）下训练时，SGD使这些网络能够学习与初始值显著不同的线性独立特征。这一丰富的特征空间捕捉到了相关的数据信息，并确保了训练过程中的任何收敛点都是全局最小值。我们的分析利用了层间特征之间的相互作用以及高斯随机变量的属性，为深度表示学习提供了新的见解。我们进一步通过在现实世界数据集上的实验验证了理论发现。
URL
https://arxiv.org/abs/2503.09565
PDF
-
Constraint-Guided Learning of Data-driven Health Indicator Models: An Application on the Pronostia Bearing Dataset
2025-03-12 07:01:27Yonas Tefera, Quinten Van Baelen, Maarten Meire, Stijn Luca, Peter KarsmakersAbstract
This paper presents a constraint-guided deep learning framework for developing physically consistent health indicators in bearing prognostics and health management. Conventional data-driven methods often lack physical plausibility, while physics-based models are limited by incomplete system knowledge. To address this, we integrate domain knowledge into deep learning using constraints to enforce monotonicity, bound output values between 1 and 0 (representing healthy to failed states), and ensure consistency between signal energy trends and health indicator estimates. This eliminates the need for complex loss term balancing. We implement constraint-guided gradient descent within an autoencoder architecture, creating a constrained autoencoder. However, the framework is adaptable to other architectures. Using time-frequency representations of accelerometer signals from the Pronostia dataset, our constrained model generates smoother, more reliable degradation profiles compared to conventional methods, aligning with expected physical behavior. Performance is assessed using three metrics: trendability, robustness, and consistency. Compared to a conventional baseline, the constrained model improves all three. Another baseline, incorporating monotonicity via a soft-ranking loss function, outperforms in trendability but falls short in robustness and consistency. An ablation study confirms that the monotonicity constraint enhances trendability, the boundary constraint ensures consistency, and the energy-health consistency constraint improves robustness. These findings highlight the effectiveness of constraint-guided deep learning in producing reliable, physically meaningful health indicators, offering a promising direction for future prognostic applications.
Abstract (translated)
本文提出了一种约束引导的深度学习框架，用于开发轴承预测与健康管理中的物理一致性的健康指标。传统数据驱动的方法往往缺乏物理合理性，而基于物理学的模型则受限于系统知识不完整的问题。为了应对这一挑战，我们将领域知识通过施加单调性、限定输出值在0到1之间（代表从健康到故障的状态）以及确保信号能量趋势与健康指标估计的一致性的约束条件整合到了深度学习中。这种做法消除了复杂损失项平衡的需要。 我们在自编码器架构内实现了引导梯度下降的约束，从而创建了一个受限制的自动编码器模型。然而，该框架可以适应其他架构使用。通过采用Pronostia数据集中的加速度计信号的时间-频率表示形式，我们的受限模型与传统方法相比生成了更平滑、更可靠的退化曲线，并且符合预期的物理行为。 性能评估采用了三种指标：趋势性（trendability）、鲁棒性和一致性。与传统的基准相比，受约束的模型在所有三项上都有所提升。另一个基准线通过使用软排序损失函数来引入单调性，在趋势性方面表现更佳，但在鲁棒性和一致性方面的表现却逊色于前者。消融实验确认了单调性约束提高了趋势性、边界约束确保了一致性以及能量-健康一致性的约束提升了鲁棒性。 这些发现强调了约束引导的深度学习在生成可靠的、物理意义明确的健康指标的有效性，为未来预测应用提供了一个有前景的方向。
URL
https://arxiv.org/abs/2503.09113
PDF
-
A Triple-Inertial Accelerated Alternating Optimization Method for Deep Learning Training
2025-03-11 14:42:17Chengcheng Yan, Jiawei Xu, Qingsong Wang, Zheng PengAbstract
The stochastic gradient descent (SGD) algorithm has achieved remarkable success in training deep learning models. However, it has several limitations, including susceptibility to vanishing gradients, sensitivity to input data, and a lack of robust theoretical guarantees. In recent years, alternating minimization (AM) methods have emerged as a promising alternative for model training by employing gradient-free approaches to iteratively update model parameters. Despite their potential, these methods often exhibit slow convergence rates. To address this challenge, we propose a novel Triple-Inertial Accelerated Alternating Minimization (TIAM) framework for neural network training. The TIAM approach incorporates a triple-inertial acceleration strategy with a specialized approximation method, facilitating targeted acceleration of different terms in each sub-problem optimization. This integration improves the efficiency of convergence, achieving superior performance with fewer iterations. Additionally, we provide a convergence analysis of the TIAM algorithm, including its global convergence properties and convergence rate. Extensive experiments validate the effectiveness of the TIAM method, showing significant improvements in generalization capability and computational efficiency compared to existing approaches, particularly when applied to the rectified linear unit (ReLU) and its variants.
Abstract (translated)
随机梯度下降（SGD）算法在训练深度学习模型方面取得了显著的成功。然而，它存在几个局限性，包括易受消失梯度的影响、对输入数据的敏感性以及缺乏坚实的理论保证。近年来，交替最小化（AM）方法作为一种有潜力的替代方案出现，通过采用无梯度的方法迭代更新模型参数来训练模型。尽管这些方法具有潜在的优势，但它们通常表现出较慢的收敛速度。为了解决这一挑战，我们提出了一种新颖的三惯性加速交替最小化（Triple-Inertial Accelerated Alternating Minimization, TIAM）框架用于神经网络训练。TIAM 方法采用了一个三惯性加速策略，并结合了专门的近似方法，使不同项在每个子问题优化中的目标加速成为可能。这种集成提高了收敛效率，在较少迭代次数的情况下实现了更好的性能表现。此外，我们还提供了对 TIAM 算法的收敛分析，包括其全局收敛性质和收敛速度。广泛的实验验证了 TIAM 方法的有效性，并表明与现有的方法相比，特别是在应用于修正线性单元（ReLU）及其变体时，TIAM 在泛化能力和计算效率方面取得了显著改进。
URL
https://arxiv.org/abs/2503.08489
PDF
-
Mirror Descent and Novel Exponentiated Gradient Algorithms Using Trace-Form Entropies and Deformed Logarithms
2025-03-11 10:50:07Andrzej Cichocki, Toshihisa Tanaka, Sergio CrucesAbstract
In this paper we propose and investigate a wide class of Mirror Descent updates (MD) and associated novel Generalized Exponentiated Gradient (GEG) algorithms by exploiting various trace-form entropies and associated deformed logarithms and their inverses - deformed (generalized) exponential functions. The proposed algorithms can be considered as extension of entropic MD and generalization of multiplicative updates. In the literature, there exist nowadays over fifty mathematically well defined generalized entropies, so impossible to exploit all of them in one research paper. So we focus on a few selected most popular entropies and associated logarithms like the Tsallis, Kaniadakis and Sharma-Taneja-Mittal and some of their extension like Tempesta or Kaniadakis-Scarfone entropies. The shape and properties of the deformed logarithms and their inverses are tuned by one or more hyperparameters. By learning these hyperparameters, we can adapt to distribution of training data, which can be designed to the specific geometry of the optimization problem, leading to potentially faster convergence and better performance. The using generalized entropies and associated deformed logarithms in the Bregman divergence, used as a regularization term, provides some new insight into exponentiated gradient descent updates.
Abstract (translated)
在这篇论文中，我们提出并研究了一类广泛的镜像下降更新（MD）和相关的新型广义指数梯度（GEG）算法。通过利用各种迹形式的熵以及相应的变形对数及其逆——即变形（广义）指数函数，我们开发了这些算法。提出的算法可以被视为熵型 MD 的扩展，并且是乘法更新的一般化。 目前文献中存在超过五十种数学上严格定义的广义熵，因此不可能在一篇研究论文中全部利用它们。所以我们集中关注几种最流行的熵及其对应的对数，如 Tsallis 熵、Kaniadakis 熵和 Sharma-Taneja-Mittal 熵，还有一些它们的扩展形式，比如 Tempesta 和 Kaniadakis-Scarfone 熵。 变形对数及其逆的形式和性质可以通过一个或多个超参数进行调整。通过学习这些超参数，我们可以适应训练数据分布，并根据优化问题的具体几何特性对其进行设计，从而可能实现更快的收敛速度和更好的性能表现。在 Bregman 散度中使用广义熵及相关变形对数作为正则化项，为指数梯度下降更新提供了一些新的见解。
URL
https://arxiv.org/abs/2503.08748
PDF
-
IDInit: A Universal and Stable Initialization Method for Neural Network Training
2025-03-06 17:12:46Yu Pan, Chaozheng Wang, Zekai Wu, Qifan Wang, Min Zhang, Zenglin XuAbstract
Deep neural networks have achieved remarkable accomplishments in practice. The success of these networks hinges on effective initialization methods, which are vital for ensuring stable and rapid convergence during training. Recently, initialization methods that maintain identity transition within layers have shown good efficiency in network training. These techniques (e.g., Fixup) set specific weights to zero to achieve identity control. However, settings of remaining weight (e.g., Fixup uses random values to initialize non-zero weights) will affect the inductive bias that is achieved only by a zero weight, which may be harmful to training. Addressing this concern, we introduce fully identical initialization (IDInit), a novel method that preserves identity in both the main and sub-stem layers of residual networks. IDInit employs a padded identity-like matrix to overcome rank constraints in non-square weight matrices. Furthermore, we show the convergence problem of an identity matrix can be solved by stochastic gradient descent. Additionally, we enhance the universality of IDInit by processing higher-order weights and addressing dead neuron problems. IDInit is a straightforward yet effective initialization method, with improved convergence, stability, and performance across various settings, including large-scale datasets and deep models.
Abstract (translated)
深度神经网络在实践中已经取得了显著的成就。这些网络的成功很大程度上取决于有效的初始化方法，这对于确保训练过程中的稳定和快速收敛至关重要。最近，保持层内恒等变换（identity transition）的初始化方法显示出在网络训练中良好的效率。例如，Fixup技术通过将特定权重设置为零来实现恒等控制。然而，剩余权重的设定（如Fixup使用随机值来初始化非零权重）会影响仅由零权重所达成的归纳偏差，这可能会对训练产生负面影响。 针对这一问题，我们引入了一种新的方法——完全相同的初始化（IDInit），该方法在残差网络的主要和子茎层中都保持恒等变换。IDInit采用填充后的类似恒等矩阵来克服非方阵权重矩阵中的秩约束。此外，我们还展示了如何通过随机梯度下降法解决恒等矩阵的收敛问题。为了增强IDInit的通用性，我们处理了高阶权重，并解决了“死神经元”（dead neuron）的问题。 总之，IDInit是一种简单而有效的方法，在各种设置下提高了收敛性、稳定性和性能，包括大规模数据集和深度模型。
URL
https://arxiv.org/abs/2503.04626
PDF
-
Provable Benefits of Task-Specific Prompts for In-context Learning
2025-03-03 22:37:03Xiangyu Chang, Yingcong Li, Muti Kara, Samet Oymak, Amit K. Roy-ChowdhuryAbstract
The in-context learning capabilities of modern language models have motivated a deeper mathematical understanding of sequence models. A line of recent work has shown that linear attention models can emulate projected gradient descent iterations to implicitly learn the task vector from the data provided in the context window. In this work, we consider a novel setting where the global task distribution can be partitioned into a union of conditional task distributions. We then examine the use of task-specific prompts and prediction heads for learning the prior information associated with the conditional task distribution using a one-layer attention model. Our results on loss landscape show that task-specific prompts facilitate a covariance-mean decoupling where prompt-tuning explains the conditional mean of the distribution whereas the variance is learned/explained through in-context learning. Incorporating task-specific head further aids this process by entirely decoupling estimation of mean and variance components. This covariance-mean perspective similarly explains how jointly training prompt and attention weights can provably help over fine-tuning after pretraining.
Abstract (translated)
现代语言模型的上下文学习能力促使了对序列模型进行更深层次的数学理解。近期的研究表明，线性注意力模型可以通过模拟投影梯度下降迭代来隐式地从提供的上下文中学习任务向量。在本研究中，我们考虑了一种新颖的情况：全球任务分布可以被划分为条件任务分布的并集。然后，我们探讨了使用特定于任务的提示和预测头部来利用单层注意力模型学习与条件任务分布相关的先验信息的方法。我们的损失景观结果表明，特定于任务的提示促进了协方差-均值解耦，在此过程中提示调整解释了分布的条件均值，而方差则是通过上下文学习被学习/解释出来的。进一步引入特定于任务的头部有助于这个过程，完全分离了对均值和方差分量的估计。这种从协方差-均值的角度同样可以解释为什么联合训练提示与注意力权重能够确凿地帮助在预训练后进行微调的效果优于单独微调。
URL
https://arxiv.org/abs/2503.02102
PDF
-
No Plan but Everything Under Control: Robustly Solving Sequential Tasks with Dynamically Composed Gradient Descent
Abstract
We introduce a novel gradient-based approach for solving sequential tasks by dynamically adjusting the underlying myopic potential field in response to feedback and the world's regularities. This adjustment implicitly considers subgoals encoded in these regularities, enabling the solution of long sequential tasks, as demonstrated by solving the traditional planning domain of Blocks World - without any planning. Unlike conventional planning methods, our feedback-driven approach adapts to uncertain and dynamic environments, as demonstrated by one hundred real-world trials involving drawer manipulation. These experiments highlight the robustness of our method compared to planning and show how interactive perception and error recovery naturally emerge from gradient descent without explicitly implementing them. This offers a computationally efficient alternative to planning for a variety of sequential tasks, while aligning with observations on biological problem-solving strategies.
Abstract (translated)
我们提出了一种基于梯度的方法，通过动态调整底层近视势场来解决序列任务，并根据反馈和世界的规律性进行响应。这种调整隐式地考虑了编码在这些规律中的子目标，从而能够解决长序列任务，例如传统的Blocks World规划领域——无需任何计划即可完成。与传统规划方法不同，我们的基于反馈的方法适应不确定性和动态环境，在一百次涉及抽屉操作的真实世界试验中得到了验证。这些实验突显了我们方法相对于规划的稳健性，并展示了交互感知和错误恢复如何自然而然地从梯度下降中出现，而无需明确实现它们。这为各种序列任务提供了计算效率更高的替代方案，并与生物解决问题策略中的观察结果相一致。
URL
https://arxiv.org/abs/2503.01732
PDF
