Gradient_Descent
-
Compositional Function Networks: A High-Performance Alternative to Deep Neural Networks with Built-in Interpretability
2025-07-28 17:18:40Fang LiAbstract
Deep Neural Networks (DNNs) deliver impressive performance but their black-box nature limits deployment in high-stakes domains requiring transparency. We introduce Compositional Function Networks (CFNs), a novel framework that builds inherently interpretable models by composing elementary mathematical functions with clear semantics. Unlike existing interpretable approaches that are limited to simple additive structures, CFNs support diverse compositional patterns -- sequential, parallel, and conditional -- enabling complex feature interactions while maintaining transparency. A key innovation is that CFNs are fully differentiable, allowing efficient training through standard gradient descent. We demonstrate CFNs' versatility across multiple domains, from symbolic regression to image classification with deep hierarchical networks. Our empirical evaluation shows CFNs achieve competitive performance against black-box models (96.24% accuracy on CIFAR-10) while outperforming state-of-the-art interpretable models like Explainable Boosting Machines. By combining the hierarchical expressiveness and efficient training of deep learning with the intrinsic interpretability of well-defined mathematical functions, CFNs offer a powerful framework for applications where both performance and accountability are paramount.
Abstract (translated)
深度神经网络（DNNs）展现出卓越的性能，但其黑箱特性限制了它们在需要透明度的关键领域中的应用。我们引入了一种新的框架——组合函数网络（CFNs），该框架通过组合具有明确语义的基本数学函数来构建固有的可解释模型。与现有的仅限于简单加法结构的可解释方法不同，CFNs支持多样化的组合模式——顺序、并行和条件组合，这使得它们能够在保持透明性的同时处理复杂的特征交互。 一个关键创新点是，CFNs完全可微分，允许通过标准梯度下降进行高效的训练。我们展示了CFNs在多个领域的灵活性，从符号回归到使用深度层次网络的图像分类。我们的实证评估表明，CFNs在与黑箱模型（如CIFAR-10数据集上的96.24%准确性）竞争性能的同时，还超过了最新的可解释性模型，例如可解释增强机。 通过将深度学习中的分层表达能力和高效的训练方法与明确定义的数学函数所固有的可解释性相结合，CFNs为那些需要兼顾高性能和问责制的应用提供了一个强大的框架。
URL
https://arxiv.org/abs/2507.21004
PDF
-
Beyond Rate Coding: Surrogate Gradients Enable Spike Timing Learning in Spiking Neural Networks
2025-07-21 20:19:19Ziqiao Yu, Pengfei Sun, Dan F. M. GoodmanAbstract
We investigate the extent to which Spiking Neural Networks (SNNs) trained with Surrogate Gradient Descent (Surrogate GD), with and without delay learning, can learn from precise spike timing beyond firing rates. We first design synthetic tasks isolating intra-neuron inter-spike intervals and cross-neuron synchrony under matched spike counts. On more complex spike-based speech recognition datasets (Spiking Heidelberg Digits (SHD) and Spiking Speech Commands (SSC), we construct variants where spike count information is eliminated and only timing information remains, and show that Surrogate GD-trained SNNs are able to perform significantly above chance whereas purely rate-based models perform at chance level. We further evaluate robustness under biologically inspired perturbations -- including Gaussian jitter per spike or per-neuron, and spike deletion -- revealing consistent but perturbation-specific degradation. Networks show a sharp performance drop when spike sequences are reversed in time, with a larger drop in performance from SNNs trained with delays, indicating that these networks are more human-like in terms of behaviour. To facilitate further studies of temporal coding, we have released our modified SHD and SSC datasets.
Abstract (translated)
我们研究了使用替代梯度下降（Surrogate Gradient Descent，Surrogate GD）训练的脉冲神经网络（Spiking Neural Networks，SNNs），在有或没有延迟学习的情况下，能否从精确的脉冲时间而非仅基于放电率来学习。首先，我们设计了一系列合成任务，这些任务孤立地考察了在匹配放电数量的前提下，单个神经元内的脉冲间隔和跨神经元同步性。对于更复杂的基于脉冲的声音识别数据集（Spiking Heidelberg Digits (SHD) 和 Spiking Speech Commands (SSC)，我们构建了一些变体，在其中消除了放电计数信息，只剩下时间信息，并展示了用替代梯度下降法训练的SNNs能够显著地超越随机水平，而单纯基于速率模型的表现则在随机水平。进一步地，我们在生物启发的扰动下评估了它们的鲁棒性——包括每个脉冲或每种神经元上的高斯抖动（Gaussian jitter）和脉冲删除，揭示了与特定干扰相关的但一致性的性能下降。当脉冲序列的时间顺序被逆转时，网络表现出现了急剧下降，而那些使用延迟训练的SNNs的表现下降幅度更大，这表明这些网络在行为上更接近于人类。 为了促进对时间编码进一步的研究，我们已经发布了修改后的SHD和SSC数据集。
URL
https://arxiv.org/abs/2507.16043
PDF
-
Comparative Analysis of Algorithms for the Fitting of Tessellations to 3D Image Data
2025-07-18 15:28:59Andreas Alpers, Orkun Furat, Christian Jung, Matthias Neumann, Claudia Redenbach, Aigerim Saken, Volker SchmidtAbstract
This paper presents a comparative analysis of algorithmic strategies for fitting tessellation models to 3D image data of materials such as polycrystals and foams. In this steadily advancing field, we review and assess optimization-based methods -- including linear and nonlinear programming, stochastic optimization via the cross-entropy method, and gradient descent -- for generating Voronoi, Laguerre, and generalized balanced power diagrams (GBPDs) that approximate voxelbased grain structures. The quality of fit is evaluated on real-world datasets using discrepancy measures that quantify differences in grain volume, surface area, and topology. Our results highlight trade-offs between model complexity, the complexity of the optimization routines involved, and the quality of approximation, providing guidance for selecting appropriate methods based on data characteristics and application needs.
Abstract (translated)
本文对用于拟合多晶材料和泡沫等材料的3D图像数据的镶嵌模型的算法策略进行了比较分析。在这一不断发展的领域中，我们回顾并评估了基于优化的方法（包括线性和非线性规划、通过交叉熵方法进行的随机优化以及梯度下降）以生成Voronoi图、Laguerre图和广义平衡功率图（GBPDs），这些图形可以近似于体素基粒结构。我们使用差异测量指标来评估拟合质量，这些指标量化了颗粒体积、表面积及拓扑结构的差异。我们的研究结果突出了模型复杂性与优化流程复杂性以及近似质量之间的权衡，并为基于数据特性和应用需求选择合适方法提供了指导。
URL
https://arxiv.org/abs/2507.14268
PDF
-
Keep the beat going: Automatic drum transcription with momentum
Abstract
A simple, interpretable way to perform automatic drum transcription is by factoring the magnitude spectrogram of a recorded musical piece using a partially fixed nonnegative matrix factorization. There are two natural ways to optimize the nonnegative matrix factorization, including a multiplicative update rule and projected gradient descent with momentum. The methods differ in their empirical accuracies and theoretical convergence guarantees. This paper summarizes the methods and their time complexities, and it applies the methods to the ENST-Drums data set and an original recording from the author's band, evaluating the empirical accuracy with respect to ground-truth drum annotations. The results indicate that projected gradient descent with momentum leads to higher accuracy for a fixed runtime, and it satisfies stronger convergence guarantees.
Abstract (translated)
一种简单且易于解释的自动鼓谱转录方法是通过使用部分固定的非负矩阵分解来因子化录制音乐作品的幅度频谱图。优化非负矩阵分解有两种自然的方法：乘法更新规则和带有动量的投影梯度下降。这些方法在经验准确性和理论收敛保证方面有所不同。 本文总结了这两种方法及其时间复杂性，并将它们应用于ENST-Drums数据集以及作者乐队的一段原始录音，以根据地面实况鼓标记评估其经验准确性。实验结果表明，在固定运行时间内，带有动量的投影梯度下降法能够达到更高的准确率，并且在收敛保证方面也更强。
URL
https://arxiv.org/abs/2507.12596
PDF
-
Non-Adaptive Adversarial Face Generation
2025-07-16 10:24:54Sunpill Kim, Seunghun Paik, Chanwoo Hwang, Minsu Kim, Jae Hong SeoAbstract
Adversarial attacks on face recognition systems (FRSs) pose serious security and privacy threats, especially when these systems are used for identity verification. In this paper, we propose a novel method for generating adversarial faces-synthetic facial images that are visually distinct yet recognized as a target identity by the FRS. Unlike iterative optimization-based approaches (e.g., gradient descent or other iterative solvers), our method leverages the structural characteristics of the FRS feature space. We figure out that individuals sharing the same attribute (e.g., gender or race) form an attributed subsphere. By utilizing such subspheres, our method achieves both non-adaptiveness and a remarkably small number of queries. This eliminates the need for relying on transferability and open-source surrogate models, which have been a typical strategy when repeated adaptive queries to commercial FRSs are impossible. Despite requiring only a single non-adaptive query consisting of 100 face images, our method achieves a high success rate of over 93% against AWS's CompareFaces API at its default threshold. Furthermore, unlike many existing attacks that perturb a given image, our method can deliberately produce adversarial faces that impersonate the target identity while exhibiting high-level attributes chosen by the adversary.
Abstract (translated)
针对面部识别系统（FRS）的对抗性攻击会对安全和隐私构成严重威胁，尤其是在这些系统用于身份验证时。本文提出了一种生成对抗性面孔的新方法——即视觉上明显不同于目标个体但被面部识别系统识别为特定身份的合成面部图像。与基于迭代优化的方法（如梯度下降或其他迭代求解器）不同，我们的方法利用了FRS特征空间中的结构特性。我们发现，具有相同属性的人（例如性别或种族）形成了有向子球体。通过使用这些子球体，我们的方法既实现了非适应性又显著减少了查询次数。这消除了依赖于迁移性和开源替代模型的需求，后者通常是当无法对商用FRS进行重复适应性查询时的典型策略。尽管仅需一次包括100张面部图像的非适应性查询，我们的方法在针对AWS的CompareFaces API默认阈值下实现了超过93%的成功率。此外，与许多现有攻击通过扰动给定图像来操作不同的是，我们的方法可以故意生成具有由对手选择的高级别属性、冒充目标身份的对抗性面孔。
URL
https://arxiv.org/abs/2507.12107
PDF
-
Deep Equilibrium models for Poisson Imaging Inverse problems via Mirror Descent
2025-07-15 16:33:01Christian Daniele, Silvia Villa, Samuel Vaiter, Luca CalatroniAbstract
Deep Equilibrium Models (DEQs) are implicit neural networks with fixed points, which have recently gained attention for learning image regularization functionals, particularly in settings involving Gaussian fidelities, where assumptions on the forward operator ensure contractiveness of standard (proximal) Gradient Descent operators. In this work, we extend the application of DEQs to Poisson inverse problems, where the data fidelity term is more appropriately modeled by the Kullback-Leibler divergence. To this end, we introduce a novel DEQ formulation based on Mirror Descent defined in terms of a tailored non-Euclidean geometry that naturally adapts with the structure of the data term. This enables the learning of neural regularizers within a principled training framework. We derive sufficient conditions to guarantee the convergence of the learned reconstruction scheme and propose computational strategies that enable both efficient training and fully parameter-free inference. Numerical experiments show that our method outperforms traditional model-based approaches and it is comparable to the performance of Bregman Plug-and-Play methods, while mitigating their typical drawbacks - namely, sensitivity to initialization and careful tuning of hyperparameters. The code is publicly available at this https URL.
Abstract (translated)
深度平衡模型（DEQs）是一种隐式神经网络，具有固定点特性，在学习图像正则化函数方面近期受到了广泛关注，特别是在涉及高斯保真度的情况下。在这种情况下，关于前向操作符的假设保证了标准梯度下降算子的收敛性。在这项工作中，我们将DEQ的应用扩展到了泊松逆问题领域，在此背景下数据保真度项更适宜用Kullback-Leibler散度来建模。 为此，我们提出了一种新的基于镜像下降（Mirror Descent）的DEQ模型，该模型根据特定于任务而非欧几里得几何结构进行定义。这种设计能够自然地适应数据项的特性，并允许在网络框架内学习神经正则化器。我们推导了保证学习重构方案收敛性的充分条件，并提出计算策略以实现高效训练和完全无需参数调整的推理过程。 数值实验表明，与传统模型驱动的方法相比，我们的方法性能更优；同时，在表现上也与Bregman Plug-and-Play方法相当，但避免了后者常见的缺点——即对初始化敏感以及需要精细调参。代码在提供的URL中公开发布。
URL
https://arxiv.org/abs/2507.11461
PDF
-
Flows and Diffusions on the Neural Manifold
2025-07-14 02:26:06Daniel Saragih, Deyu Cao, Tejas BalajiAbstract
Diffusion and flow-based generative models have achieved remarkable success in domains such as image synthesis, video generation, and natural language modeling. In this work, we extend these advances to weight space learning by leveraging recent techniques to incorporate structural priors derived from optimization dynamics. Central to our approach is modeling the trajectory induced by gradient descent as a trajectory inference problem. We unify several trajectory inference techniques under the framework of gradient flow matching, providing a theoretical framework for treating optimization paths as inductive bias. We further explore architectural and algorithmic choices, including reward fine-tuning by adjoint matching, the use of autoencoders for latent weight representation, conditioning on task-specific context data, and adopting informative source distributions such as Kaiming uniform. Experiments demonstrate that our method matches or surpasses baselines in generating in-distribution weights, improves initialization for downstream training, and supports fine-tuning to enhance performance. Finally, we illustrate a practical application in safety-critical systems: detecting harmful covariate shifts, where our method outperforms the closest comparable baseline.
Abstract (translated)
扩散模型和基于流的生成模型在图像合成、视频生成和自然语言建模等领域取得了显著的成功。在这项工作中，我们通过利用最近的技术将这些进展扩展到权重空间学习中，这些技术从优化动力学中推导出结构先验并将其纳入考虑。我们的方法的核心是将梯度下降诱导的轨迹作为轨迹推理问题进行建模。我们将几种轨迹推理技术统一在梯度流匹配框架下，提供了一种理论框架，用于处理优化路径作为归纳偏置。 此外，我们进一步探索了架构和算法选择，包括通过伴随匹配进行奖励微调、使用自编码器表示潜在权重、根据特定任务的上下文数据进行条件设置以及采用如Kaiming均匀分布等信息源分布。实验结果表明，我们的方法在生成符合分布的权重方面与基线相匹敌或超越，并且可以改善下游训练的初始化，支持微调以增强性能。 最后，我们展示了这种方法在一个实际应用中的优势：安全关键系统的有害协变量偏移检测，在此领域中，我们的方法优于最接近的同类基准。
URL
https://arxiv.org/abs/2507.10623
PDF
-
Customize Harmonic Potential Fields via Hybrid Optimization over Homotopic Paths
2025-07-14 01:55:25Shuaikang Wang, Tiecheng Guo, Meng GuoAbstract
Safe navigation within a workspace is a fundamental skill for autonomous robots to accomplish more complex tasks. Harmonic potentials are artificial potential fields that are analytical, globally convergent and provably free of local minima. Thus, it has been widely used for generating safe and reliable robot navigation control policies. However, most existing methods do not allow customization of the harmonic potential fields nor the resulting paths, particularly regarding their topological properties. In this paper, we propose a novel method that automatically finds homotopy classes of paths that can be generated by valid harmonic potential fields. The considered complex workspaces can be as general as forest worlds consisting of numerous overlapping star-obstacles. The method is based on a hybrid optimization algorithm that searches over homotopy classes, selects the structure of each tree-of-stars within the forest, and optimizes over the continuous weight parameters for each purged tree via the projected gradient descent. The key insight is to transform the forest world to the unbounded point world via proper diffeomorphic transformations. It not only facilitates a simpler design of the multi-directional D-signature between non-homotopic paths, but also retain the safety and convergence properties. Extensive simulations and hardware experiments are conducted for non-trivial scenarios, where the navigation potentials are customized for desired homotopic properties. Project page: this https URL.
Abstract (translated)
工作空间内的安全导航是自主机器人完成更复杂任务的一项基本技能。谐波势场是一种人工势场，具有解析性、全局收敛性和无局部极小值的特性，因此被广泛用于生成安全可靠的机器人导航控制策略。然而，现有大多数方法不允许对谐波势场及其路径进行定制化调整，尤其是在拓扑性质方面。本文提出了一种新的方法，可以自动寻找由有效谐波势场所产生的同伦类路径。所考虑的复杂工作空间可以是一片包含大量重叠星形障碍物的森林世界。 该方法基于一种混合优化算法，在这种算法中搜索同伦类、选择森林中的每棵树结构，并通过投影梯度下降法对每个去除障碍后的树进行连续权重参数的优化。关键见解在于，通过适当的微分同胚变换将森林世界转换为无界点世界。这不仅简化了不同非同伦路径之间的多向D签名的设计，而且还保留了安全性和收敛性属性。 进行了广泛的仿真和硬件实验，涵盖了非平凡场景，在这些场景中导航势场被定制化以满足所需同伦性质。项目主页在此链接: [此链接应为实际的URL地址]。
URL
https://arxiv.org/abs/2507.09858
PDF
-
Transformers Don't In-Context Learn Least Squares Regression
2025-07-13 01:09:26Joshua Hill, Benjamin Eyre, Elliot CreagerAbstract
In-context learning (ICL) has emerged as a powerful capability of large pretrained transformers, enabling them to solve new tasks implicit in example input-output pairs without any gradient updates. Despite its practical success, the mechanisms underlying ICL remain largely mysterious. In this work we study synthetic linear regression to probe how transformers implement learning at inference time. Previous works have demonstrated that transformers match the performance of learning rules such as Ordinary Least Squares (OLS) regression or gradient descent and have suggested ICL is facilitated in transformers through the learned implementation of one of these techniques. In this work, we demonstrate through a suite of out-of-distribution generalization experiments that transformers trained for ICL fail to generalize after shifts in the prompt distribution, a behaviour that is inconsistent with the notion of transformers implementing algorithms such as OLS. Finally, we highlight the role of the pretraining corpus in shaping ICL behaviour through a spectral analysis of the learned representations in the residual stream. Inputs from the same distribution as the training data produce representations with a unique spectral signature: inputs from this distribution tend to have the same top two singular vectors. This spectral signature is not shared by out-of-distribution inputs, and a metric characterizing the presence of this signature is highly correlated with low loss.
Abstract (translated)
上下文学习（ICL）已经成为大规模预训练变压器的强大能力，使它们能够在不进行任何梯度更新的情况下解决隐含在示例输入-输出对中的新任务。尽管其实用性得到了成功验证，但其背后的机制仍然很大程度上是个谜。在这项工作中，我们研究了合成线性回归问题，以探讨变压器如何实现在推理过程中学习的能力。先前的工作已经证明，变压器可以达到诸如普通最小二乘（OLS）回归或梯度下降等学习规则的性能，并且提出了变压器通过实现这些技术之一来促进上下文学习的观点。在本项工作中，我们通过一系列分布外泛化实验展示了针对ICL进行训练的变压器在提示分布发生变化后无法推广的行为，这种行为与认为变压器实施诸如OLS算法的观点不符。 最后，我们通过对残差流中所学表示的谱分析强调了预训练语料库在塑造ICL行为中的作用。来自与训练数据相同分布的输入会产生具有独特谱特征的表示：这些分布中的输入通常具有相同的前两个奇异向量。这种谱签名不会被分布外的输入共享，且一个表征该签名存在性的度量高度相关于较低损失值。 简而言之，这项研究表明预训练的数据对于上下文学习行为至关重要，并且变压器在处理新任务时的行为并不总是遵循传统算法如OLS的学习规则。
URL
https://arxiv.org/abs/2507.09440
PDF
-
Optimization Guarantees for Square-Root Natural-Gradient Variational Inference
2025-07-10 15:33:28Navish Kumar, Thomas M\"ollenhoff, Mohammad Emtiyaz Khan, Aurelien LucchiAbstract
Variational inference with natural-gradient descent often shows fast convergence in practice, but its theoretical convergence guarantees have been challenging to establish. This is true even for the simplest cases that involve concave log-likelihoods and use a Gaussian approximation. We show that the challenge can be circumvented for such cases using a square-root parameterization for the Gaussian covariance. This approach establishes novel convergence guarantees for natural-gradient variational-Gaussian inference and its continuous-time gradient flow. Our experiments demonstrate the effectiveness of natural gradient methods and highlight their advantages over algorithms that use Euclidean or Wasserstein geometries.
Abstract (translated)
基于自然梯度下降的变分推理在实践中常常表现出快速收敛的特点，但其理论上的收敛保证却难以确立。即使是最简单的情况——涉及凹对数似然函数并使用高斯近似的情形也是如此。我们展示了在这种情况下，可以通过为高斯协方差采用平方根参数化来克服这一挑战。这种方法建立了自然梯度变分-高斯推理及其连续时间梯度流的新收敛保证。我们的实验表明了自然梯度方法的有效性，并强调了它们相对于使用欧几里得或 Wasserstein 几何算法的优势。
URL
https://arxiv.org/abs/2507.07853
PDF
-
IPPRO: Importance-based Pruning with PRojective Offset for Magnitude-indifferent Structural Pruning
2025-07-10 13:26:24Jaeheun Jung, Jaehyuk Lee, Yeajin Lee, Donghun LeeAbstract
With the growth of demand on neural network compression methods, the structured pruning methods including importance-based approach are actively studied. The magnitude importance and many correlated modern importance criteria often limit the capacity of pruning decision, since the filters with larger magnitudes are not likely to be pruned if the smaller one didn't, even if it is redundant. In this paper, we propose a novel pruning strategy to challenge this dominating effect of magnitude and provide fair chance to each filter to be pruned, by placing it on projective space. After that, we observe the gradient descent movement whether the filters move toward the origin or not, to measure how the filter is likely to be pruned. This measurement is used to construct PROscore, a novel importance score for IPPRO, a novel importance-based structured pruning with magnitude-indifference. Our evaluation results shows that the proposed importance criteria using the projective space achieves near-lossless pruning by reducing the performance drop in pruning, with promising performance after the finetuning. Our work debunks the ``size-matters'' myth in pruning and expands the frontier of importance-based pruning both theoretically and empirically.
Abstract (translated)
随着对神经网络压缩方法需求的增长，包括基于重要性方法的结构化剪枝方法得到了积极的研究。然而，基于幅度的重要性以及其他许多相关现代重要性标准往往限制了剪枝决策的能力，因为即使较大的滤波器是冗余的，如果较小的一个没有被剪枝的话，它也不大可能被剪枝。 在本文中，我们提出了一种新的剪枝策略来挑战这一主导效应，并为每个滤波器提供公平的机会去进行剪枝。我们将这些滤波器置于投影空间中，然后观察梯度下降运动是否朝向原点移动，以此衡量一个滤波器被剪枝的可能性。这种测量用于构建PROscore，这是一种新的重要性分数，专为IPPRO设计——一种基于重要性的结构化剪枝方法，在这种方法中，幅度的差异不再具有决定性作用。 我们的评估结果显示，使用投影空间中的新提出的重要的标准可以实现接近无损的剪枝，并通过减少剪枝后的性能下降来获得令人满意的结果。经过微调后，其表现仍然很有竞争力。我们的工作打破了“大小有关系”的剪枝神话，在理论和经验上都扩展了基于重要性的剪枝边界。
URL
https://arxiv.org/abs/2507.14171
PDF
-
An Enhanced Privacy-preserving Federated Few-shot Learning Framework for Respiratory Disease Diagnosis
2025-07-10 07:47:58Ming Wang, Zhaoyang Duan, Dong Xue, Fangzhou Liu, Zhongheng ZhangAbstract
The labor-intensive nature of medical data annotation presents a significant challenge for respiratory disease diagnosis, resulting in a scarcity of high-quality labeled datasets in resource-constrained settings. Moreover, patient privacy concerns complicate the direct sharing of local medical data across institutions, and existing centralized data-driven approaches, which rely on amounts of available data, often compromise data privacy. This study proposes a federated few-shot learning framework with privacy-preserving mechanisms to address the issues of limited labeled data and privacy protection in diagnosing respiratory diseases. In particular, a meta-stochastic gradient descent algorithm is proposed to mitigate the overfitting problem that arises from insufficient data when employing traditional gradient descent methods for neural network training. Furthermore, to ensure data privacy against gradient leakage, differential privacy noise from a standard Gaussian distribution is integrated into the gradients during the training of private models with local data, thereby preventing the reconstruction of medical images. Given the impracticality of centralizing respiratory disease data dispersed across various medical institutions, a weighted average algorithm is employed to aggregate local diagnostic models from different clients, enhancing the adaptability of a model across diverse scenarios. Experimental results show that the proposed method yields compelling results with the implementation of differential privacy, while effectively diagnosing respiratory diseases using data from different structures, categories, and distributions.
Abstract (translated)
医疗数据标注的劳动密集型特性对呼吸道疾病诊断构成了重大挑战，导致资源受限环境中高质量标记数据集的稀缺。此外，患者隐私问题使得难以直接跨机构共享本地医学数据，并且现有的集中式数据驱动方法依赖于大量可用的数据，在保护数据隐私方面往往存在妥协。本研究提出了一种带有隐私保护机制的联邦小样本学习框架，以解决在诊断呼吸道疾病时面临的有限标注数据和隐私保护的问题。具体来说，提出了一个元随机梯度下降算法来减轻由于缺乏数据而在使用传统梯度下降方法训练神经网络时出现的过拟合问题。此外，为了确保在本地数据上训练私有模型时对抗梯度泄露的数据隐私，在梯度中加入了来自标准高斯分布的差分隐私噪声，从而防止医学图像的重建。鉴于集中化分散于不同医疗机构中的呼吸道疾病数据不切实际，采用了加权平均算法来聚合来自不同客户端的局部诊断模型，以增强模型在各种场景下的适应性。实验结果表明，所提出的方法在实现差分隐私的情况下取得了令人信服的结果，并且能够有效地使用不同结构、类别和分布的数据进行呼吸道疾病的诊断。
URL
https://arxiv.org/abs/2507.08050
PDF
-
Q-STAC: Q-Guided Stein Variational Model Predictive Actor-Critic
2025-07-09 07:53:53Shizhe Cai, Jayadeep Jacob, Zeya Yin, Fabio RamosAbstract
Deep reinforcement learning has shown remarkable success in continuous control tasks, yet often requires extensive training data, struggles with complex, long-horizon planning, and fails to maintain safety constraints during operation. Meanwhile, Model Predictive Control (MPC) offers explainability and constraint satisfaction, but typically yields only locally optimal solutions and demands careful cost function design. This paper introduces the Q-guided STein variational model predictive Actor-Critic (Q-STAC), a novel framework that bridges these approaches by integrating Bayesian MPC with actor-critic reinforcement learning through constrained Stein Variational Gradient Descent (SVGD). Our method optimizes control sequences directly using learned Q-values as objectives, eliminating the need for explicit cost function design while leveraging known system dynamics to enhance sample efficiency and ensure control signals remain within safe boundaries. Extensive experiments on 2D navigation and robotic manipulation tasks demonstrate that Q-STAC achieves superior sample efficiency, robustness, and optimality compared to state-of-the-art algorithms, while maintaining the high expressiveness of policy distributions. Experiment videos are available on our website: this https URL
Abstract (translated)
深度强化学习在连续控制任务中表现出显著的成功，但仍需要大量的训练数据，并且在处理复杂、长期的规划问题以及确保操作过程中的安全约束时面临挑战。与此同时，模型预测控制（MPC）提供了可解释性和满足约束条件的能力，但通常只能提供局部最优解，并要求精心设计的成本函数。本文介绍了Q-guided STein变分模型预测Actor-Critic (Q-STAC)框架，这是一种新的方法，通过将贝叶斯MPC与基于演员-评论家的强化学习相结合来弥合这两种方法之间的差距，利用了约束Stein Variational Gradient Descent（SVGD）。 我们的方法直接使用已学得的状态动作价值函数（Q值）作为目标优化控制序列，从而消除了显式成本函数设计的需求，并且利用已知的系统动力学来提高样本效率并确保控制信号保持在安全范围内。在二维导航和机器人操作任务上的广泛实验表明，与最先进的算法相比，Q-STAC实现了更优的样本效率、鲁棒性和最优性，同时保持了策略分布的高度表达能力。 实验视频可在我们的网站上查看：[提供的链接]
URL
https://arxiv.org/abs/2507.06625
PDF
-
KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networks
2025-07-08 20:33:15James Hazelden, Laura Driscoll, Eli Shlizerman, Eric Shea-BrownAbstract
Gradient Descent (GD) and its variants are the primary tool for enabling efficient training of recurrent dynamical systems such as Recurrent Neural Networks (RNNs), Neural ODEs and Gated Recurrent units (GRUs). The dynamics that are formed in these models exhibit features such as neural collapse and emergence of latent representations that may support the remarkable generalization properties of networks. In neuroscience, qualitative features of these representations are used to compare learning in biological and artificial systems. Despite recent progress, there remains a need for theoretical tools to rigorously understand the mechanisms shaping learned representations, especially in finite, non-linear models. Here, we show that the gradient flow, which describes how the model's dynamics evolve over GD, can be decomposed into a product that involves two operators: a Parameter Operator, K, and a Linearized Flow Propagator, P. K mirrors the Neural Tangent Kernel in feed-forward neural networks, while P appears in Lyapunov stability and optimal control theory. We demonstrate two applications of our decomposition. First, we show how their interplay gives rise to low-dimensional latent dynamics under GD, and, specifically, how the collapse is a result of the network structure, over and above the nature of the underlying task. Second, for multi-task training, we show that the operators can be used to measure how objectives relevant to individual sub-tasks align. We experimentally and theoretically validate these findings, providing an efficient Pytorch package, \emph{KPFlow}, implementing robust analysis tools for general recurrent architectures. Taken together, our work moves towards building a next stage of understanding of GD learning in non-linear recurrent models.
Abstract (translated)
梯度下降（GD）及其变体是使递归动力系统（如循环神经网络（RNN）、神经ODE和门控循环单元（GRU））高效训练的主要工具。这些模型中形成的动态特性，例如神经坍缩和潜在表示的出现，可能支持网络显著的泛化能力。在神经科学领域，人们使用这些表示的定性特征来比较生物系统与人工系统的学习过程。尽管近年来取得了一些进展，但仍需要理论工具来严格理解塑造所学表示机制的过程，尤其是在有限且非线性的模型中。 本文展示了梯度流（描述GD过程中模型动态演变的方式）可以分解为一个涉及两个操作符的乘积：参数操作符K和线性化流动传播器P。其中，K在前馈神经网络中类似于神经切片核(Neural Tangent Kernel)，而P出现在李雅普诺夫稳定性(Lyapunov stability)和最优控制理论(optimal control theory)中。 我们展示了这种分解的两个应用：首先，展示了它们之间的交互如何在GD过程中产生低维潜在动态，并且具体地说明了坍缩是如何由网络结构决定的，而不仅仅是任务的本质。其次，在多任务训练的情况下，展示这些操作符可以用来衡量与各个子任务相关的目标的一致性。 我们通过实验和理论验证了这些发现，并提供了一个高效实用的Pytorch包*KPFlow*，用于实现对通用递归架构进行稳健分析的工具。总的来说，我们的工作朝着建立理解非线性递归模型中GD学习的新阶段迈进了一步。
URL
https://arxiv.org/abs/2507.06381
PDF
-
Simple Convergence Proof of Adam From a Sign-like Descent Perspective
2025-07-08 13:19:26Hanyang Peng, Shuang Qin, Yue Yu, Fangqing Jiang, Hui Wang, Zhouchen LinAbstract
Adam is widely recognized as one of the most effective optimizers for training deep neural networks (DNNs). Despite its remarkable empirical success, its theoretical convergence analysis remains unsatisfactory. Existing works predominantly interpret Adam as a preconditioned stochastic gradient descent with momentum (SGDM), formulated as $\bm{x}_{t+1} = \bm{x}_t - \frac{\gamma_t}{{\sqrt{\bm{v}_t}+\epsilon}} \circ \bm{m}_t$. This perspective necessitates strong assumptions and intricate techniques, resulting in lengthy and opaque convergence proofs that are difficult to verify and extend. In contrast, we propose a novel interpretation by treating Adam as a sign-like optimizer, expressed as $\bm{x}_{t+1} = \bm{x}_t - \gamma_t \frac{|\bm{m}_t|}{{\sqrt{\bm{v}_t}+\epsilon}} \circ {\rm Sign}(\bm{m}_t)$. This reformulation significantly simplifies the convergence analysis. For the first time, with some mild conditions, we prove that Adam achieves the optimal rate of ${\cal O}(\frac{1}{T^{\sfrac{1}{4}}})$ rather than the previous ${\cal O} \left(\frac{\ln T}{T^{\sfrac{1}{4}}}\right)$ under weak assumptions of the generalized $p$-affine variance and $(L_0, L_1, q)$-smoothness, without dependence on the model dimensionality or the numerical stability parameter $\epsilon$. Additionally, our theoretical analysis provides new insights into the role of momentum as a key factor ensuring convergence and offers practical guidelines for tuning learning rates in Adam, further bridging the gap between theory and practice.
Abstract (translated)
亚当（Adam）优化算法被广泛认为是训练深度神经网络（DNN）中最有效的优化器之一。尽管其实验结果令人印象深刻，但其理论收敛性分析仍然不够完善。现有研究主要将Adam解释为带有动量的预处理随机梯度下降（SGDM），公式化表达为 $\bm{x}_{t+1} = \bm{x}_t - \frac{\gamma_t}{\sqrt{\bm{v}_t}+\epsilon} \circ \bm{m}_t$。这种视角需要强假设和复杂的技术，导致收敛性证明冗长且难以验证和扩展。相比之下，我们提出了一种新的解释方法，即将Adam视为类似符号的优化器（sign-like optimizer），公式化表达为 $\bm{x}_{t+1} = \bm{x}_t - \gamma_t \frac{|\bm{m}_t|}{\sqrt{\bm{v}_t}+\epsilon} \circ {\rm Sign}(\bm{m}_t)$。这种重新表述显著简化了收敛性分析。 我们首次证明，在广义$p$仿射方差和$(L_0, L_1, q)$平滑性的弱假设下，且在不依赖于模型维度或数值稳定性参数$\epsilon$的情况下，Adam的收敛速度为${\cal O}(\frac{1}{T^{\sfrac{1}{4}}})$，而不是之前的${\cal O} \left(\frac{\ln T}{T^{\sfrac{1}{4}}}\right)$。此外，我们的理论分析还提供了关于动量作为确保收敛的关键因素的新见解，并为调整Adam中的学习率提供实用指导，进一步缩小了理论与实践之间的差距。
URL
https://arxiv.org/abs/2507.05966
PDF
-
Energy-Based Transformers are Scalable Learners and Thinkers
2025-07-02 19:17:29Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq IqbalAbstract
Inference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances. However, most existing approaches suffer from several limitations: they are modality-specific (e.g., working only in text), problem-specific (e.g., verifiable domains like math and coding), or require additional supervision/training on top of unsupervised pretraining (e.g., verifiers or verifiable rewards). In this paper, we ask the question "Is it possible to generalize these System 2 Thinking approaches, and develop models that learn to think solely from unsupervised learning?" Interestingly, we find the answer is yes, by learning to explicitly verify the compatibility between inputs and candidate-predictions, and then re-framing prediction problems as optimization with respect to this verifier. Specifically, we train Energy-Based Transformers (EBTs) -- a new class of Energy-Based Models (EBMs) -- to assign an energy value to every input and candidate-prediction pair, enabling predictions through gradient descent-based energy minimization until convergence. Across both discrete (text) and continuous (visual) modalities, we find EBTs scale faster than the dominant Transformer++ approach during training, achieving an up to 35% higher scaling rate with respect to data, batch size, parameters, FLOPs, and depth. During inference, EBTs improve performance with System 2 Thinking by 29% more than the Transformer++ on language tasks, and EBTs outperform Diffusion Transformers on image denoising while using fewer forward passes. Further, we find that EBTs achieve better results than existing models on most downstream tasks given the same or worse pretraining performance, suggesting that EBTs generalize better than existing approaches. Consequently, EBTs are a promising new paradigm for scaling both the learning and thinking capabilities of models.
Abstract (translated)
类似于人类的系统2思维，推理时的计算技术最近因其在提升模型性能方面的潜力而变得流行。然而，大多数现有的方法存在若干局限性：它们只适用于特定模式（例如仅限于文本），针对特定问题领域（如数学和编程验证域），或者需要在无监督预训练基础上额外的监督或训练（例如校验器或可验证奖励）。在这篇论文中，我们提出了一个问题：“是否有可能将这些系统2思维方法进行泛化，并开发出仅仅通过无监督学习就能学会思考的模型？”令人惊讶的是，我们发现答案是肯定的。具体而言，我们的研究发现通过学习明确地验证输入与候选预测之间的兼容性，并重新框架为基于校验器的优化问题，可以实现这一目标。也就是说，我们训练了一种新型的能量基础模型（EBMs），即能量基础变换器（EBTs）来给每个输入和候选预测对分配一个能量值，使通过基于梯度下降的最小化直至收敛来进行预测成为可能。 在离散（文本）和连续（视觉）模式上，我们发现与目前主导的方法Transformer++相比，在训练过程中，EBTs的学习速率更快。具体来说，在数据量、批次大小、参数数量、FLOPs以及模型深度等方面，EBTs的扩展速度比Transformer++快35%以上。在推理阶段，相对于Transformer++方法，EBTs在语言任务上的系统2思维性能提高了29%，并且在图像去噪上优于扩散变换器，同时使用了更少的前向传递次数。此外，我们还发现，在给定相同或较差预训练表现的情况下，EBTs在大多数下游任务中的表现比现有的模型更好，这表明EBTs具有更好的泛化能力。 因此，能量基础变换器（EBTs）为增强模型的学习和推理能力提供了一种有前景的新范式。
URL
https://arxiv.org/abs/2507.02092
PDF
-
Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
2025-07-02 17:25:26Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, Qing HeAbstract
Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful technique for aligning large language models (LLMs) with human preferences. However, effectively aligning LLMs with diverse human preferences remains a significant challenge, particularly when they are conflict. To address this issue, we frame human value alignment as a multi-objective optimization problem, aiming to maximize a set of potentially conflicting objectives. We introduce Gradient-Adaptive Policy Optimization (GAPO), a novel fine-tuning paradigm that employs multiple-gradient descent to align LLMs with diverse preference distributions. GAPO adaptively rescales the gradients for each objective to determine an update direction that optimally balances the trade-offs between objectives. Additionally, we introduce P-GAPO, which incorporates user preferences across different objectives and achieves Pareto solutions that better align with the user's specific needs. Our theoretical analysis demonstrates that GAPO converges towards a Pareto optimal solution for multiple objectives. Empirical results on Mistral-7B show that GAPO outperforms current state-of-the-art methods, achieving superior performance in both helpfulness and harmlessness.
Abstract (translated)
从人类反馈中的强化学习（RLHF）已成为将大型语言模型（LLM）与人类偏好对齐的强大技术。然而，有效地使LLM与多样化的甚至是冲突的人类偏好相一致仍是一个重大挑战。为了解决这个问题，我们将人类价值观的对齐问题构架成一个多目标优化问题，旨在最大化一组潜在冲突的目标。我们引入了一种新型微调范式——梯度自适应策略优化（GAPO），它利用多梯度下降来使LLM与多种偏好分布相一致。GAPO能够根据每个目标自适应地调整梯度的尺度，以确定一个最优平衡各目标间权衡的更新方向。此外，我们还引入了P-GAPO，该方法结合不同目标上的用户偏好，并实现帕累托解，更好地满足用户的特定需求。理论分析表明，GAPO可以朝着多目标问题的帕累托最优解收敛。在Mistral-7B模型上的实证结果显示，GAPO超越了当前最先进的方法，在有用性和无害性方面表现出色。
URL
https://arxiv.org/abs/2507.01915
PDF
-
Description of the Training Process of Neural Networks via Ergodic Theorem : Ghost nodes
Abstract
Recent studies have proposed interpreting the training process from an ergodic perspective. Building on this foundation we present a unified framework for understanding and accelerating the training of deep neural networks via stochastic gradient descent. By analyzing the geometric landscape of the objective function we introduce a practical diagnostic, the running estimate of the largest Lyapunov exponent, which provably distinguishes genuine convergence toward stable minimizers from mere statistical stabilization near saddle points. We then propose a ghost category extension for standard classifiers that adds auxiliary ghost output nodes so the model gains extra descent directions that open a lateral corridor around narrow loss barriers and enable the optimizer to bypass poor basins during the early training phase. We show that this extension strictly reduces approximation error and that after sufficient convergence the ghost dimensions collapse and the extended model's invariant law coincides with that of the original and there exists a path in the enlarged parameter space along which the total loss does not increase while the original loss decreases by an arbitrary margin. Taken together these results provide a principled architecture level intervention that accelerates early stage trainability while preserving asymptotic behavior.
Abstract (translated)
近期的研究提出从遍历性视角来解释训练过程。在此基础上，我们提出了一个统一的框架，通过随机梯度下降法理解并加速深度神经网络的训练。通过对目标函数几何景观的分析，我们引入了一个实用的诊断工具——运行中的最大Lyapunov指数估计值，该方法可以证明区分真正的向稳定极小值点收敛和仅仅在鞍点附近的统计稳定。 接着，我们提出了一种标准分类器的幽灵类别扩展方法，即添加辅助的幽灵输出节点。这样模型可以获得额外的下降方向，在狭窄损失障碍物周围打开横向通道，并使优化器能够在训练初期绕过不良盆地。 我们展示了这种扩展能严格减少近似误差，并且在充分收敛后，幽灵维度会坍缩，扩展后的模型不变性定律与原模型一致，同时在扩大的参数空间中存在一条路径，在这条路径上总损失不会增加而原始损失可以任意减小。 综上所述，这些结果提供了一种基于架构的干预方法，可以在保持渐近行为的同时加速早期训练阶段。
URL
https://arxiv.org/abs/2507.01003
PDF
-
Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regime
2025-06-30 17:58:30Yuqing Wang, Shangding GuAbstract
Data selection plays a crucial role in data-driven decision-making, including in large language models (LLMs), and is typically task-dependent. Properties such as data quality and diversity have been extensively studied and are known to enhance model performance. However, it remains unclear whether there exist other quantitative and general principles of data selection that can consistently improve performance, especially for complex tasks with limited prior knowledge. In this paper, we demonstrate that selecting more uniformly distributed data can improve training efficiency while enhancing performance. Specifically, we establish that more uniform (less biased) distribution leads to a larger minimum pairwise distance between data points, denoted by $h_{\min}$, and prove that a smaller $h_{\min}$ can slow down the training dynamics of gradient descent (GD). Moreover, we theoretically show that the approximation error of neural networks decreases as $h_{\min}$ increases. Our analysis introduces a convergence framework for GD beyond the Neural Tangent Kernel (NTK) regime, applicable to a broad class of architectures, including transformers, without requiring Lipschitz smoothness. This framework further provides theoretical justification for the use of residual connections and function compositions in deep neural architectures. In the end, we conduct comprehensive experiments for supervised fine-tuning across various settings, including different optimization strategies, model sizes, and training datasets. The results consistently demonstrate that selecting data by maximizing pairwise distance significantly accelerates training and achieves comparable or better performance in LLMs across diverse datasets. Code and Datasets are available at the link: this https URL.
Abstract (translated)
数据选择在基于数据的决策制定中扮演着关键角色，特别是在大型语言模型（LLMs）中，并且通常是任务相关的。诸如数据质量和多样性的属性已经被广泛研究并且被证明可以提高模型性能。然而，是否存在其他能够一致提升性能的数据选择定量和通用原则，尤其是对于那些先验知识有限的复杂任务来说，这一点尚不清楚。 在本文中，我们展示了选择更均匀分布的数据可以在提高训练效率的同时增强性能。具体而言，我们确立了数据点之间最小成对距离（记为 $h_{\min}$）随分布更加均匀而增加，并证明较小的 $h_{\min}$ 可以减缓梯度下降 (GD) 的训练动态。此外，理论上显示神经网络的近似误差随着 $h_{\min}$ 的增大而减少。 我们的分析引入了一个超越神经切片核（NTK）框架的收敛性理论框架，适用于包括变压器在内的广泛架构类别，并且不需要利普希茨平滑度。此框架进一步提供了残差连接和函数组合在深度神经网络结构中的使用依据。 最终，我们在各种设置下进行了全面的监督微调实验，包括不同的优化策略、模型大小和训练数据集。结果一致表明，在大型语言模型中，通过最大化成对距离来选择数据可以显著加速训练，并且在不同数据集上可获得相当或更好的性能。 代码和数据集可以在以下链接获取：[此URL]（请将 [此URL] 替换为实际的链接地址）。
URL
https://arxiv.org/abs/2506.24120
PDF
-
SPI-BoTER: Error Compensation for Industrial Robots via Sparse Attention Masking and Hybrid Loss with Spatial-Physical Information
2025-06-28 07:00:59Xuao Hou, Yongquan Jia, Shijin Zhang, Yuqiang WuAbstract
The widespread application of industrial robots in fields such as cutting and welding has imposed increasingly stringent requirements on the trajectory accuracy of end-effectors. However, current error compensation methods face several critical challenges, including overly simplified mechanism modeling, a lack of physical consistency in data-driven approaches, and substantial data requirements. These issues make it difficult to achieve both high accuracy and strong generalization simultaneously. To address these challenges, this paper proposes a Spatial-Physical Informed Attention Residual Network (SPI-BoTER). This method integrates the kinematic equations of the robotic manipulator with a Transformer architecture enhanced by sparse self-attention masks. A parameter-adaptive hybrid loss function incorporating spatial and physical information is employed to iteratively optimize the network during training, enabling high-precision error compensation under small-sample conditions. Additionally, inverse joint angle compensation is performed using a gradient descent-based optimization method. Experimental results on a small-sample dataset from a UR5 robotic arm (724 samples, with a train:test:validation split of 8:1:1) demonstrate the superior performance of the proposed method. It achieves a 3D absolute positioning error of 0.2515 mm with a standard deviation of 0.15 mm, representing a 35.16\% reduction in error compared to conventional deep neural network (DNN) methods. Furthermore, the inverse angle compensation algorithm converges to an accuracy of 0.01 mm within an average of 147 iterations. This study presents a solution that combines physical interpretability with data adaptability for high-precision control of industrial robots, offering promising potential for the reliable execution of precision tasks in intelligent manufacturing.
Abstract (translated)
工业机器人在切割和焊接等领域广泛应用，对末端执行器的轨迹精度提出了越来越严格的要求。然而，目前的误差补偿方法面临着几个关键挑战，包括过于简化的机制建模、数据驱动方法中物理一致性的缺乏以及大量数据的需求。这些问题使得同时实现高精度和强泛化能力变得困难。为了解决这些挑战，本文提出了一种基于空间-物理信息注意残差网络（SPI-BoTER）的方法。该方法将机器人操作臂的运动学方程与增强型稀疏自注意力掩码的Transformer架构相结合，并在训练过程中采用结合了空间和物理信息的参数自适应混合损失函数进行迭代优化，从而能够在小样本条件下实现高精度误差补偿。此外，还使用基于梯度下降的优化方法来进行反向关节角度补偿。 实验结果表明，在一个UR5机器人手臂的小样本数据集（724个样本，训练、测试和验证的比例为8:1:1）上，所提出的方法表现出优越性能。它实现了0.2515毫米的三维绝对定位误差，标准差为0.15毫米，相较于传统的深度神经网络方法，误差降低了35.16%。此外，反向角度补偿算法在平均147次迭代后收敛到0.01毫米精度。 这项研究提供了一个结合物理可解释性和数据适应性的解决方案，用于实现工业机器人的高精度控制，在智能制造中可靠地执行精密任务方面展现了巨大的潜力。
URL
https://arxiv.org/abs/2506.22788
PDF
