Gradient_Descent
-
NC-Reg : Neural Cortical Maps for Rigid Registration
2026-01-26 23:53:04Ines Vati, Pierrick Bourgeat, Rodrigo Santa Cruz, Vincent Dore, Olivier Salvado, Clinton Fookes, L\'eo LebratAbstract
We introduce neural cortical maps, a continuous and compact neural representation for cortical feature maps, as an alternative to traditional discrete structures such as grids and meshes. It can learn from meshes of arbitrary size and provide learnt features at any resolution. Neural cortical maps enable efficient optimization on the sphere and achieve runtimes up to 30 times faster than classic barycentric interpolation (for the same number of iterations). As a proof of concept, we investigate rigid registration of cortical surfaces and propose NC-Reg, a novel iterative algorithm that involves the use of neural cortical feature maps, gradient descent optimization and a simulated annealing strategy. Through ablation studies and subject-to-template experiments, our method demonstrates sub-degree accuracy ($<1^\circ$ from the global optimum), and serves as a promising robust pre-alignment strategy, which is critical in clinical settings.
Abstract (translated)
URL
https://arxiv.org/abs/2601.19042
PDF
-
Differentiable Architecture Search for Adversarially Robust Quantum Computer Vision
2026-01-26 01:21:44Mohamed Afane, Quanjiang Long, Haoting Shen, Ying Mao, Junaid Farooq, Ying Wang, Juntao ChenAbstract
Current quantum neural networks suffer from extreme sensitivity to both adversarial perturbations and hardware noise, creating a significant barrier to real-world deployment. Existing robustness techniques typically sacrifice clean accuracy or require prohibitive computational resources. We propose a hybrid quantum-classical Differentiable Quantum Architecture Search (DQAS) framework that addresses these limitations by jointly optimizing circuit structure and robustness through gradient-based methods. Our approach enhances traditional DQAS with a lightweight Classical Noise Layer applied before quantum processing, enabling simultaneous optimization of gate selection and noise parameters. This design preserves the quantum circuit's integrity while introducing trainable perturbations that enhance robustness without compromising standard performance. Experimental validation on MNIST, FashionMNIST, and CIFAR datasets shows consistent improvements in both clean and adversarial accuracy compared to existing quantum architecture search methods. Under various attack scenarios, including Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), Basic Iterative Method (BIM), and Momentum Iterative Method (MIM), and under realistic quantum noise conditions, our hybrid framework maintains superior performance. Testing on actual quantum hardware confirms the practical viability of discovered architectures. These results demonstrate that strategic classical preprocessing combined with differentiable quantum architecture optimization can significantly enhance quantum neural network robustness while maintaining computational efficiency.
Abstract (translated)
URL
https://arxiv.org/abs/2601.18058
PDF
-
Finite-Time Analysis of Gradient Descent for Shallow Transformers
2026-01-23 07:28:17Enes Arda, Semih Cayci, Atilla EryilmazAbstract
Understanding why Transformers perform so well remains challenging due to their non-convex optimization landscape. In this work, we analyze a shallow Transformer with $m$ independent heads trained by projected gradient descent in the kernel regime. Our analysis reveals two main findings: (i) the width required for nonasymptotic guarantees scales only logarithmically with the sample size $n$, and (ii) the optimization error is independent of the sequence length $T$. This contrasts sharply with recurrent architectures, where the optimization error can grow exponentially with $T$. The trade-off is memory: to keep the full context, the Transformer's memory requirement grows with the sequence length. We validate our theoretical results numerically in a teacher-student setting and confirm the predicted scaling laws for Transformers.
Abstract (translated)
URL
https://arxiv.org/abs/2601.16514
PDF
-
A Cosine Network for Image Super-Resolution
2026-01-23 02:58:57Chunwei Tian, Chengyuan Zhang, Bob Zhang, Zhiwu Li, C. L. Philip Chen, David ZhangAbstract
Deep convolutional neural networks can use hierarchical information to progressively extract structural information to recover high-quality images. However, preserving the effectiveness of the obtained structural information is important in image super-resolution. In this paper, we propose a cosine network for image super-resolution (CSRNet) by improving a network architecture and optimizing the training strategy. To extract complementary homologous structural information, odd and even heterogeneous blocks are designed to enlarge the architectural differences and improve the performance of image super-resolution. Combining linear and non-linear structural information can overcome the drawback of homologous information and enhance the robustness of the obtained structural information in image super-resolution. Taking into account the local minimum of gradient descent, a cosine annealing mechanism is used to optimize the training procedure by performing warm restarts and adjusting the learning rate. Experimental results illustrate that the proposed CSRNet is competitive with state-of-the-art methods in image super-resolution.
Abstract (translated)
URL
https://arxiv.org/abs/2601.16413
PDF
-
A Mobile Application for Flower Recognition System Based on Convolutional Neural Networks
2026-01-22 09:52:04Mustafa Yurdakul, Enes Ayan, Fahrettin Horasan, Sakir TasdemirAbstract
A convolutional neural network (CNN) is a deep learning algorithm that has been specifically designed for computer vision applications. The CNNs proved successful in handling the increasing amount of data in many computer vision problems, where classical machine learning algorithms were insufficient. Flowers have many uses in our daily lives, from decorating to making medicines to detoxifying the environment. Identifying flower types requires expert knowledge. However, accessing experts at any time and in any location may not always be feasible. In this study a mobile application based on CNNs was developed to recognize different types of flowers to provide non-specialists with quick and easy access to information about flower types. The study employed three distinct CNN models, namely MobileNet, DenseNet121, and Xception, to determine the most suitable model for the mobile application. The classification performances of the models were evaluated by training them with seven different optimization algorithms. The DenseNet-121 architecture, which uses the stochastic gradient descent (SGD) optimization algorithm, was the most successful, achieving 95.84 % accuracy, 96.00% precision, recall, and F1-score. This result shows that CNNs can be used for flower classification in mobile applications.
Abstract (translated)
卷积神经网络（CNN）是一种专门为计算机视觉应用设计的深度学习算法。在许多计算机视觉问题中，随着数据量的增长，传统的机器学习算法变得不够用，而CNN们成功地解决了这一挑战。花朵在我们的日常生活中有着多种用途，从装饰到制作药物再到净化环境。识别花种需要专业知识。然而，在任何时间和地点都能随时访问专家并不总是可行的。为此，本研究开发了一款基于CNN的移动应用，用于识别不同种类的花卉，为非专业人士提供快速便捷地获取有关花种的信息途径。该研究采用了三种不同的CNN模型，即MobileNet、DenseNet121和Xception，以确定最适合应用于移动端的模型。通过使用七种不同的优化算法对这些模型进行训练后对其分类性能进行了评估。结果表明，在使用随机梯度下降（SGD）优化算法的情况下，DenseNet-121架构表现最佳，达到了95.84%的准确率、96.00%的精确度、召回率和F1值。这一结果显示了CNN在移动应用中用于花卉分类的有效性。
URL
https://arxiv.org/abs/2601.15810
PDF
-
Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Data
2026-01-21 16:36:19Yuval Ran-Milo, Yotam Alexander, Shahar Mendel, Nadav CohenAbstract
Transformers trained via Reinforcement Learning (RL) with outcome-based supervision can spontaneously develop the ability to generate intermediate reasoning steps (Chain-of-Thought). Yet the mechanism by which sparse rewards drive gradient descent to discover such systematic reasoning remains poorly understood. We address this by analyzing the gradient flow dynamics of single-layer Transformers on a synthetic graph traversal task that cannot be solved without Chain-of-Thought (CoT) but admits a simple iterative solution. We prove that despite training solely on final-answer correctness, gradient flow drives the model to converge to a structured, interpretable algorithm that iteratively traverses the graph vertex-by-vertex. We characterize the distributional properties required for this emergence, identifying the critical role of "simple examples": instances requiring fewer reasoning steps. When the training distribution places sufficient mass on these simpler instances, the model learns a generalizable traversal strategy that extrapolates to longer chains; when this mass vanishes, gradient-based learning becomes infeasible. We corroborate our theoretical results through experiments on synthetic data and with real-world language models on mathematical reasoning tasks, validating that our theoretical findings carry over to practical settings.
Abstract (translated)
通过强化学习（RL）并基于结果进行监督训练的Transformers可以自发地发展生成中间推理步骤的能力（Chain-of-Thought）。然而，稀疏奖励如何驱动梯度下降以发现这种系统性推理机制仍然不清楚。我们通过分析单层Transformer在合成图遍历任务中的梯度流动动态来解决这个问题，该任务无法仅凭直觉解决但允许简单的迭代解决方案。 我们证明了尽管训练中只关注最终答案的准确性，梯度流动仍能驱动模型收敛到一种结构化且可解释的算法上，这种算法能够逐节点地遍历图。我们还描述了这一现象出现所需的分布特性，并确定“简单示例”的关键作用：那些需要较少推理步骤的例子。当训练数据集中足够多的是这些简单的例子时，模型会学习出一种泛化的遍历策略并能推广到更长的链路上；而如果这种简单的实例在数据集中的比例下降，基于梯度的学习就变得不可行。 我们通过合成数据上的实验以及对现实世界语言模型在数学推理任务上的研究来验证我们的理论结果，并且证实了这些发现能够应用于实际场景中。
URL
https://arxiv.org/abs/2601.15158
PDF
-
Differential Privacy Image Generation with Reconstruction Loss and Noise Injection Using an Error Feedback SGD
Abstract
Traditional data masking techniques such as anonymization cannot achieve the expected privacy protection while ensuring data utility for privacy-preserving machine learning. Synthetic data plays an increasingly important role as it generates a large number of training samples and prevents information leakage in real data. The existing methods suffer from the repeating trade-off processes between privacy and utility. We propose a novel framework for differential privacy generation, which employs an Error Feedback Stochastic Gradient Descent(EFSGD) method and introduces a reconstruction loss and noise injection mechanism into the training process. We generate images with higher quality and usability under the same privacy budget as the related work. Extensive experiments demonstrate the effectiveness and generalization of our proposed framework for both grayscale and RGB images. We achieve state-of-the-art results over almost all metrics on three benchmarks: MNIST, Fashion-MNIST, and CelebA.
Abstract (translated)
传统的数据屏蔽技术，如匿名化，在确保隐私保护的同时无法实现预期的数据效用，这对于隐私保护的机器学习来说是一个挑战。合成数据因其能够生成大量训练样本并防止真实数据的信息泄露而扮演着越来越重要的角色。现有的方法在隐私和实用性之间反复进行权衡过程。我们提出了一种新颖的微分隐私生成框架，该框架采用误差反馈随机梯度下降（EFSGD）方法，并在训练过程中引入了重建损失和噪声注入机制。与相关工作相比，在相同的隐私预算下，我们的框架能够生成质量更高且更具实用性的图像。 广泛的实验展示了我们提出的框架对于灰度图和RGB图像的有效性和泛化能力。我们在三个基准数据集（MNIST、Fashion-MNIST 和 CelebA）上实现了几乎所有的指标上的最新成果。
URL
https://arxiv.org/abs/2601.15061
PDF
-
Interpretable and Sparse Linear Attention with Decoupled Membership-Subspace Modeling via MCR2 Objective
2026-01-20 10:11:40Tianyuan Liu, Libin Hou, Linyuan Wang, Bin YanAbstract
Maximal Coding Rate Reduction (MCR2)-driven white-box transformer, grounded in structured representation learning, unifies interpretability and efficiency, providing a reliable white-box solution for visual modeling. However, in existing designs, tight coupling between "membership matrix" and "subspace matrix U" in MCR2 causes redundant coding under incorrect token projection. To this end, we decouple the functional relationship between the "membership matrix" and "subspaces U" in the MCR2 objective and derive an interpretable sparse linear attention operator from unrolled gradient descent of the optimized objective. Specifically, we propose to directly learn the membership matrix from inputs and subsequently derive sparse subspaces from the fullspace S. Consequently, gradient unrolling of the optimized MCR2 objective yields an interpretable sparse linear attention operator: Decoupled Membership-Subspace Attention (DMSA). Experimental results on visual tasks show that simply replacing the attention module in Token Statistics Transformer (ToST) with DMSA (we refer to as DMST) not only achieves a faster coding reduction rate but also outperforms ToST by 1.08%-1.45% in top-1 accuracy on the ImageNet-1K dataset. Compared with vanilla Transformer architectures, DMST exhibits significantly higher computational efficiency and interpretability.
Abstract (translated)
URL
https://arxiv.org/abs/2601.17042
PDF
-
Performance and Complexity Trade-off Optimization of Speech Models During Training
2026-01-20 08:00:05Esteban G\'omez, Tom B\"ackstr\"omAbstract
In speech machine learning, neural network models are typically designed by choosing an architecture with fixed layer sizes and structure. These models are then trained to maximize performance on metrics aligned with the task's objective. While the overall architecture is usually guided by prior knowledge of the task, the sizes of individual layers are often chosen heuristically. However, this approach does not guarantee an optimal trade-off between performance and computational complexity; consequently, post hoc methods such as weight quantization or model pruning are typically employed to reduce computational cost. This occurs because stochastic gradient descent (SGD) methods can only optimize differentiable functions, while factors influencing computational complexity, such as layer sizes and floating-point operations per second (FLOP/s), are non-differentiable and require modifying the model structure during training. We propose a reparameterization technique based on feature noise injection that enables joint optimization of performance and computational complexity during training using SGD-based methods. Unlike traditional pruning methods, our approach allows the model size to be dynamically optimized for a target performance-complexity trade-off, without relying on heuristic criteria to select which weights or structures to remove. We demonstrate the effectiveness of our method through three case studies, including a synthetic example and two practical real-world applications: voice activity detection and audio anti-spoofing. The code related to our work is publicly available to encourage further research.
Abstract (translated)
在语音机器学习中，神经网络模型通常是通过选择具有固定层数和结构的架构来设计的。这些模型经过训练以最大化与任务目标一致的指标性能。虽然整体架构通常由任务的先验知识指导，但各个层的大小往往是由直觉或经验法则确定的。然而，这种方法不能保证在性能和计算复杂性之间达到最优权衡；因此，人们常常采用事后方法（如权重量化或模型剪枝）来降低计算成本。这是因为随机梯度下降(SGD)方法只能优化可微函数，而影响计算复杂性的因素（如层数和每秒浮点运算数(FLOP/s)）是不可微的，并且需要在训练过程中修改模型结构。 我们提出了一种基于特征噪声注入的重新参数化技术，该技术能够使用SGD基方法在训练期间同时优化性能和计算复杂性。与传统的剪枝方法不同，我们的方法允许动态调整模型大小以适应目标性能-复杂性的权衡，而无需依赖于选择要移除哪些权重或结构的经验准则。 我们通过三个案例研究展示了这种方法的有效性，包括一个合成示例以及两个实际应用：语音活动检测和音频防欺骗。与我们的工作相关的代码是公开的，旨在促进进一步的研究。
URL
https://arxiv.org/abs/2601.13704
PDF
-
TextBFGS: Quasi-Newton Optimization for Discrete Executable Text via Gradient-Operator Retrieval
2026-01-20 05:45:56Zizheng Zhang, Yuyang Liao, Chen Chen, Jian He, Dun Wu, Qianjin Yu, Yanqin Gao, Jin Yang, Kailai Zhang, Eng Siong Chng, Xionghu ZhongAbstract
Optimizing discrete executable text such as prompts and code has recently been framed as a gradient-based process, effectively translating backpropagation concepts to the semantic space. However, existing methods predominantly operate as first-order optimizers akin to Stochastic Gradient Descent, which are suffering from slow convergence and instability because they neglect the semantic curvature of the optimization landscape. To bridge this gap, we introduce TextBFGS, a second-order framework to implement a Quasi-Newton optimization method for discrete text. Unlike traditional memory-based approaches that retrieve similar textual instances, TextBFGS approximates the inverse Hessian matrix by retrieving Gradient-Operators from the memory of pre-learned successful trajectories. Specifically, given a textual gradient feedback, TextBFGS identifies historical correction patterns from the optimization knowledge base and tries to apply these abstract operators to the current variable. This mechanism enables a One-Pass Update, combining feedback generation and second-order correction into a single inference step. Empirical evaluations on code optimization across diverse domains (e.g., HumanEval, MBPP) demonstrate that TextBFGS significantly outperforms first-order baselines. It achieves superior pass rates with fewer model calls and exhibits strong cross-task transferability, thus establishes a mathematically grounded paradigm for efficient, memory-aware text optimization.
Abstract (translated)
URL
https://arxiv.org/abs/2602.00059
PDF
-
Preconditioning Benefits of Spectral Orthogonalization in Muon
2026-01-20 00:08:31Jianhao Ma, Yu Huang, Yuejie Chi, Yuxin ChenAbstract
The Muon optimizer, a matrix-structured algorithm that leverages spectral orthogonalization of gradients, is a milestone in the pretraining of large language models. However, the underlying mechanisms of Muon -- particularly the role of gradient orthogonalization -- remain poorly understood, with very few works providing end-to-end analyses that rigorously explain its advantages in concrete applications. We take a step by studying the effectiveness of a simplified variant of Muon through two case studies: matrix factorization, and in-context learning of linear transformers. For both problems, we prove that simplified Muon converges linearly with iteration complexities independent of the relevant condition number, provably outperforming gradient descent and Adam. Our analysis reveals that the Muon dynamics decouple into a collection of independent scalar sequences in the spectral domain, each exhibiting similar convergence behavior. Our theory formalizes the preconditioning effect induced by spectral orthogonalization, offering insight into Muon's effectiveness in these matrix optimization problems and potentially beyond.
Abstract (translated)
穆安（Muon）优化器是一种基于矩阵结构的算法，利用梯度谱正交化技术，在大规模语言模型预训练中取得了重要进展。然而，关于穆安的具体工作机制——特别是梯度正交化的角色——仍然知之甚少，鲜有研究能提供严格的端到端分析来详细解释其在实际应用中的优势。 为了更好地理解这一点，我们通过两个案例研究简化版的穆安优化器的有效性：矩阵分解和线性变换器的上下文学习。对于这两个问题，我们证明了简化的穆安算法能够以与相关条件数无关的迭代复杂度进行线性收敛，并且在理论上可以优于梯度下降法和Adam优化器。 我们的分析揭示出，当进入频谱域时，穆安动力学会分解为一系列独立的标量序列，每个序列都表现出相似的收敛行为。这一理论形式化了由谱正交化引起的预处理效果，不仅解释了在矩阵优化问题中穆安的有效性，还可能提供对其它领域的见解和应用潜力。
URL
https://arxiv.org/abs/2601.13474
PDF
-
Online Continual Learning for Time Series: a Natural Score-driven Approach
2026-01-19 10:31:01Edoardo Urettini, Daniele Atzeni, Ioanna-Yvonni Tsaknaki, Antonio CartaAbstract
Online continual learning (OCL) methods adapt to changing environments without forgetting past knowledge. Similarly, online time series forecasting (OTSF) is a real-world problem where data evolve in time and success depends on both rapid adaptation and long-term memory. Indeed, time-varying and regime-switching forecasting models have been extensively studied, offering a strong justification for the use of OCL in these settings. Building on recent work that applies OCL to OTSF, this paper aims to strengthen the theoretical and practical connections between time series methods and OCL. First, we reframe neural network optimization as a parameter filtering problem, showing that natural gradient descent is a score-driven method and proving its information-theoretic optimality. Then, we show that using a Student's t likelihood in addition to natural gradient induces a bounded update, which improves robustness to outliers. Finally, we introduce Natural Score-driven Replay (NatSR), which combines our robust optimizer with a replay buffer and a dynamic scale heuristic that improves fast adaptation at regime drifts. Empirical results demonstrate that NatSR achieves stronger forecasting performance than more complex state-of-the-art methods.
Abstract (translated)
在线连续学习（OCL）方法能够在不忘记过去知识的情况下适应环境的变化。同样，实时序列预测（OTSF）是现实世界中的一个问题，在这个问题中数据会随时间演变，成功取决于快速适应和长期记忆的能力。实际上，时变和制度转换的预测模型已经得到了广泛的研究，这为在这些场景下使用OCL提供了强大的理论依据。基于最近将OCL应用于OTSF的工作，本文旨在加强时间序列方法与OCL之间的理论和实践联系。 首先，我们将神经网络优化重新定义为参数过滤问题，并展示了自然梯度下降是一种评分驱动的方法，并证明了其信息论上的最优性。然后，我们表明，在使用自然梯度的同时采用Student's t似然函数可以实现有界更新，从而提高对离群值的鲁棒性。最后，我们提出了自然评分驱动重放（NatSR），该方法结合了我们的鲁棒优化器、回放缓存以及动态尺度启发式策略，以在制度漂移期间改善快速适应能力。 实验证据表明，NatSR在预测性能方面优于更复杂的最新方法。
URL
https://arxiv.org/abs/2601.12931
PDF
-
Mining Citywide Dengue Spread Patterns in Singapore Through Hotspot Dynamics from Open Web Data
2026-01-19 09:10:50Liping Huang, Gaoxi Xiao, Stefan Ma, Hechang Chen, Shisong Tang, Flora SalimAbstract
Dengue, a mosquito-borne disease, continues to pose a persistent public health challenge in urban areas, particularly in tropical regions such as Singapore. Effective and affordable control requires anticipating where transmission risks are likely to emerge so that interventions can be deployed proactively rather than reactively. This study introduces a novel framework that uncovers and exploits latent transmission links between urban regions, mined directly from publicly available dengue case data. Instead of treating cases as isolated reports, we model how hotspot formation in one area is influenced by epidemic dynamics in neighboring regions. While mosquito movement is highly localized, long-distance transmission is often driven by human mobility, and in our case study, the learned network aligns closely with commuting flows, providing an interpretable explanation for citywide spread. These hidden links are optimized through gradient descent and used not only to forecast hotspot status but also to verify the consistency of spreading patterns, by examining the stability of the inferred network across consecutive weeks. Case studies on Singapore during 2013-2018 and 2020 show that four weeks of hotspot history are sufficient to achieve an average F-score of 0.79. Importantly, the learned transmission links align with commuting flows, highlighting the interpretable interplay between hidden epidemic spread and human mobility. By shifting from simply reporting dengue cases to mining and validating hidden spreading dynamics, this work transforms open web-based case data into a predictive and explanatory resource. The proposed framework advances epidemic modeling while providing a scalable, low-cost tool for public health planning, early intervention, and urban resilience.
Abstract (translated)
登革热，一种由蚊子传播的疾病，在城市地区尤其是新加坡这样的热带区域持续构成一个持久的公共卫生挑战。有效的控制措施需要预测病原体在不同地区的传播风险，以便主动部署干预手段而不是被动应对。本研究提出了一种新颖框架，通过挖掘公开可用的登革热病例数据来发现并利用城市地区之间的潜在传播联系。不同于将个案视为孤立报告，我们的模型探讨了在一个区域内热点形成如何受邻近区域疫情动态的影响。虽然蚊子活动范围有限，但远距离传播往往由人类移动驱动，在我们案例研究中，学习到的网络与通勤流动高度一致，为城市范围内疾病的扩散提供了一个解释性的说明。 这些隐含联系通过梯度下降方法进行优化，并不仅用于预测热点状态，还通过检查连续几周内推断出的网络传播模式稳定性来验证传播模式的一致性。2013年至2018年以及2020年的新加坡案例研究显示，只需利用四周的历史数据便足以达到平均F值（即准确率和召回率的调和平均数）为0.79。尤为重要的是，所学习到的传播联系与通勤流动相吻合，强调了隐含流行病扩散和人类移动之间的可解释互动。 通过从单纯报告登革热病例转变为挖掘并验证隐藏的传播动态，这项工作将开放式的网络案例数据转化为预测性和说明性的资源。提出的框架不仅推进了疫情建模，还为公共卫生规划、早期干预和城市韧性提供了一种规模大且成本低的工具。
URL
https://arxiv.org/abs/2601.12856
PDF
-
Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning
2026-01-19 08:23:12Ishir Garg, Neel Kolhe, Andy Peng, Rohan GopalamAbstract
Continual learning aims to enable neural networks to acquire new knowledge on sequential tasks. However, the key challenge in such settings is to learn new tasks without catastrophically forgetting previously learned tasks. We propose the Fisher-Orthogonal Projected Natural Gradient Descent (FOPNG) optimizer, which enforces Fisher-orthogonal constraints on parameter updates to preserve old task performance while learning new tasks. Unlike existing methods that operate in Euclidean parameter space, FOPNG projects gradients onto the Fisher-orthogonal complement of previous task gradients. This approach unifies natural gradient descent with orthogonal gradient methods within an information-geometric framework. The resulting update direction is invariant under reparameterization, guarantees descent in the Fisher metric, and helps preserve prior task outputs. We provide theoretical analysis establishing the properties of the projected update, describe efficient and practical implementations using the diagonal Fisher, and demonstrate strong results on standard continual learning benchmarks such as Permuted-MNIST, Split-MNIST, Rotated-MNIST, Split-CIFAR10, and Split-CIFAR100.
Abstract (translated)
持续学习旨在使神经网络能够在顺序任务中获取新知识。然而，在这种设置下的关键挑战是学习新任务的同时不灾难性地忘记以前学过的任务。我们提出了Fisher-正交投影自然梯度下降（FOPNG）优化器，它在参数更新时施加了Fisher-正交约束，以保持旧任务的性能同时学习新任务。与现有方法在欧几里得参数空间中操作不同，FOPNG将梯度投影到以前任务梯度的Fisher-正交补集中。这种方法统一了自然梯度下降和正交梯度方法在一个信息几何框架内。由此产生的更新方向对于重参数化是不变的，在Fisher度量下保证了下降，并有助于保持先前任务的输出。 我们提供了关于投影更新性质的理论分析，描述了使用对角Fisher的有效且实用实现，并在Permuted-MNIST、Split-MNIST、Rotated-MNIST、Split-CIFAR10和Split-CIFAR100等标准持续学习基准测试中展示了强大的结果。
URL
https://arxiv.org/abs/2601.12816
PDF
-
Contextual Distributionally Robust Optimization with Causal and Continuous Structure: An Interpretable and Tractable Approach
2026-01-16 06:18:22Fenglin Zhang, Jie WangAbstract
In this paper, we introduce a framework for contextual distributionally robust optimization (DRO) that considers the causal and continuous structure of the underlying distribution by developing interpretable and tractable decision rules that prescribe decisions using covariates. We first introduce the causal Sinkhorn discrepancy (CSD), an entropy-regularized causal Wasserstein distance that encourages continuous transport plans while preserving the causal consistency. We then formulate a contextual DRO model with a CSD-based ambiguity set, termed Causal Sinkhorn DRO (Causal-SDRO), and derive its strong dual reformulation where the worst-case distribution is characterized as a mixture of Gibbs distributions. To solve the corresponding infinite-dimensional policy optimization, we propose the Soft Regression Forest (SRF) decision rule, which approximates optimal policies within arbitrary measurable function spaces. The SRF preserves the interpretability of classical decision trees while being fully parametric, differentiable, and Lipschitz smooth, enabling intrinsic interpretation from both global and local perspectives. To solve the Causal-SDRO with parametric decision rules, we develop an efficient stochastic compositional gradient algorithm that converges to an $\varepsilon$-stationary point at a rate of $O(\varepsilon^{-4})$, matching the convergence rate of standard stochastic gradient descent. Finally, we validate our method through numerical experiments on synthetic and real-world datasets, demonstrating its superior performance and interpretability.
Abstract (translated)
在这篇论文中，我们提出了一种框架用于上下文分布鲁棒优化（DRO），该框架通过开发可解释且易于处理的决策规则来考虑潜在分布的因果和连续结构。这些决策规则使用协变量来制定决定。 首先，我们引入了因果Sinkhorn偏差（CSD），这是一种带熵正则化的因果Wasserstein距离，它鼓励连续的运输计划同时保持因果一致性。然后，我们提出了一种基于CSD不确定性集的上下文DRO模型，并将其命名为因果Sinkhorn DRO（Causal-SDRO）。我们推导了其强对偶重构形式，在其中最坏情况下的分布被描述为吉布斯分布的混合体。 为了求解相应的无限维策略优化问题，我们提出了软回归森林（SRF）决策规则。该规则可以在任意可测量的功能空间中逼近最优政策。与经典的决策树相比，SRF保持了其解释性的同时，也具有完全参数化、可微和Lipschitz光滑的特点，从而可以实现全局和局部视角的内在解释。 为了解决带有参数决策规则的Causal-SDRO问题，我们开发了一种高效的随机合成梯度算法。该算法以$O(\varepsilon^{-4})$的速度收敛到$\varepsilon$次平稳点，其速度与标准随机梯度下降一致。 最后，通过在合成和实际数据集上的数值实验验证了我们的方法的有效性和解释性，展示了它的优越性能。
URL
https://arxiv.org/abs/2601.11016
PDF
-
Algorithmic Stability in Infinite Dimensions: Characterizing Unconditional Convergence in Banach Spaces
Abstract
The distinction between conditional, unconditional, and absolute convergence in infinite-dimensional spaces has fundamental implications for computational algorithms. While these concepts coincide in finite dimensions, the Dvoretzky-Rogers theorem establishes their strict separation in general Banach spaces. We present a comprehensive characterization theorem unifying seven equivalent conditions for unconditional convergence: permutation invariance, net convergence, subseries tests, sign stability, bounded multiplier properties, and weak uniform convergence. These theoretical results directly inform algorithmic stability analysis, governing permutation invariance in gradient accumulation for Stochastic Gradient Descent and justifying coefficient thresholding in frame-based signal processing. Our work bridges classical functional analysis with contemporary computational practice, providing rigorous foundations for order-independent and numerically robust summation processes.
Abstract (translated)
在无限维空间中，条件收敛、无条件收敛和绝对收敛之间的区别对计算算法具有根本性的影响。尽管这些概念在有限维度下是重合的，但Dvoretzky-Rogers定理却证明了它们在一般的Banach空间中存在严格的分离。我们提出了一项全面的特征定理，该定理统一了无条件收敛的七个等价条件：置换不变性、网收敛性、子级数测试、符号稳定性、有界乘法器性质以及弱一致收敛性。这些理论结果直接指导算法稳定性的分析，在随机梯度下降中的梯度累积过程中的置换不变性和基于框架的信号处理中系数阈值的选择方面提供了依据。 我们的工作将经典的功能分析与当代计算实践联系起来，为顺序无关和数值稳健的求和过程提供严格的理论基础。
URL
https://arxiv.org/abs/2601.08512
PDF
-
Demystifying the Slash Pattern in Attention: The Role of RoPE
2026-01-13 07:40:57Yuan Cheng, Fengzhuo Zhang, Yunlong Hou, Cunxiao Du, Chao Du, Tianyu Pang, Aixin Sun, Zhuoran YangAbstract
Large Language Models (LLMs) often exhibit slash attention patterns, where attention scores concentrate along the $\Delta$-th sub-diagonal for some offset $\Delta$. These patterns play a key role in passing information across tokens. But why do they emerge? In this paper, we demystify the emergence of these Slash-Dominant Heads (SDHs) from both empirical and theoretical perspectives. First, by analyzing open-source LLMs, we find that SDHs are intrinsic to models and generalize to out-of-distribution prompts. To explain the intrinsic emergence, we analyze the queries, keys, and Rotary Position Embedding (RoPE), which jointly determine attention scores. Our empirical analysis reveals two characteristic conditions of SDHs: (1) Queries and keys are almost rank-one, and (2) RoPE is dominated by medium- and high-frequency components. Under these conditions, queries and keys are nearly identical across tokens, and interactions between medium- and high-frequency components of RoPE give rise to SDHs. Beyond empirical evidence, we theoretically show that these conditions are sufficient to ensure the emergence of SDHs by formalizing them as our modeling assumptions. Particularly, we analyze the training dynamics of a shallow Transformer equipped with RoPE under these conditions, and prove that models trained via gradient descent exhibit SDHs. The SDHs generalize to out-of-distribution prompts.
Abstract (translated)
大型语言模型（LLMs）常常表现出斜向注意力模式，其中注意分数集中在第$\Delta$个副对角线上。这种模式在跨令牌传递信息中起着关键作用。但是为什么会出现这种情况呢？在这篇论文中，我们从实证和理论两个角度解开了这些斜向主导头(Slash-Dominant Heads, SDHs)出现的神秘面纱。 首先，通过对开源LLMs进行分析，我们发现SDHs是模型固有的，并且能够推广到分布外提示。为了解释这种内在出现的现象，我们分析了查询、键和旋转位置嵌入（RoPE），它们共同决定了注意分数。我们的实证分析揭示了SDH的两个特征条件：(1) 查询和键几乎是秩一的；(2) RoPE主要由中频和高频成分主导。 在这些条件下，查询和键几乎在各个令牌之间是相同的，并且RoPE中的中频和高频部分之间的相互作用会导致SDHs。除了实证证据之外，我们还从理论上证明了在给定条件下的训练过程中会出现SDHs。具体来说，我们在浅层Transformer装备有RoPE的情况下分析这些条件下的训练动态，并证明通过梯度下降训练的模型会表现出SDHs。 SDH能够推广到分布外提示中。
URL
https://arxiv.org/abs/2601.08297
PDF
-
Spiking Neural-Invariant Kalman Fusion for Accurate Localization Using Low-Cost IMUs
2026-01-13 06:12:12Yaohua Liu, Qiao Xu, Yemin Wang, Hui Yi Leong, Binkai OuAbstract
Low-cost inertial measurement units (IMUs) are widely utilized in mobile robot localization due to their affordability and ease of integration. However, their complex, nonlinear, and time-varying noise characteristics often lead to significant degradation in localization accuracy when applied directly for dead reckoning. To overcome this limitation, we propose a novel brain-inspired state estimation framework that combines a spiking neural network (SNN) with an invariant extended Kalman filter (InEKF). The SNN is designed to extract motion-related features from long sequences of IMU data affected by substantial random noise and is trained via a surrogate gradient descent algorithm to enable dynamic adaptation of the covariance noise parameter within the InEKF. By fusing the SNN output with raw IMU measurements, the proposed method enhances the robustness and accuracy of pose estimation. Extensive experiments conducted on the KITTI dataset and real-world data collected using a mobile robot equipped with a low-cost IMU demonstrate that the proposed approach outperforms state-of-the-art methods in localization accuracy and exhibits strong robustness to sensor noise, highlighting its potential for real-world mobile robot applications.
Abstract (translated)
低成本惯性测量单元（IMUs）由于其经济性和易于集成的特性，在移动机器人定位中被广泛应用。然而，当直接用于航位推算时，这些设备复杂的、非线性的和随时间变化的噪声特征常常会导致定位精度显著下降。为克服这一限制，我们提出了一种新型仿脑状态估计框架，该框架结合了脉冲神经网络（SNN）与不变扩展卡尔曼滤波器（InEKF）。SNN被设计用于从受大量随机噪声影响的长时间IMU数据序列中提取运动相关特征，并通过代理梯度下降算法进行训练，以使InEKF中的协方差噪声参数能够动态调整。通过融合SNN输出与原始IMU测量值，所提出的方法增强了姿态估计的鲁棒性和准确性。 在KITTI数据集和使用低成本IMU装备的移动机器人采集的真实世界数据上进行了广泛的实验表明，该方法在定位精度方面优于现有的先进方法，并且对传感器噪声表现出强大的鲁棒性。这一结果凸显了其在现实世界的移动机器人应用中的潜力。
URL
https://arxiv.org/abs/2601.08248
PDF
-
BlindU: Blind Machine Unlearning without Revealing Erasing Data
2026-01-12 05:09:09Weiqi Wang, Zhiyi Tian, Chenhan Zhang, Shui YuAbstract
Machine unlearning enables data holders to remove the contribution of their specified samples from trained models to protect their privacy. However, it is paradoxical that most unlearning methods require the unlearning requesters to firstly upload their data to the server as a prerequisite for unlearning. These methods are infeasible in many privacy-preserving scenarios where servers are prohibited from accessing users' data, such as federated learning (FL). In this paper, we explore how to implement unlearning under the condition of not uncovering the erasing data to the server. We propose \textbf{Blind Unlearning (BlindU)}, which carries out unlearning using compressed representations instead of original inputs. BlindU only involves the server and the unlearning user: the user locally generates privacy-preserving representations, and the server performs unlearning solely on these representations and their labels. For the FL model training, we employ the information bottleneck (IB) mechanism. The encoder of the IB-based FL model learns representations that distort maximum task-irrelevant information from inputs, allowing FL users to generate compressed representations locally. For effective unlearning using compressed representation, BlindU integrates two dedicated unlearning modules tailored explicitly for IB-based models and uses a multiple gradient descent algorithm to balance forgetting and utility retaining. While IB compression already provides protection for task-irrelevant information of inputs, to further enhance the privacy protection, we introduce a noise-free differential privacy (DP) masking method to deal with the raw erasing data before compressing. Theoretical analysis and extensive experimental results illustrate the superiority of BlindU in privacy protection and unlearning effectiveness compared with the best existing privacy-preserving unlearning benchmarks.
Abstract (translated)
机器遗忘技术允许数据持有者从训练好的模型中移除特定样本的贡献，以保护隐私。然而，大多数遗忘方法要求发起请求的用户首先将他们的数据上传到服务器，这在许多需要保护隐私的情景下是不可行的，比如联邦学习（FL），因为在这些情况下，服务器被禁止访问用户的原始数据。 本文探讨了如何实现在不向服务器揭示要删除的数据的情况下进行机器遗忘。我们提出了**盲遗忘 (BlindU)** 方法，该方法使用压缩表示而非原始输入来进行遗忘操作。在 BlindU 中，只有服务端和发起请求的用户参与：用户在当地生成隐私保护表示，并且服务端仅基于这些表示及其标签执行遗忘操作。 对于联邦学习模型训练，我们采用了信息瓶颈（IB）机制。基于 IB 的联邦学习模型中的编码器从输入中提取最大任务无关的信息失真表示，使得 FL 用户能够在本地生成压缩表示。为了有效使用压缩表示进行遗忘，BlindU 集成了两个专门针对 IB 模型设计的遗忘模块，并采用多梯度下降算法来平衡遗忘和保持效用的能力。 虽然信息瓶颈压缩已经为输入的任务无关信息提供了保护，但为了进一步增强隐私保护，我们引入了一种无噪声差分隐私 (DP) 掩码方法，在压缩之前处理原始删除数据。理论分析和广泛的实验结果表明，BlindU 在隐私保护和遗忘有效性方面优于现有的最佳隐私保护遗忘基准。 总结来说，BlindU 为在不向服务器揭示敏感数据的情况下实现机器遗忘提供了一种新颖的方法，并通过使用信息瓶颈机制、专用的遗忘模块和差分隐私掩码方法提高了隐私保护和模型性能。
URL
https://arxiv.org/abs/2601.07214
PDF
-
The Normalized Difference Layer: A Differentiable Spectral Index Formulation for Deep Learning
2026-01-11 05:03:01Ali Lotfi, Adam Carter, Mohammad Meysami, Thuan Ha, Kwabena Nketia, Steve ShirtliffeAbstract
Normalized difference indices have been a staple in remote sensing for decades. They stay reliable under lighting changes produce bounded values and connect well to biophysical signals. Even so, they are usually treated as a fixed pre processing step with coefficients set to one, which limits how well they can adapt to a specific learning task. In this study, we introduce the Normalized Difference Layer that is a differentiable neural network module. The proposed method keeps the classical idea but learns the band coefficients from data. We present a complete mathematical framework for integrating this layer into deep learning architectures that uses softplus reparameterization to ensure positive coefficients and bounded denominators. We describe forward and backward pass algorithms enabling end to end training through backpropagation. This approach preserves the key benefits of normalized differences, namely illumination invariance and outputs bounded to $[-1,1]$ while allowing gradient descent to discover task specific band weightings. We extend the method to work with signed inputs, so the layer can be stacked inside larger architectures. Experiments show that models using this layer reach similar classification accuracy to standard multilayer perceptrons while using about 75\% fewer parameters. They also handle multiplicative noise well, at 10\% noise accuracy drops only 0.17\% versus 3.03\% for baseline MLPs. The learned coefficient patterns stay consistent across different depths.
Abstract (translated)
几十年来，归一化差异指数（Normalized Difference Indices）一直是遥感中的一个重要工具。它们在光照变化下依然可靠，并产生有界值和与生物物理信号的良好连接性。尽管如此，这些指标通常被视为固定的预处理步骤，其中系数被固定为1，这限制了它们适应特定学习任务的能力。在这项研究中，我们引入了一个可微的神经网络模块——归一化差异层（Normalized Difference Layer）。该方法保留了经典思想，但允许从数据中学习波段权重。 提出了一种完整的数学框架，用于将这个层整合到深度学习架构中，并使用softplus重新参数化来确保正系数和有界的分母。描述了前向和反向传递算法，使通过反向传播进行端到端训练成为可能。这种方法保留了归一化差异的关键优势，即光照不变性和输出限制在[-1,1]范围内，同时允许梯度下降发现任务特定的波段权重。还扩展了该方法以适应有符号输入的情况，使得这个层可以嵌入更大的架构中。 实验表明，使用此层的模型达到了与标准多层感知器（MLPs）相似的分类准确率，但使用的参数减少了约75%。它们在处理乘法噪声时表现良好，在10%的噪声情况下，准确性仅下降了0.17%，而基线MLPs则下降了3.03%。学习到的系数模式在不同深度下保持一致。 总结来说，归一化差异层结合了经典方法的优势和现代机器学习技术的能力，提供了一种新颖且有效的遥感图像处理方式。
URL
https://arxiv.org/abs/2601.06777
PDF
