Gradient_Descent
-
SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization
2026-02-08 11:12:45Taolin Zhang, Hang Guo, Wang Lu, Tao Dai, Shu-Tao Xia, Jindong WangAbstract
As large language models (LLMs) continue to scale up, their performance on various downstream tasks has significantly improved. However, evaluating their capabilities has become increasingly expensive, as performing inference on a large number of benchmark samples incurs high computational costs. In this paper, we revisit the model-item performance matrix and show that it exhibits sparsity, that representative items can be selected as anchors, and that the task of efficient benchmarking can be formulated as a sparse optimization problem. Based on these insights, we propose SparseEval, a method that, for the first time, adopts gradient descent to optimize anchor weights and employs an iterative refinement strategy for anchor selection. We utilize the representation capacity of MLP to handle sparse optimization and propose the Anchor Importance Score and Candidate Importance Score to evaluate the value of each item for task-aware refinement. Extensive experiments demonstrate the low estimation error and high Kendall's~$\tau$ of our method across a variety of benchmarks, showcasing its superior robustness and practicality in real-world scenarios. Code is available at {this https URL}.
Abstract (translated)
随着大型语言模型（LLMs）的规模不断扩大，它们在各种下游任务中的性能有了显著提升。然而，评估这些模型的能力变得越来越昂贵，因为对大量基准样本进行推理会带来高昂的计算成本。在这篇论文中，我们重新审视了模型-项目表现矩阵，并展示了它表现出稀疏性、代表性的项目可以被选为锚点，以及高效基准测试的任务可以被表述为一个稀疏优化问题。基于这些见解，我们提出了SparseEval方法，首次采用梯度下降来优化锚权重，并采用迭代细化策略进行锚选择。我们利用MLP的表示能力处理稀疏优化，并提出锚重要性得分和候选重要性得分以评估每个项目在任务感知细化中的价值。广泛的实验展示了我们的方法在各种基准测试中具有低估计误差和高肯德尔$\tau$，体现了其在现实世界场景中的优越稳健性和实用性。代码可在{此链接}获取。
URL
https://arxiv.org/abs/2602.07909
PDF
-
Efficient Representations are Controllable Representations
Abstract
What is the most brute-force way to install interpretable, controllable features into a model's activations? Controlling how LLMs internally represent concepts typically requires sophisticated methods to first identify, then intervene on the model's existing feature geometry. We bypass all of this. We finetune an LLM with a simple auxiliary loss, training 16 of its 3072 residual stream dimensions to be inert interpretability flags that simply indicate what concepts are required for generation. The model reorganizes around them anyway, learning to rely on these flags during actual generation tasks. As a result, these inert flags become genuine internal features: interpretable control switches that allow us to steer generation at inference time. Why does this work? When a feature is reliably supplied at a fixed location, gradient descent gradually eliminates redundant encodings elsewhere, and the model erodes its own alternative representations. A model's efficiency pressure is a lever - exploitable to induce interpretable, controllable representations.
Abstract (translated)
最直接的强行方式是在模型激活中安装可解释和可控的功能是什么？控制LLM（大型语言模型）内部的概念表示通常需要先通过复杂的方法来识别，然后干预模型现有的特征几何结构。我们绕过了这一切。 我们通过对LLM进行微调，并添加一个简单的辅助损失函数，训练其3072个残差流维度中的16个维度成为惰性的可解释标志，这些标志简单地指示生成所需的概念。尽管如此，模型还是围绕这些标志进行了重组，并学会了在实际生成任务中依赖这些标志。 结果是，这些原本的惰性标志变成了真正的内部特征：允许我们在推理时间进行引导的可解释控制开关。 为什么这种方法有效？当一个功能在一个固定位置可靠地提供时，梯度下降会逐渐消除其他地方冗余的编码，模型也会侵蚀其自身的替代表示。模型的效率压力是一个杠杆，可以被利用来诱导出可解释且可控的表示形式。
URL
https://arxiv.org/abs/2602.07828
PDF
-
Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Induction
2026-02-07 14:18:11Antoine Gonon, Alexandre Cordonnier, Nicolas BoumalAbstract
Match-and-copy is a core retrieval primitive used at inference time by large language models to retrieve a matching token from the context then copy its successor. Yet, understanding how this behavior emerges on natural data is challenging because retrieval and memorization are entangled. To disentangle the two, we introduce Gaussian Match-and-Copy (GMC), a minimalist benchmark that isolates long-range retrieval through pure second-order correlation signals. Numerical investigations show that this task retains key qualitative aspects of how Transformers develop match-and-copy circuits in practice, and separates architectures by their retrieval capabilities. We also analyze the optimization dynamics in a simplified attention setting. Although many solutions are a priori possible under a regression objective, including ones that do not implement retrieval, we identify an implicit-bias regime in which gradient descent drives the parameters to diverge while their direction aligns with the max-margin separator, yielding hard match selection. We prove this max-margin alignment for GD trajectories that reach vanishing empirical loss under explicit technical conditions.
Abstract (translated)
匹配与复制（Match-and-Copy）是大型语言模型在推理过程中用于从上下文中检索匹配的标记然后复制其后继标记的核心检索机制。然而，由于检索和记忆之间的纠缠关系，在自然数据上理解这种行为的出现具有挑战性。为了分离这两种现象，我们引入了高斯匹配与复制（GMC），这是一个极简主义基准测试，通过纯粹的二阶相关信号来隔离长距离检索。 数值研究显示，这个任务保留了Transformer在实践中发展匹配与复制电路的关键定性方面，并通过其检索能力将不同架构区分开。我们还在简化注意力设置中分析了优化动力学。尽管根据回归目标有许多可能的解决方案（包括不实施检索的方法），但我们确定了一个隐含偏置（implicit bias）的区域，在此区域内，梯度下降会驱使参数发散的同时让它们的方向与最大间隔分割器对齐，从而实现硬匹配选择。我们证明了在明确的技术条件下，当GD轨迹达到零经验损失时，会发生这种最大间隔对齐。 简而言之，这项研究通过引入GMC基准测试和分析优化动力学来探究大型语言模型中的检索机制，并揭示了梯度下降如何促进特定的检索行为。
URL
https://arxiv.org/abs/2602.07562
PDF
-
Empirical Analysis of Adversarial Robustness and Explainability Drift in Cybersecurity Classifiers
2026-02-06 05:30:37Mona Rajhans, Vishal KhawareyAbstract
Machine learning (ML) models are increasingly deployed in cybersecurity applications such as phishing detection and network intrusion prevention. However, these models remain vulnerable to adversarial perturbations small, deliberate input modifications that can degrade detection accuracy and compromise interpretability. This paper presents an empirical study of adversarial robustness and explainability drift across two cybersecurity domains phishing URL classification and network intrusion detection. We evaluate the impact of L (infinity) bounded Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) perturbations on model accuracy and introduce a quantitative metric, the Robustness Index (RI), defined as the area under the accuracy perturbation curve. Gradient based feature sensitivity and SHAP based attribution drift analyses reveal which input features are most susceptible to adversarial manipulation. Experiments on the Phishing Websites and UNSW NB15 datasets show consistent robustness trends, with adversarial training improving RI by up to 9 percent while maintaining clean-data accuracy. These findings highlight the coupling between robustness and interpretability degradation and underscore the importance of quantitative evaluation in the design of trustworthy, AI-driven cybersecurity systems.
Abstract (translated)
机器学习（ML）模型在网络安全应用中越来越广泛，例如钓鱼检测和网络入侵预防。然而，这些模型仍然容易受到对抗性扰动的影响——即一些小的、故意的输入修改可以降低检测精度并破坏可解释性。本文通过两个网络安全领域：钓鱼网址分类和网络入侵检测，对模型的鲁棒性和可解释性的漂移进行了实证研究。我们评估了L（无穷）界Fast Gradient Sign Method (FGSM) 和 Projected Gradient Descent (PGD) 扰动对模型准确率的影响，并引入了一个量化指标——鲁棒性指数(RI)，定义为精度扰动曲线下的面积。基于梯度的特征敏感性和SHAP归因漂移分析揭示了哪些输入特征最容易受到对抗操作的影响。 在钓鱼网站和UNSW NB15数据集上的实验显示，对抗训练可以将RI提高最多9%，同时保持干净数据的准确率不变。这些发现强调了鲁棒性与可解释性退化的耦合关系，并且突出了在设计可信的人工智能驱动网络安全系统时进行量化评估的重要性。
URL
https://arxiv.org/abs/2602.06395
PDF
-
Emergent Low-Rank Training Dynamics in MLPs with Smooth Activations
2026-02-05 21:38:17Alec S. Xu, Can Yaras, Matthew Asato, Qing Qu, Laura BalzanoAbstract
Recent empirical evidence has demonstrated that the training dynamics of large-scale deep neural networks occur within low-dimensional subspaces. While this has inspired new research into low-rank training, compression, and adaptation, theoretical justification for these dynamics in nonlinear networks remains limited. %compared to deep linear settings. To address this gap, this paper analyzes the learning dynamics of multi-layer perceptrons (MLPs) under gradient descent (GD). We demonstrate that the weight dynamics concentrate within invariant low-dimensional subspaces throughout training. Theoretically, we precisely characterize these invariant subspaces for two-layer networks with smooth nonlinear activations, providing insight into their emergence. Experimentally, we validate that this phenomenon extends beyond our theoretical assumptions. Leveraging these insights, we empirically show there exists a low-rank MLP parameterization that, when initialized within the appropriate subspaces, matches the classification performance of fully-parameterized counterparts on a variety of classification tasks.
Abstract (translated)
最近的实证证据表明，大规模深度神经网络的训练动力学发生在低维子空间中。尽管这激发了对低秩训练、压缩和适应性的新研究，但非线性网络中这些动态行为的理论依据仍然有限。 为填补这一空白，本文分析了在梯度下降（GD）下多层感知器（MLPs）的学习动力学。我们展示了权重的动力学在整个训练过程中集中在不变的低维子空间内。理论上，对于具有平滑非线性激活的两层网络，我们精确地描述了这些不变子空间，为它们的出现提供了见解。实验上，我们验证了这一现象超越了我们的理论假设。 借助这些见解，我们在经验上展示了存在一个低秩MLP参数化方式，在适当初始化后，其在各种分类任务上的分类性能与完全参数化的同类网络相当。
URL
https://arxiv.org/abs/2602.06208
PDF
-
It's not a Lottery, it's a Race: Understanding How Gradient Descent Adapts the Network's Capacity to the Task
Abstract
Our theoretical understanding of neural networks is lagging behind their empirical success. One of the important unexplained phenomena is why and how, during the process of training with gradient descent, the theoretical capacity of neural networks is reduced to an effective capacity that fits the task. We here investigate the mechanism by which gradient descent achieves this through analyzing the learning dynamics at the level of individual neurons in single hidden layer ReLU networks. We identify three dynamical principles -- mutual alignment, unlocking and racing -- that together explain why we can often successfully reduce capacity after training through the merging of equivalent neurons or the pruning of low norm weights. We specifically explain the mechanism behind the lottery ticket conjecture, or why the specific, beneficial initial conditions of some neurons lead them to obtain higher weight norms.
Abstract (translated)
我们对神经网络的理论理解滞后于它们的经验成功。一个重要的未解释现象是，在使用梯度下降法进行训练的过程中，为什么和如何，神经网络的理论容量会被减少到一种有效容量以适应任务需求。本文通过分析单隐藏层ReLU网络中个别神经元的学习动力学来探讨梯度下降是如何实现这一过程的机制。我们识别出三个动态原则——互对齐、解锁和竞速，这些共同解释了为什么在训练后往往可以通过等效神经元合并或低权重裁剪成功地减少容量。具体来说，本文解释了彩票票证猜想背后的机制，即为何一些特定初始条件有利的神经元会获得更高的权重范数。
URL
https://arxiv.org/abs/2602.04832
PDF
-
DCER: Dual-Stage Compression and Energy-Based Reconstruction
2026-02-03 22:03:16Yiwen Wang, Jiahao QinAbstract
Multimodal fusion faces two robustness challenges: noisy inputs degrade representation quality, and missing modalities cause prediction failures. We propose DCER, a unified framework addressing both challenges through dual-stage compression and energy-based reconstruction. The compression stage operates at two levels: within-modality frequency transforms (wavelet for audio, DCT for video) remove noise while preserving task-relevant patterns, and cross-modality bottleneck tokens force genuine integration rather than modality-specific shortcuts. For missing modalities, energy-based reconstruction recovers representations via gradient descent on a learned energy function, with the final energy providing intrinsic uncertainty quantification (\r{ho} > 0.72 correlation with prediction error). Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate state-of-the-art performance across all benchmarks, with a U-shaped robustness pattern favoring multimodal fusion at both complete and high-missing conditions. The code will be available on Github.
Abstract (translated)
多模态融合面临两个鲁棒性挑战：噪声输入会降低表示质量，而缺失的模式会导致预测失败。我们提出了DCER框架，这是一个通过双阶段压缩和基于能量的重构同时解决这两个问题的统一方法。在压缩阶段，该框架运行于两个层面：在同一模式内的频域变换（音频使用小波变换，视频使用离散余弦变换）可以去除噪声并保留与任务相关的模式；跨模式瓶颈令牌则强制进行真实的整合而不是单一模式特有的捷径。对于缺失的模式，基于能量的重构通过在学习的能量函数上进行梯度下降来恢复表示，并且最终能量提供预测误差（\rho > 0.72 相关性）的内在不确定性量化。 实验结果表明，在CMU-MOSI、CMU-MOSEI和CH-SIMS数据集上的所有基准测试中，DCER框架表现出最先进的性能，并展示了一个U形鲁棒性模式，在完整和高缺失条件下均有利于多模态融合。代码将在Github上提供。
URL
https://arxiv.org/abs/2602.04904
PDF
-
Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations
2026-02-03 08:15:57Yuxuan Yao, Haonan Sheng, Qingsong Lv, Han Wu, Shuqi Liu, Zehua Liu, Zengyan Liu, Jiahui Gao, Haochen Tan, Xiaojin Fu, Haoli Bai, Hing Cheung So, Zhijiang Guo, Linqi SongAbstract
The escalating scale of Large Language Models (LLMs) necessitates efficient adaptation techniques. Model merging has gained prominence for its efficiency and controllability. However, existing merging techniques typically serve as post-hoc refinements or focus on mitigating task interference, often failing to capture the dynamic optimization benefits of supervised fine-tuning (SFT). In this work, we propose Streaming Merging, an innovative model updating paradigm that conceptualizes merging as an iterative optimization process. Central to this paradigm is \textbf{ARM} (\textbf{A}ctivation-guided \textbf{R}otation-aware \textbf{M}erging), a strategy designed to approximate gradient descent dynamics. By treating merging coefficients as learning rates and deriving rotation vectors from activation subspaces, ARM effectively steers parameter updates along data-driven trajectories. Unlike conventional linear interpolation, ARM aligns semantic subspaces to preserve the geometric structure of high-dimensional parameter evolution. Remarkably, ARM requires only early SFT checkpoints and, through iterative merging, surpasses the fully converged SFT model. Experimental results across model scales (1.7B to 14B) and diverse domains (e.g., math, code) demonstrate that ARM can transcend converged checkpoints. Extensive experiments show that ARM provides a scalable and lightweight framework for efficient model adaptation.
Abstract (translated)
大型语言模型（LLM）规模的不断扩大要求更高效的适应技术。模型合并因其高效性和可控性而备受关注。然而，现有的合并技术通常作为事后优化或专注于减少任务干扰，并且往往未能捕捉到有监督微调（SFT）中的动态优化收益。在这项工作中，我们提出了流式合并（Streaming Merging），这是一种创新的模型更新范式，它将合并视为一个迭代优化过程。这一范式的中心是**ARM**（激活指导的旋转感知合并），这是一种旨在近似梯度下降动力学的战略。通过将合并系数视为学习率并从激活子空间中推导出旋转向量，ARM能够引导参数沿数据驱动轨迹进行更新。不同于传统的线性插值方法，ARM对齐语义子空间以保持高维参数演化的几何结构。值得注意的是，ARM只需要早期的SFT检查点，并通过迭代合并超越了完全收敛的SFT模型。跨不同规模（从17亿到140亿）和多样领域（如数学、代码等）进行实验的结果表明，ARM能够超越已达成共识的检查点。广泛的实验显示，ARM提供了一个可扩展且轻量级的框架，用于高效地调整模型适应性。
URL
https://arxiv.org/abs/2602.03237
PDF
-
Reward-free Alignment for Conflicting Objectives
2026-02-02 18:59:52Peter Chen, Xiaopeng Li, Xi Chen, Tianyi LinAbstract
Direct alignment methods are increasingly used to align large language models (LLMs) with human preferences. However, many real-world alignment problems involve multiple conflicting objectives, where naive aggregation of preferences can lead to unstable training and poor trade-offs. In particular, weighted loss methods may fail to identify update directions that simultaneously improve all objectives, and existing multi-objective approaches often rely on explicit reward models, introducing additional complexity and distorting user-specified preferences. The contributions of this paper are two-fold. First, we propose a Reward-free Alignment framework for Conflicted Objectives (RACO) that directly leverages pairwise preference data and resolves gradient conflicts via a novel clipped variant of conflict-averse gradient descent. We provide convergence guarantees to Pareto-critical points that respect user-specified objective weights, and further show that clipping can strictly improve convergence rate in the two-objective setting. Second, we improve our method using some heuristics and conduct experiments to demonstrate the compatibility of the proposed framework for LLM alignment. Both qualitative and quantitative evaluations on multi-objective summarization and safety alignment tasks across multiple LLM families (Qwen 3, Llama 3, Gemma 3) show that our method consistently achieves better Pareto trade-offs compared to existing multi-objective alignment baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2602.02495
PDF
-
Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge
2026-02-02 18:50:57Xutao Ma, Yixiao Huang, Hanlin Zhu, Somayeh SojoudiAbstract
Autoregressive large language models (LLMs) have achieved remarkable success in many complex tasks, yet they can still fail in very simple logical reasoning such as the "reversal curse" -- when trained on forward knowledge data of the form "$A \rightarrow B$" (e.g., Alice's husband is Bob), the model is unable to deduce the reversal knowledge "$B \leftarrow A$" (e.g., Bob's wife is Alice) during test. Extensive prior research suggests that this failure is an inherent, fundamental limit of autoregressive causal LLMs, indicating that these models tend to memorize factual-level knowledge rather than capture higher-level rules. In this paper, we challenge this view by showing that this seemingly fundamental limit can be mitigated by slightly tweaking the training data with a simple regularization data recipe called the Identity Bridge of the form "$A \to A$" (e.g., The name of Alice is Alice). Theoretically, we prove that under this recipe, even a one-layer transformer can break the reversal curse by analyzing the implicit bias of gradient descent. Empirically, we show that a 1B pretrained language model finetuned with the proposed data recipe achieves a 40% success rate on reversal tasks, in stark contrast to a near-zero success rate when trained solely on forward-knowledge data. Our work provides a novel theoretical foundation for the reversal curse and offers a principled, low-cost path to encouraging LLMs to learn higher-level rules from data.
Abstract (translated)
URL
https://arxiv.org/abs/2602.02470
PDF
-
Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability
Abstract
How can we understand gradient-based training over non-convex landscapes? The edge of stability phenomenon, introduced in Cohen et al. (2021), indicates that the answer is not so simple: namely, gradient descent (GD) with large step sizes often diverges away from the gradient flow. In this regime, the "Central Flow", recently proposed in Cohen et al. (2025), provides an accurate ODE approximation to the GD dynamics over many architectures. In this work, we propose Rod Flow, an alternative ODE approximation, which carries the following advantages: (1) it rests on a principled derivation stemming from a physical picture of GD iterates as an extended one-dimensional object -- a "rod"; (2) it better captures GD dynamics for simple toy examples and matches the accuracy of Central Flow for representative neural network architectures, and (3) is explicit and cheap to compute. Theoretically, we prove that Rod Flow correctly predicts the critical sharpness threshold and explains self-stabilization in quartic potentials. We validate our theory with a range of numerical experiments.
Abstract (translated)
URL
https://arxiv.org/abs/2602.01480
PDF
-
Multi-Head Attention Is a Multi-Player Game
2026-01-31 18:49:49Kushal Chakrabarti, Nirmal BalachundarAbstract
Modern transformer attention is internally multi-agent -- heads compete and coordinate -- yet we train it as if it were a monolithic optimizer. We formalize this gap: cross-entropy training induces an implicit potential game among heads, and gradient descent converges to Nash equilibria with potentially unbounded inefficiency due to unpriced externalities (redundancy, correlated errors). Our main result bounds the Price of Anarchy by $\Gamma(G)$, the off-diagonal mass of a head interaction matrix capturing weight and gradient coupling. Under mild smoothness assumptions, we prove that both \emph{excess hallucination probability} and \emph{excess head redundancy} scale with PoA, unifying two distinct failure modes into a single mechanism. The bound is prescriptive: regularization that reduces $\Gamma(G)$ provably tightens PoA. We instantiate this as GAME-LoRA, combining Barlow Twins decorrelation with log-determinant coordination pressure. Experiments validate the theory: $\Gamma(G)$ predicts hallucination ($p{<}0.05$), emergent coalitions exhibit selective coordination, and GAME-LoRA achieves up to 18\% hallucination reduction (8\% average) with no knowledge degradation -- a Pareto improvement inaccessible to methods ignoring the game structure.
Abstract (translated)
URL
https://arxiv.org/abs/2602.00861
PDF
-
Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMs
2026-01-31 07:40:36Tushaar Gangavarapu, Jiping Li, Christopher Vattheuer, Zhangyang Wang, Baharan MirzasoleimanAbstract
Can modifying the training data distribution guide optimizers toward solutions with improved generalization when training large language models (LLMs)? In this work, we theoretically analyze an in-context linear regression model with multi-head linear self-attention, and compare the training dynamics of two gradient based optimizers, namely gradient descent (GD) and sharpness-aware minimization (SAM), the latter exhibiting superior generalization properties but is prohibitively expensive for training even medium-sized LLMs. We show, for the first time, that SAM induces a lower simplicity bias (SB)-the tendency of an optimizer to preferentially learn simpler features earlier in training-and identify this reduction as a key factor underlying its improved generalization performance. Motivated by this insight, we demonstrate that altering the training data distribution by upsampling or augmenting examples learned later in training similarly reduces SB and leads to improved generalization. Our extensive experiments show that our strategy improves the performance of multiple LLMs-including Phi2-2.7B , Llama3.2-1B, Gemma3-1B-PT, and Qwen3-0.6B-Base-achieving relative accuracy gains up to 18% when fine-tuned with AdamW and Muon on mathematical reasoning tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2602.00576
PDF
-
In-Run Data Shapley for Adam Optimizer
2026-01-30 21:31:40Meng Ding, Zeqing Zhang, Di Wang, Lijie HuAbstract
Reliable data attribution is essential for mitigating bias and reducing computational waste in modern machine learning, with the Shapley value serving as the theoretical gold standard. While recent "In-Run" methods bypass the prohibitive cost of retraining by estimating contributions dynamically, they heavily rely on the linear structure of Stochastic Gradient Descent (SGD) and fail to capture the complex dynamics of adaptive optimizers like Adam. In this work, we demonstrate that data attribution is inherently optimizer-dependent: we show that SGD-based proxies diverge significantly from true contributions under Adam (Pearson $R \approx 0.11$), rendering them ineffective for modern training pipelines. To bridge this gap, we propose Adam-Aware In-Run Data Shapley. We derive a closed-form approximation that restores additivity by redefining utility under a fixed-state assumption and enable scalable computation via a novel Linearized Ghost Approximation. This technique linearizes the variance-dependent scaling term, allowing us to compute pairwise gradient dot-products without materializing per-sample gradients. Extensive experiments show that our method achieves near-perfect fidelity to ground-truth marginal contributions ($R > 0.99$) while retaining $\sim$95\% of standard training throughput. Furthermore, our Adam-aware attribution significantly outperforms SGD-based baselines in data attribution downstream tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2602.00329
PDF
-
YuriiFormer: A Suite of Nesterov-Accelerated Transformers
2026-01-30 18:06:21Aleksandr Zimin, Yury Polyanskiy, Philippe RigolletAbstract
We propose a variational framework that interprets transformer layers as iterations of an optimization algorithm acting on token embeddings. In this view, self-attention implements a gradient step of an interaction energy, while MLP layers correspond to gradient updates of a potential energy. Standard GPT-style transformers emerge as vanilla gradient descent on the resulting composite objective, implemented via Lie--Trotter splitting between these two energy functionals. This perspective enables principled architectural design using classical optimization ideas. As a proof of concept, we introduce a Nesterov-style accelerated transformer that preserves the same attention and MLP oracles. The resulting architecture consistently outperforms a nanoGPT baseline on TinyStories and OpenWebText, demonstrating that optimization-theoretic insights can translate into practical gains.
Abstract (translated)
URL
https://arxiv.org/abs/2601.23236
PDF
-
Optimization, Generalization and Differential Privacy Bounds for Gradient Descent on Kolmogorov-Arnold Networks
2026-01-29 23:43:26Puyu Wang, Junyu Zhou, Philipp Liznerski, Marius KloftAbstract
Kolmogorov--Arnold Networks (KANs) have recently emerged as a structured alternative to standard MLPs, yet a principled theory for their training dynamics, generalization, and privacy properties remains limited. In this paper, we analyze gradient descent (GD) for training two-layer KANs and derive general bounds that characterize their training dynamics, generalization, and utility under differential privacy (DP). As a concrete instantiation, we specialize our analysis to logistic loss under an NTK-separable assumption, where we show that polylogarithmic network width suffices for GD to achieve an optimization rate of order $1/T$ and a generalization rate of order $1/n$, with $T$ denoting the number of GD iterations and $n$ the sample size. In the private setting, we characterize the noise required for $(\epsilon,\delta)$-DP and obtain a utility bound of order $\sqrt{d}/(n\epsilon)$ (with $d$ the input dimension), matching the classical lower bound for general convex Lipschitz problems. Our results imply that polylogarithmic width is not only sufficient but also necessary under differential privacy, revealing a qualitative gap between non-private (sufficiency only) and private (necessity also emerges) training regimes. Experiments further illustrate how these theoretical insights can guide practical choices, including network width selection and early stopping.
Abstract (translated)
URL
https://arxiv.org/abs/2601.22409
PDF
-
Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization
2026-01-29 23:16:31Kanishk AwadhiyaAbstract
Rotary Positional Embeddings (RoPE) have become the standard for Large Language Models (LLMs) due to their ability to encode relative positions through geometric rotation. However, we identify a significant limitation we term ''Spectral Rigidity'': standard RoPE utilizes a fixed geometric decay ($\theta^{-i}$) optimized for local syntactic coherence, which fails to capture the long-range, periodic structures inherent in recursive logic and algorithmic reasoning. This results in a ''Structure Gap'', where models trained on shallow reasoning chains fail to extrapolate to deeper recursive steps. In this work, we introduce Bifocal Attention, an architectural paradigm that decouples positional encoding into two distinct modalities: Geometric Eyes (Standard RoPE) for precise token-level manipulation, and Spectral Eyes (Learnable Harmonic Operators) for tracking long-range recursive depth. We propose a novel training protocol, Spectral Evolution, which initializes positional frequencies as static geometric parameters but allows them to evolve via gradient descent into a harmonic basis optimized for the specific algorithmic topology of the task.
Abstract (translated)
URL
https://arxiv.org/abs/2601.22402
PDF
-
Improving Classifier-Free Guidance of Flow Matching via Manifold Projection
2026-01-29 15:49:31Jian-Feng Cai, Haixia Liu, Zhengyi Su, Chao WangAbstract
Classifier-free guidance (CFG) is a widely used technique for controllable generation in diffusion and flow-based models. Despite its empirical success, CFG relies on a heuristic linear extrapolation that is often sensitive to the guidance scale. In this work, we provide a principled interpretation of CFG through the lens of optimization. We demonstrate that the velocity field in flow matching corresponds to the gradient of a sequence of smoothed distance functions, which guides latent variables toward the scaled target image set. This perspective reveals that the standard CFG formulation is an approximation of this gradient, where the prediction gap, the discrepancy between conditional and unconditional outputs, governs guidance sensitivity. Leveraging this insight, we reformulate the CFG sampling as a homotopy optimization with a manifold constraint. This formulation necessitates a manifold projection step, which we implement via an incremental gradient descent scheme during sampling. To improve computational efficiency and stability, we further enhance this iterative process with Anderson Acceleration without requiring additional model evaluations. Our proposed methods are training-free and consistently refine generation fidelity, prompt alignment, and robustness to the guidance scale. We validate their effectiveness across diverse benchmarks, demonstrating significant improvements on large-scale models such as DiT-XL-2-256, Flux, and Stable Diffusion 3.5.
Abstract (translated)
URL
https://arxiv.org/abs/2601.21892
PDF
-
Scalable Analytic Classifiers with Associative Drift Compensation for Class-Incremental Learning of Vision Transformers
2026-01-29 06:42:20Xuan Rao, Mingming Ha, Bo Zhao, Derong Liu, Cesare AlippiAbstract
Class-incremental learning (CIL) with Vision Transformers (ViTs) faces a major computational bottleneck during the classifier reconstruction phase, where most existing methods rely on costly iterative stochastic gradient descent (SGD). We observe that analytic Regularized Gaussian Discriminant Analysis (RGDA) provides a Bayes-optimal alternative with accuracy comparable to SGD-based classifiers; however, its quadratic inference complexity limits its use in large-scale CIL scenarios. To overcome this, we propose Low-Rank Factorized RGDA (LR-RGDA), a scalable classifier that combines RGDA's expressivity with the efficiency of linear classifiers. By exploiting the low-rank structure of the covariance via the Woodbury matrix identity, LR-RGDA decomposes the discriminant function into a global affine term refined by a low-rank quadratic perturbation, reducing the inference complexity from $\mathcal{O}(Cd^2)$ to $\mathcal{O}(d^2 + Crd^2)$, where $C$ is the class number, $d$ the feature dimension, and $r \ll d$ the subspace rank. To mitigate representation drift caused by backbone updates, we further introduce Hopfield-based Distribution Compensator (HopDC), a training-free mechanism that uses modern continuous Hopfield Networks to recalibrate historical class statistics through associative memory dynamics on unlabeled anchors, accompanied by a theoretical bound on the estimation error. Extensive experiments on diverse CIL benchmarks demonstrate that our framework achieves state-of-the-art performance, providing a scalable solution for large-scale class-incremental learning with ViTs. Code: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.00144
PDF
-
Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts
2026-01-27 17:12:15TrungKhang Tran, TrungTin Nguyen, Gersende Fort, Tung Doan, Hien Duy Nguyen, Binh T. Nguyen, Florence Forbes, Christopher DrovandiAbstract
Processing high-volume, streaming data is increasingly common in modern statistics and machine learning, where batch-mode algorithms are often impractical because they require repeated passes over the full dataset. This has motivated incremental stochastic estimation methods, including the incremental stochastic Expectation-Maximization (EM) algorithm formulated via stochastic approximation. In this work, we revisit and analyze an incremental stochastic variant of the Majorization-Minimization (MM) algorithm, which generalizes incremental stochastic EM as a special case. Our approach relaxes key EM requirements, such as explicit latent-variable representations, enabling broader applicability and greater algorithmic flexibility. We establish theoretical guarantees for the incremental stochastic MM algorithm, proving consistency in the sense that the iterates converge to a stationary point characterized by a vanishing gradient of the objective. We demonstrate these advantages on a softmax-gated mixture of experts (MoE) regression problem, for which no stochastic EM algorithm is available. Empirically, our method consistently outperforms widely used stochastic optimizers, including stochastic gradient descent, root mean square propagation, adaptive moment estimation, and second-order clipped stochastic optimization. These results support the development of new incremental stochastic algorithms, given the central role of softmax-gated MoE architectures in contemporary deep neural networks for heterogeneous data modeling. Beyond synthetic experiments, we also validate practical effectiveness on two real-world datasets, including a bioinformatics study of dent maize genotypes under drought stress that integrates high-dimensional proteomics with ecophysiological traits, where incremental stochastic MM yields stable gains in predictive performance.
Abstract (translated)
URL
https://arxiv.org/abs/2601.19811
PDF
