Gradient_Descent
-
Evaluating the Impact of Adversarial Attacks on Traffic Sign Classification using the LISA Dataset
2025-09-08 16:06:41Nabeyou Tadessa, Balaji Iyangar, Mashrur ChowdhuryAbstract
Adversarial attacks pose significant threats to machine learning models by introducing carefully crafted perturbations that cause misclassification. While prior work has primarily focused on MNIST and similar datasets, this paper investigates the vulnerability of traffic sign classifiers using the LISA Traffic Sign dataset. We train a convolutional neural network to classify 47 different traffic signs and evaluate its robustness against Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks. Our results show a sharp decline in classification accuracy as the perturbation magnitude increases, highlighting the models susceptibility to adversarial examples. This study lays the groundwork for future exploration into defense mechanisms tailored for real-world traffic sign recognition systems.
Abstract (translated)
对抗性攻击通过引入精心设计的扰动来误导机器学习模型，从而对这些模型构成重大威胁。虽然之前的研究主要集中在MNIST和类似的数据库上，但本文探讨了使用LISA交通标志数据集评估交通标志分类器的安全漏洞问题。我们训练了一个卷积神经网络来识别47种不同的交通标志，并用快速梯度符号方法（FGSM）和投影梯度下降法（PGD）攻击对其进行鲁棒性测试。我们的结果显示，随着扰动大小的增加，分类准确率急剧下降，这凸显了模型对于对抗样本的脆弱性。这项研究为未来开发针对现实世界交通标志识别系统的防御机制奠定了基础。
URL
https://arxiv.org/abs/2509.06835
PDF
-
MachineLearningLM: Continued Pretraining Language Models on Millions of Synthetic Tabular Prediction Tasks Scales In-Context ML
2025-09-08 15:38:31Haoyu Dong, Pengkun Zhang, Mingzhe Lu, Yanzhen Shen, Guolin KeAbstract
Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks, that is, to leverage many-shot demonstrations purely via in-context learning (ICL) without gradient descent. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge and reasoning for broader chat workflows. Our pretraining procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. We begin with a random-forest teacher, distilling tree-based decision strategies into the LLM to strengthen robustness in numerical modeling. All tasks are serialized with a token-efficient prompt, enabling 3x to 6x more examples per context window and delivering up to 50x amortized throughput via batch inference. Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8), MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an average of about 15% on out-of-distribution tabular classification across finance, physics, biology, and healthcare domains. It exhibits a striking many-shot scaling law: accuracy increases monotonically as in-context demonstrations grow from 8 to 1,024. Without any task-specific training, it attains random-forest-level accuracy across hundreds of shots. General chat capabilities, including knowledge and reasoning, are preserved: it achieves 75.4% on MMLU.
Abstract (translated)
大型语言模型（LLMs）拥有广泛的世界知识和强大的通用推理能力，但它们在标准机器学习（ML）任务上难以从许多上下文示例中进行学习，即无法通过纯上下文学习（ICL）而不使用梯度下降方法来利用多示例演示。我们引入了MachineLearningLM，这是一个便携式的持续预训练框架，它为通用的LLM配备了稳健的上下文机器学习能力，同时保留了其广泛的常识和推理能力，适用于更广泛的任务流程。 我们的预训练过程从数百万个结构因果模型（SCMs）中合成ML任务，并扩展到1,024个示例数量。我们首先使用随机森林教师将基于树的决策策略提炼进LLM，以增强数值建模中的鲁棒性。所有任务都使用一个高效的提示进行序列化，这使得每个上下文窗口内的示例数量最多可增加3至6倍，并通过批量推理实现了最高50倍的平均吞吐量。 尽管采用了适度配置（如Qwen-2.5-7B-Instruct和LoRA秩8），MachineLearningLM在跨金融、物理、生物及医疗领域分布外表格分类任务上，相对于强大的LLM基线模型（例如GPT-5-mini）表现出约15%的平均性能提升。它还展示了显著的多示例扩展法则：随着上下文演示从8个增加到1,024个，准确率单调上升；无需特定任务训练，在数百个示例的情况下也能达到随机森林级别的精度。 此外，其通用聊天能力（包括知识和推理）得到了保持：在MMLU基准测试中，它达到了75.4%的分数。
URL
https://arxiv.org/abs/2509.06806
PDF
-
Reasoning-Intensive Regression
2025-08-29 16:37:42Diane Tchuindjo, Omar KhattabAbstract
AI researchers and practitioners increasingly apply large language models (LLMs) to what we call reasoning-intensive regression (RiR), i.e. deducing subtle numerical properties from text. Unlike standard language regression tasks, e.g. for sentiment or similarity, RiR often appears instead in ad-hoc problems like rubric-based scoring or domain-specific retrieval, where much deeper analysis of text is required while only limited task-specific training data and computation are available. We cast three realistic problems as RiR tasks to establish an initial benchmark, and use that to test our hypothesis that prompting frozen LLMs and finetuning Transformer encoders via gradient descent will both often struggle in RiR. We then propose MENTAT, a simple and lightweight method that combines batch-reflective prompt optimization with neural ensemble learning. MENTAT achieves up to 65% improvement over both baselines, though substantial room remains for future advances in RiR.
Abstract (translated)
AI研究人员和实践者越来越多地将大型语言模型（LLMs）应用于我们所说的推理密集型回归（RiR），即从文本中推断出细微的数值属性。与标准的语言回归任务，如情感分析或相似度评估不同，RiR通常出现在一些临时性问题上，例如基于评分细则的评分或特定领域的检索，在这些问题中需要对文本进行更深入的分析，而同时可用的任务特定训练数据和计算资源却非常有限。我们把三个现实中的问题作为RiR任务来建立初步基准，并利用这些任务测试我们的假设：即通过提示冻结的语言模型以及通过梯度下降微调变换器编码器在处理RiR时常常会遇到困难。然后，我们提出了MENTAT方法，这是一种简单而轻量级的方法，结合了批量自省的提示优化与神经集成学习。相比基准线，MENTAT实现了高达65%的改进，尽管未来在RiR领域仍然存在很大的进步空间。
URL
https://arxiv.org/abs/2508.21762
PDF
-
Unfolding Framework with Complex-Valued Deformable Attention for High-Quality Computer-Generated Hologram Generation
2025-08-29 14:21:22Haomiao Zhang, Zhangyuan Li, Yanling Piao, Zhi Li, Xiaodong Wang, Miao Cao, Xiongfei Su, Qiang Song, Xin YuanAbstract
Computer-generated holography (CGH) has gained wide attention with deep learning-based algorithms. However, due to its nonlinear and ill-posed nature, challenges remain in achieving accurate and stable reconstruction. Specifically, ($i$) the widely used end-to-end networks treat the reconstruction model as a black box, ignoring underlying physical relationships, which reduces interpretability and flexibility. ($ii$) CNN-based CGH algorithms have limited receptive fields, hindering their ability to capture long-range dependencies and global context. ($iii$) Angular spectrum method (ASM)-based models are constrained to finite this http URL this paper, we propose a Deep Unfolding Network (DUN) that decomposes gradient descent into two modules: an adaptive bandwidth-preserving model (ABPM) and a phase-domain complex-valued denoiser (PCD), providing more flexibility. ABPM allows for wider working distances compared to ASM-based methods. At the same time, PCD leverages its complex-valued deformable self-attention module to capture global features and enhance performance, achieving a PSNR over 35 dB. Experiments on simulated and real data show state-of-the-art results.
Abstract (translated)
计算机生成全息图（CGH）由于基于深度学习的算法而受到广泛关注。然而，由于其非线性和不适定特性，实现准确且稳定的重建仍面临挑战。具体来说： (i) 广泛使用的端到端网络将重建模型视为黑盒模型，忽略了潜在的物理关系，降低了可解释性与灵活性。 (ii) 基于卷积神经网络（CNN）的CGH算法具有有限的感受野，难以捕捉长距离依赖和全局上下文信息。 (iii) 基于角谱法（ASM）的方法受到计算域限制。本文中，我们提出了一种深度展开网络（DUN），将梯度下降分解为两个模块：自适应带宽保持模型（ABPM）和相位域复值去噪器（PCD）。这种方法提供了更高的灵活性。 ABPM 允许与基于 ASM 的方法相比具有更宽的工作距离。同时，PCD 通过其复数可变形的自我注意机制来捕获全局特征并增强性能，实现了超过35dB的PSNR。在模拟和真实数据上的实验显示了最先进的结果。
URL
https://arxiv.org/abs/2508.21657
PDF
-
Adaptive Heavy-Tailed Stochastic Gradient Descent
2025-08-29 06:32:26Bodu Gong, Gustavo Enrique Batista, Pierre Lafaye de MicheauxAbstract
In the era of large-scale neural network models, optimization algorithms often struggle with generalization due to an overreliance on training loss. One key insight widely accepted in the machine learning community is the idea that wide basins (regions around a local minimum where the loss increases gradually) promote better generalization by offering greater stability to small changes in input data or model parameters. In contrast, sharp minima are typically more sensitive and less stable. Motivated by two key empirical observations - the inherent heavy-tailed distribution of gradient noise in stochastic gradient descent and the Edge of Stability phenomenon during neural network training, in which curvature grows before settling at a plateau, we introduce Adaptive Heavy Tailed Stochastic Gradient Descent (AHTSGD). The algorithm injects heavier-tailed noise into the optimizer during the early stages of training to enhance exploration and gradually transitions to lighter-tailed noise as sharpness stabilizes. By dynamically adapting to the sharpness of the loss landscape throughout training, AHTSGD promotes accelerated convergence to wide basins. AHTSGD is the first algorithm to adjust the nature of injected noise into an optimizer based on the Edge of Stability phenomenon. AHTSGD consistently outperforms SGD and other noise-based methods on benchmarks like MNIST and CIFAR-10, with marked gains on noisy datasets such as SVHN. It ultimately accelerates early training from poor initializations and improves generalization across clean and noisy settings, remaining robust to learning rate choices.
Abstract (translated)
在大规模神经网络模型的时代，优化算法由于过度依赖训练损失而常常难以实现泛化。机器学习社区广泛接受的一个关键见解是：宽盆地（即围绕局部最小值的区域，在该区域内损失逐渐增加）通过提供对输入数据或模型参数的小变化更稳定的特性来促进更好的泛化能力；相比之下，尖锐极小值通常更加敏感且不够稳定。 受到两个重要经验观察结果的启发——随机梯度下降中的梯度噪声呈现固有的重尾分布以及在神经网络训练过程中边缘稳定性现象的存在，在这一现象中曲率会在达到平台期之前先增长。我们提出了自适应重尾随机梯度下降（AHTSGD）。该算法在训练初期向优化器注入更重尾的噪声以增强探索，并随着尖锐性的稳定逐渐过渡到较轻尾的噪声。通过在整个训练过程中动态地调整注入噪声的性质，AHTSGD促进了加速收敛于宽盆地。 AHTSGD是首个根据边缘稳定性现象来调整输入给优化器中的噪声特性的算法。在MNIST和CIFAR-10等基准测试中，AHTSGD始终优于随机梯度下降（SGD）和其他基于噪声的方法，在SVHN这样的嘈杂数据集上表现出显著的改进效果。它最终加快了从较差初始化开始时的早期训练速度，并在整个清洁与嘈杂环境设置下提高了泛化能力，同时对学习率的选择保持鲁棒性。
URL
https://arxiv.org/abs/2508.21353
PDF
-
Observer Design for Optical Flow-Based Visual-Inertial Odometry with Almost-Global Convergence
2025-08-28 18:55:16Tarek Bouazza, Soulaimane Berkane, Minh-Duc Hua, Tarek HamelAbstract
This paper presents a novel cascaded observer architecture that combines optical flow and IMU measurements to perform continuous monocular visual-inertial odometry (VIO). The proposed solution estimates body-frame velocity and gravity direction simultaneously by fusing velocity direction information from optical flow measurements with gyro and accelerometer data. This fusion is achieved using a globally exponentially stable Riccati observer, which operates under persistently exciting translational motion conditions. The estimated gravity direction in the body frame is then employed, along with an optional magnetometer measurement, to design a complementary observer on $\mathbf{SO}(3)$ for attitude estimation. The resulting interconnected observer architecture is shown to be almost globally asymptotically stable. To extract the velocity direction from sparse optical flow data, a gradient descent algorithm is developed to solve a constrained minimization problem on the unit sphere. The effectiveness of the proposed algorithms is validated through simulation results.
Abstract (translated)
这篇论文提出了一种新颖的级联观测器架构，该架构结合了光流和惯性测量单元（IMU）的数据来执行连续单目视觉惯性里程计（VIO）。所提出的解决方案通过融合来自光流测量的速度方向信息与陀螺仪和加速度计数据，在同一个框架内同时估计机体坐标系中的速度和重力方向。这种融合是利用一个在持续激励平移运动条件下全局指数稳定的里卡蒂观测器来实现的。 然后，以机体系中估算出的重力方向为基础，并可选择性地结合磁力计测量值，设计了一个定义在$\mathbf{SO}(3)$上的互补观测器来进行姿态估计。所得到的互联观测器架构被证明是几乎全局渐近稳定的。 为了从稀疏光流数据中提取速度方向，开发了一种梯度下降算法来解决单位球面上的一个约束最小化问题。通过仿真结果验证了所提出的算法的有效性。
URL
https://arxiv.org/abs/2508.21163
PDF
-
Self-Supervised Pre-Training with Equilibrium Constraints
2025-08-27 15:48:50Xiaodong Cui, A F M Saif, Brian Kingsbury, Tianyi ChenAbstract
Self-supervised pre-training using unlabeled data is widely used in machine learning. In this paper, we propose a new self-supervised pre-training approach to dealing with heterogeneous data. Instead of mixing all the data and minimizing the averaged global loss in the conventional way, we impose additional equilibrium constraints to ensure that the models optimizes each source of heterogeneous data to its local optima after $K$-step gradient descent initialized from the model. We formulate this as a bilevel optimization problem, and use the first-order approximation method to solve the problem. We discuss its connection to model-agnostic meta learning (MAML). Experiments are carried out on self-supervised pre-training using multi-domain and multilingual datasets, demonstrating that the proposed approach can significantly improve the adaptivity of the self-supervised pre-trained model for the downstream supervised fine-tuning tasks.
Abstract (translated)
无监督预训练在使用未标记数据的机器学习中广泛应用。本文提出了一种新的自监督预训练方法，用于处理异构数据。不同于传统方法将所有数据混合并最小化平均全局损失，我们引入了额外的平衡约束条件，以确保模型从初始化开始经过 $K$ 步梯度下降后能够优化每一种异构数据到其局部最优解。我们将此问题形式化为双层优化问题，并使用一阶近似方法来求解。讨论了该方法与无模型元学习（MAML）的联系。在多领域和多语言数据集上的自监督预训练实验表明，所提出的这种方法可以显著提高自监督预训练模型对下游有监督微调任务的适应性。
URL
https://arxiv.org/abs/2508.19990
PDF
-
Attention is also needed for form design
2025-08-27 09:15:21B. Sankar, Dibakar SenAbstract
Conventional product design is a cognitively demanding process, limited by its time-consuming nature, reliance on subjective expertise, and the opaque translation of inspiration into tangible concepts. This research introduces a novel, attention-aware framework that integrates two synergistic systems: EUPHORIA, an immersive Virtual Reality environment using eye-tracking to implicitly capture a designer's aesthetic preferences, and RETINA, an agentic AI pipeline that translates these implicit preferences into concrete design outputs. The foundational principles were validated in a two-part study. An initial study correlated user's implicit attention with explicit preference and the next one correlated mood to attention. A comparative study where 4 designers solved challenging design problems using 4 distinct workflows, from a manual process to an end-to-end automated pipeline, showed the integrated EUPHORIA-RETINA workflow was over 4 times more time-efficient than the conventional method. A panel of 50 design experts evaluated the 16 final renderings. Designs generated by the fully automated system consistently received the highest Worthiness (calculated by an inverse Plackett-Luce model based on gradient descent optimization) and Design Effectiveness scores, indicating superior quality across 8 criteria: novelty, visual appeal, emotional resonance, clarity of purpose, distinctiveness of silhouette, implied materiality, proportional balance, & adherence to the brief. This research presents a validated paradigm shift from traditional Computer-Assisted Design (CAD) to a collaborative model of Designer-Assisting Computers (DAC). By automating logistical and skill-dependent generative tasks, the proposed framework elevates the designer's role to that of a creative director, synergizing human intuition with the generative power of agentic AI to produce higher-quality designs more efficiently.
Abstract (translated)
传统的商品设计是一个认知负担重的过程，受限于其耗时性、依赖主观专业知识以及将灵感转化为具体概念的不透明过程。这项研究介绍了一种新颖且具有注意力感知能力的框架，整合了两个协同系统：EUPHORIA和RETINA。其中，EUPHORIA是一个利用眼动追踪技术捕捉设计师审美偏好的沉浸式虚拟现实环境；而RETINA则是一条代理人工智能管道，能够将这些隐性的偏好转化为具体的设 计输出。 该基础原理在两项研究中得到了验证：一项初步研究将用户隐性注意力与显性偏好进行了关联，另一项研究将情绪和注意力相关联。在一项比较研究中，4位设计师使用了四种不同的工作流程来解决设计难题，从手动过程到端到端的自动化管道，结果显示集成的EUPHORIA-RETINA工作流比传统方法的时间效率高出四倍以上。 一个由50名设计专家组成的评审团对16个最终渲染进行了评估。全自动化系统生成的设计在新颖性、视觉吸引力、情感共鸣、目标清晰度、轮廓独特性、材料暗示、比例平衡以及符合设计简报的八个标准上，持续获得最高的“价值”（通过基于梯度下降优化的逆Plackett-Luce模型计算）和“设计有效性”评分，表明其质量明显优于传统方法。 这项研究提出了一种从传统的计算机辅助设计(CAD)转向设计师与计算机协作模式的设计范式转变。通过自动化物流性和技能依赖性的生成任务，该框架提升了设计师的角色至创意总监，并将人类的直觉与代理人工智能的生成能力相结合，以更高效地创造出更高品质的设计。
URL
https://arxiv.org/abs/2508.19708
PDF
-
A Symbolic Adversarial Learning Framework for Evolving Fake News Generation and Detection
2025-08-27 07:14:17Chong Tian, Qirong Ho, Xiuying ChenAbstract
Rapid LLM advancements heighten fake news risks by enabling the automatic generation of increasingly sophisticated misinformation. Previous detection methods, including fine-tuned small models or LLM-based detectors, often struggle with its dynamically evolving nature. In this work, we propose a novel framework called the Symbolic Adversarial Learning Framework (SALF), which implements an adversarial training paradigm by an agent symbolic learning optimization process, rather than relying on numerical updates. SALF introduces a paradigm where the generation agent crafts deceptive narratives, and the detection agent uses structured debates to identify logical and factual flaws for detection, and they iteratively refine themselves through such adversarial interactions. Unlike traditional neural updates, we represent agents using agent symbolic learning, where learnable weights are defined by agent prompts, and simulate back-propagation and gradient descent by operating on natural language representations of weights, loss, and gradients. Experiments on two multilingual benchmark datasets demonstrate SALF's effectiveness, showing it generates sophisticated fake news that degrades state-of-the-art detection performance by up to 53.4% in Chinese and 34.2% in English on average. SALF also refines detectors, improving detection of refined content by up to 7.7%. We hope our work inspires further exploration into more robust, adaptable fake news detection systems.
Abstract (translated)
快速发展的大型语言模型（LLM）技术提高了虚假新闻的风险，因为它可以自动产生越来越复杂的误导信息。以往的检测方法，包括微调的小型模型或基于LLM的检测器，往往难以应对这种动态变化的特性。在这项工作中，我们提出了一种新的框架——符号对抗学习框架（SALF），该框架通过一个代理符号学习优化过程实施了对抗性训练范式，而不是依赖数值更新。在SALF中，生成者创建误导性的叙述，而检测器则使用结构化的辩论来识别逻辑和事实上的错误进行检测；两者通过这种对抗性互动不断自我完善。 与传统的神经网络更新不同，我们用代理符号学习表示代理，其中可学习的权重由代理提示定义，并通过操作自然语言表达形式的权重、损失和梯度来模拟反向传播和梯度下降。在两个多语言基准数据集上的实验表明，SALF表现出色：它生成了复杂且具有欺骗性的虚假新闻，在中文上降低了最先进的检测性能平均高达53.4%，在英文上则为34.2%。同时，SALF还改进了探测器的准确性，提高了对经过优化的内容进行识别的能力最多可达7.7%。 我们希望我们的工作能够激发更多关于更强大、适应性更强的虚假新闻检测系统的研究探索。
URL
https://arxiv.org/abs/2508.19633
PDF
-
Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
2025-08-27 04:46:56Yuhang Liu, Tao Li, Zhehao Huang, Zuopeng Yang, Xiaolin HuangAbstract
Fine-tuning large-scale pre-trained models with limited data presents significant challenges for generalization. While Sharpness-Aware Minimization (SAM) has proven effective in improving generalization by seeking flat minima, its substantial extra memory and computation overhead make it impractical for large models. Integrating SAM with parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) is a promising direction. However, we find that directly applying SAM to LoRA parameters limits the sharpness optimization to a restricted subspace, hindering its effectiveness. To address this limitation, we propose Bi-directional Low-Rank Adaptation (Bi-LoRA), which introduces an auxiliary LoRA module to model SAM's adversarial weight perturbations. It decouples SAM's weight perturbations from LoRA optimization: the primary LoRA module adapts to specific tasks via standard gradient descent, while the auxiliary module captures the sharpness of the loss landscape through gradient ascent. Such dual-module design enables Bi-LoRA to capture broader sharpness for achieving flatter minima while remaining memory-efficient. Another important benefit is that the dual design allows for simultaneous optimization and perturbation, eliminating SAM's doubled training costs. Extensive experiments across diverse tasks and architectures demonstrate Bi-LoRA's efficiency and effectiveness in enhancing generalization.
Abstract (translated)
在数据有限的情况下，对大规模预训练模型进行微调以实现泛化是一项重大挑战。虽然敏锐度感知最小化（Sharpness-Aware Minimization, SAM）通过寻找平坦的极小值来提高泛化的有效性已经得到了证明，但其显著增加的记忆和计算开销使其在大型模型上的应用变得不切实际。将SAM与低秩适应（Low-Rank Adaptation, LoRA）等参数高效微调方法结合是一种有前景的方向。然而，我们发现直接将SAM应用于LoRA参数会限制尖锐度优化到一个受限的子空间中，从而影响其效果。 为了解决这一局限性，我们提出了一种双向低秩适应（Bi-directional Low-Rank Adaptation, Bi-LoRA）方法，通过引入辅助的LoRA模块来模拟SAM的对抗权重扰动。这种方法将SAM的权重扰动与LoRA优化过程解耦：主LoRA模块通过标准梯度下降法针对特定任务进行调整，而辅助模块则通过梯度上升捕捉损失地形图的锐利性。这种双模块设计使Bi-LoRA能够在保持内存效率的同时捕获更广泛的锐度以实现更平坦的极小值。此外，双重设计还允许同时进行优化和扰动，消除了SAM加倍训练成本的问题。 跨多种任务和架构的大量实验表明了Bi-LoRA在提高泛化能力方面既高效又有效。
URL
https://arxiv.org/abs/2508.19564
PDF
-
Robustness Feature Adapter for Efficient Adversarial Training
2025-08-25 05:23:50Quanwei Wu, Jun Guo, Wei Wang, Yi WangAbstract
Adversarial training (AT) with projected gradient descent is the most popular method to improve model robustness under adversarial attacks. However, computational overheads become prohibitively large when AT is applied to large backbone models. AT is also known to have the issue of robust overfitting. This paper contributes to solving both problems simultaneously towards building more trustworthy foundation models. In particular, we propose a new adapter-based approach for efficient AT directly in the feature space. We show that the proposed adapter-based approach can improve the inner-loop convergence quality by eliminating robust overfitting. As a result, it significantly increases computational efficiency and improves model accuracy by generalizing adversarial robustness to unseen attacks. We demonstrate the effectiveness of the new adapter-based approach in different backbone architectures and in AT at scale.
Abstract (translated)
URL
https://arxiv.org/abs/2508.17680
PDF
-
Consistent Opponent Modeling of Static Opponents in Imperfect-Information Games
Abstract
The goal of agents in multi-agent environments is to maximize total reward against the opposing agents that are encountered. Following a game-theoretic solution concept, such as Nash equilibrium, may obtain a strong performance in some settings; however, such approaches fail to capitalize on historical and observed data from repeated interactions against our opponents. Opponent modeling algorithms integrate machine learning techniques to exploit suboptimal opponents utilizing available data; however, the effectiveness of such approaches in imperfect-information games to date is quite limited. We show that existing opponent modeling approaches fail to satisfy a simple desirable property even against static opponents drawn from a known prior distribution; namely, they do not guarantee that the model approaches the opponent's true strategy even in the limit as the number of game iterations approaches infinity. We develop a new algorithm that is able to achieve this property and runs efficiently by solving a convex minimization problem based on the sequence-form game representation using projected gradient descent. The algorithm is guaranteed to efficiently converge to the opponent's true strategy given observations from gameplay and possibly additional historical data if it is available.
Abstract (translated)
在多代理环境中的代理目标是最大化与遇到的对立代理之间的总奖励。根据博弈论解决方案概念，如纳什均衡，在某些情况下可以获得强大的性能；然而，这类方法未能充分利用反复互动中从对手那里获得的历史和观察数据。对手建模算法通过整合机器学习技术来利用已有的数据对次优对手进行优化；然而，这些方法在不完全信息游戏中迄今为止的有效性相当有限。 我们展示了现有的对手建模方法甚至对于来自已知先验分布的静态对手都无法满足一个简单的理想属性：即它们无法保证模型会随着游戏轮数趋近于无穷时逐渐接近对手的真实策略。为此，我们开发了一种新算法，该算法能够解决这一问题，并通过基于序列形式博弈表示求解凸优化问题（使用投影梯度下降）高效运行。 该算法确保在给定游戏观察数据和可能的额外历史数据的情况下，能有效地收敛到对手的真实策略。
URL
https://arxiv.org/abs/2508.17671
PDF
-
In-Context Algorithm Emulation in Fixed-Weight Transformers
2025-08-24 23:20:31Jerry Yao-Chieh Hu, Hude Liu, Jennifer Yuntong Zhang, Han LiuAbstract
We prove that a minimal Transformer architecture with frozen weights is capable of emulating a broad class of algorithms by in-context prompting. In particular, for any algorithm implementable by a fixed-weight attention head (e.g. one-step gradient descent or linear/ridge regression), there exists a prompt that drives a two-layer softmax attention module to reproduce the algorithm's output with arbitrary precision. This guarantee extends even to a single-head attention layer (using longer prompts if necessary), achieving architectural minimality. Our key idea is to construct prompts that encode an algorithm's parameters into token representations, creating sharp dot-product gaps that force the softmax attention to follow the intended computation. This construction requires no feed-forward layers and no parameter updates. All adaptation happens through the prompt alone. These findings forge a direct link between in-context learning and algorithmic emulation, and offer a simple mechanism for large Transformers to serve as prompt-programmable libraries of algorithms. They illuminate how GPT-style foundation models may swap algorithms via prompts alone, establishing a form of algorithmic universality in modern Transformer models.
Abstract (translated)
我们证明，一个权重固定化的最小Transformer架构能够通过上下文提示来模拟一大类算法。具体来说，对于任何可以通过固定权重的注意力头实现的算法（例如一步梯度下降或线性/岭回归），都存在一种提示，可以驱动两层softmax注意力模块以任意精度重现该算法的输出。这一保证甚至适用于单个头部注意层（如果需要的话使用更长的提示），实现了架构上的最小化。我们的核心思想是构造能够将算法参数编码为标记表示形式的提示，创造出明确的点积差距，迫使softmax注意机制按照预期进行计算。这种构建不需要前向传播层和参数更新。所有适应性都通过提示单独完成。 这些发现直接建立了上下文学习与算法模拟之间的联系，并提供了一种简单的机制，使大型Transformer可以充当可通过提示编程的算法库。它们揭示了GPT风格的基础模型如何仅通过提示即可交换算法，从而在现代Transformer模型中建立了一种形式的算法普适性。
URL
https://arxiv.org/abs/2508.17550
PDF
-
ONG: Orthogonal Natural Gradient Descent
2025-08-24 00:27:23Yajat Yadav, Jathin Korrapati, Patrick MendozaAbstract
Orthogonal gradient descent has emerged as a powerful method for continual learning tasks. However, its Euclidean projections overlook the underlying information-geometric structure of the space of distributions parametrized by neural networks, which can lead to suboptimal convergence in learning tasks. To counteract this, we combine it with the idea of the natural gradient and present ONG (Orthogonal Natural Gradient Descent). ONG preconditions each new task gradient with an efficient EKFAC approximation of the inverse Fisher information matrix, yielding updates that follow the steepest descent direction under a Riemannian metric. To preserve performance on previously learned tasks, ONG projects these natural gradients onto the orthogonal complement of prior task gradients. We provide a theoretical justification for this procedure, introduce the ONG algorithm, and benchmark its performance on the Permuted and Rotated MNIST datasets. All code for our experiments/reproducibility can be found at this https URL.
Abstract (translated)
正交梯度下降（Orthogonal gradient descent）已成为连续学习任务中的一种强大方法。然而，其欧几里得投影忽视了由神经网络参数化分布空间所具有的底层信息几何结构，这可能导致在学习任务中的次优收敛。为解决这一问题，我们将正交梯度与自然梯度的概念相结合，并提出了ONG（Orthogonal Natural Gradient Descent）。ONG通过高效地使用EKFAC近似逆费雪信息矩阵来预处理每个新任务的梯度，从而生成遵循黎曼度量下最陡下降方向的更新。为了保持在之前学习的任务上的性能，ONG将这些自然梯度投影到先前任务梯度正交补集上。我们为这一过程提供了理论依据，并引入了ONG算法，在Permuted和Rotated MNIST数据集上对其进行了基准测试以展示其效果。 所有实验代码及重现性所需的内容都可以在以下网址找到：[此处插入具体URL]。
URL
https://arxiv.org/abs/2508.17169
PDF
-
Enhanced NIRMAL Optimizer With Damped Nesterov Acceleration: A Comparative Analysis
2025-08-22 17:16:06Nirmal Gaud, Prasad Krishna Murthy, Mostaque Md. Morshedur Hassan, Abhijit Ganguly, Vinay Mali, Ms Lalita Bhagwat Randive, Abhaypratap SinghAbstract
This study introduces the Enhanced NIRMAL (Novel Integrated Robust Multi-Adaptation Learning with Damped Nesterov Acceleration) optimizer, an improved version of the original NIRMAL optimizer. By incorporating an $(\alpha, r)$-damped Nesterov acceleration mechanism, Enhanced NIRMAL improves convergence stability while retaining chess-inspired strategies of gradient descent, momentum, stochastic perturbations, adaptive learning rates, and non-linear transformations. We evaluate Enhanced NIRMAL against Adam, SGD with Momentum, Nesterov, and the original NIRMAL on four benchmark image classification datasets: MNIST, FashionMNIST, CIFAR-10, and CIFAR-100, using tailored convolutional neural network (CNN) architectures. Enhanced NIRMAL achieves a test accuracy of 46.06\% and the lowest test loss (1.960435) on CIFAR-100, surpassing the original NIRMAL (44.34\% accuracy) and closely rivaling SGD with Momentum (46.43\% accuracy). These results underscore Enhanced NIRMAL's superior generalization and stability, particularly on complex datasets.
Abstract (translated)
这项研究介绍了增强型NIRMAL（集成鲁棒多适应学习与阻尼Nesterov加速）优化器，它是原始NIRMAL优化器的一个改进版本。通过引入$(\alpha, r)$-阻尼的Nesterov加速机制，Enhanced NIRMAL在保持棋盘启发策略的同时提高了收敛稳定性，这些策略包括梯度下降、动量、随机扰动、自适应学习率和非线性变换。我们在四个基准图像分类数据集（MNIST、FashionMNIST、CIFAR-10 和 CIFAR-100）上使用定制的卷积神经网络（CNN）架构对Enhanced NIRMAL进行了评估，并将其与Adam、带有动量的SGD、Nesterov以及原始NIRMAL进行对比。在CIFAR-100数据集上，Enhanced NIRMAL达到了46.06%的测试准确率和最低的测试损失（1.960435），超过了原版NIRMAL（44.34%准确率）并且与带有动量的SGD（46.43%准确率）非常接近。这些结果强调了Enhanced NIRMAL在复杂数据集上的优越泛化能力和稳定性。
URL
https://arxiv.org/abs/2508.16550
PDF
-
On Task Vectors and Gradients
2025-08-22 04:16:42Luca Zhou, Daniele Solombrino, Donato Crisostomi, Maria Sofia Bucarelli, Giuseppe Alessio D'Inverno, Fabrizio Silvestri, Emanuele Rodol\`aAbstract
Task arithmetic has emerged as a simple yet powerful technique for model merging, enabling the combination of multiple finetuned models into one. Despite its empirical success, a clear theoretical explanation of why and when it works is lacking. This paper provides a rigorous theoretical foundation for task arithmetic by establishing a connection between task vectors and gradients of the task losses. We show that under standard gradient descent, a task vector generated from one epoch of finetuning is exactly equivalent to the negative gradient of the loss, scaled by the learning rate. For the practical multi-epoch setting, we prove that this equivalence holds approximately, with a second-order error term that we explicitly bound for feed-forward networks. Our empirical analysis across seven vision benchmarks corroborates our theory, demonstrating that the first-epoch gradient dominates the finetuning trajectory in both norm and direction. A key implication is that merging models finetuned for only a single epoch often yields performance comparable to merging fully converged models. These findings reframe task arithmetic as a form of approximate multitask learning, providing a clear rationale for its effectiveness and highlighting the critical role of early training dynamics in model merging.
Abstract (translated)
任务算术作为一种简单而强大的模型合并技术已经浮现，它能够将多个微调后的模型合成为一个。尽管在实践中取得了成功，但关于该方法为何以及何时有效的确切理论解释却一直缺失。本文为任务算术提供了一个严格的理论基础，通过建立任务向量与任务损失梯度之间的联系来实现这一点。我们展示了，在标准的梯度下降过程中，从一个微调周期生成的任务向量等同于负梯度乘以学习率。对于实际中的多轮次设置，我们证明这种等价性近似成立，并且对前馈网络中的二阶误差项进行了明确界定。 我们的实证分析跨越了七个视觉基准测试集，验证了这一理论，表明第一个微调周期的梯度在范数和方向上主导着后续的微调过程。一个关键结论是，合并仅经过单个周期微调的模型通常可以取得与完全收敛后的模型相当的性能。这些发现将任务算术重新定义为一种近似的多任务学习形式，并清晰地解释了其有效性，同时也强调了早期训练动态在模型合并中的关键作用。
URL
https://arxiv.org/abs/2508.16082
PDF
-
Deep Equilibrium Convolutional Sparse Coding for Hyperspectral Image Denoising
2025-08-21 13:35:11Jin Ye, Jingran Wang, Fengchao Xiong, Jingzhou Chen, Yuntao QianAbstract
Hyperspectral images (HSIs) play a crucial role in remote sensing but are often degraded by complex noise patterns. Ensuring the physical property of the denoised HSIs is vital for robust HSI denoising, giving the rise of deep unfolding-based methods. However, these methods map the optimization of a physical model to a learnable network with a predefined depth, which lacks convergence guarantees. In contrast, Deep Equilibrium (DEQ) models treat the hidden layers of deep networks as the solution to a fixed-point problem and models them as infinite-depth networks, naturally consistent with the optimization. Under the framework of DEQ, we propose a Deep Equilibrium Convolutional Sparse Coding (DECSC) framework that unifies local spatial-spectral correlations, nonlocal spatial self-similarities, and global spatial consistency for robust HSI denoising. Within the convolutional sparse coding (CSC) framework, we enforce shared 2D convolutional sparse representation to ensure global spatial consistency across bands, while unshared 3D convolutional sparse representation captures local spatial-spectral details. To further exploit nonlocal self-similarities, a transformer block is embedded after the 2D CSC. Additionally, a detail enhancement module is integrated with the 3D CSC to promote image detail preservation. We formulate the proximal gradient descent of the CSC model as a fixed-point problem and transform the iterative updates into a learnable network architecture within the framework of DEQ. Experimental results demonstrate that our DECSC method achieves superior denoising performance compared to state-of-the-art methods.
Abstract (translated)
高光谱图像（HSI）在遥感领域中扮演着重要角色，但通常会受到复杂噪声模式的影响。确保去噪后的HSI保持其物理特性对于稳健的HSI去噪至关重要，这促使了基于深度展开的方法的发展。然而，这些方法将一个物理模型的优化映射到具有预定义深度的学习网络上，缺乏收敛保证。相比之下，Deep Equilibrium (DEQ) 模型将深层网络中的隐藏层视为固定点问题的解，并将其建模为无限深度的网络，自然地与优化保持一致。 在DEQ框架下，我们提出了一种名为Deep Equilibrium Convolutional Sparse Coding（DECSC）的框架，该框架统一了局部空间-光谱相关性、非局部的空间自相似性和全局的空间一致性，以实现稳健的HSI去噪。在卷积稀疏编码(CSC)框架内，我们在所有频带之间强制执行共享2D卷积稀疏表示，以确保跨频带的全球空间一致性，同时利用非共享3D卷积稀疏表示来捕捉局部空间-光谱细节。为了进一步挖掘非本地自相似性，在2D CSC之后嵌入了一个变压器块。此外，还集成了一个细节增强模块，与3D CSC结合使用以促进图像细节保留。 我们把CSC模型的近似梯度下降公式化为固定点问题，并将其迭代更新转化为DEQ框架内的可学习网络架构。实验结果表明，我们的DECSC方法在去噪性能上优于现有最先进的方法。
URL
https://arxiv.org/abs/2508.15553
PDF
-
Hybrid Least Squares/Gradient Descent Methods for DeepONets
2025-08-21 09:34:06Jun Choi, Chang-Ock Lee, Minam MoonAbstract
We propose an efficient hybrid least squares/gradient descent method to accelerate DeepONet training. Since the output of DeepONet can be viewed as linear with respect to the last layer parameters of the branch network, these parameters can be optimized using a least squares (LS) solve, and the remaining hidden layer parameters are updated by means of gradient descent form. However, building the LS system for all possible combinations of branch and trunk inputs yields a prohibitively large linear problem that is infeasible to solve directly. To address this issue, our method decomposes the large LS system into two smaller, more manageable subproblems $\unicode{x2014}$ one for the branch network and one for the trunk network $\unicode{x2014}$ and solves them separately. This method is generalized to a broader type of $L^2$ loss with a regularization term for the last layer parameters, including the case of unsupervised learning with physics-informed loss.
Abstract (translated)
我们提出了一种高效的混合最小二乘法/梯度下降方法，以加速DeepONet的训练过程。由于DeepONet的输出可以被视为与分支网络最后一层参数呈线性关系，因此这些参数可以通过最小二乘（LS）求解进行优化，而其余隐藏层参数则通过梯度下降方式进行更新。然而，为所有可能的分支和主干输入组合构建LS系统会产生一个规模过大且难以直接求解的线性问题。为了应对这一挑战，我们的方法将大型LS系统分解为两个较小、更易管理的子问题——一个是针对分支网络的问题，另一个是针对主干网络的问题，并分别解决这两个子问题。该方法还被推广到一种更为广泛的$L^2$损失类型，包括最后一层参数的正则化项，在无监督学习中涉及物理信息损失的情况也适用。
URL
https://arxiv.org/abs/2508.15394
PDF
-
Twin-Boot: Uncertainty-Aware Optimization via Online Two-Sample Bootstrapping
2025-08-20 19:20:38Carlos Stein BritoAbstract
Standard gradient descent methods yield point estimates with no measure of confidence. This limitation is acute in overparameterized and low-data regimes, where models have many parameters relative to available data and can easily overfit. Bootstrapping is a classical statistical framework for uncertainty estimation based on resampling, but naively applying it to deep learning is impractical: it requires training many replicas, produces post-hoc estimates that cannot guide learning, and implicitly assumes comparable optima across runs - an assumption that fails in non-convex landscapes. We introduce Twin-Bootstrap Gradient Descent (Twin-Boot), a resampling-based training procedure that integrates uncertainty estimation into optimization. Two identical models are trained in parallel on independent bootstrap samples, and a periodic mean-reset keeps both trajectories in the same basin so that their divergence reflects local (within-basin) uncertainty. During training, we use this estimate to sample weights in an adaptive, data-driven way, providing regularization that favors flatter solutions. In deep neural networks and complex high-dimensional inverse problems, the approach improves calibration and generalization and yields interpretable uncertainty maps.
Abstract (translated)
标准梯度下降方法只能提供点估计，而没有置信度的衡量。这种限制在过度参数化和低数据量的情况下尤为明显，此时模型相对于可用的数据拥有大量参数，并且很容易过拟合。自助法（Bootstrapping）是一种基于重采样的经典统计框架，用于不确定性估计，但直接将其应用于深度学习是不切实际的：它需要训练许多复制模型，产生事后估计无法指导学习过程，而且隐含假设各次运行之间的最优解具有可比性——这一假设在非凸景观中往往不成立。我们引入了双自助梯度下降法（Twin-Bootstrap Gradient Descent, Twin-Boot），这是一种基于重采样的训练程序，它将不确定性估计融入到优化过程中。两个相同的模型并行训练，在独立的自助样本上进行，并通过周期性的均值重置来保持两者在同一个局部最优区域内，从而使它们之间的差异反映该区域内的局部（局域）不确定性。在训练过程中，我们利用这种估计以一种自适应和数据驱动的方式采样权重，提供了一种倾向于更平坦解的正则化方法。这种方法在深度神经网络以及复杂的高维逆问题中改善了校准性和泛化能力，并提供了可解释的不确定性图谱。
URL
https://arxiv.org/abs/2508.15019
PDF
-
Fisher-Orthogonal Projection Methods for Natural Gradient Descent with Large Batches
Abstract
Modern GPUs are equipped with large amounts of high-bandwidth memory, enabling them to support mini-batch sizes of up to tens of thousands of training samples. However, most existing optimizers struggle to perform effectively at such a large batch size. As batch size increases, gradient noise decreases due to averaging over many samples, limiting the ability of first-order methods to escape sharp or suboptimal minima and reach the global minimum. Meanwhile, second-order methods like the natural gradient with Kronecker-Factored Approximate Curvature (KFAC) often require excessively high damping to remain stable at large batch sizes. This high damping effectively washes out the curvature information that gives these methods their advantage, reducing their performance to that of simple gradient descent. In this paper, we introduce Fisher-Orthogonal Projection (FOP), a novel technique that restores the effectiveness of the second-order method at very large batch sizes, enabling scalable training with improved generalization and faster convergence. FOP constructs a variance-aware update direction by leveraging gradients from two sub-batches, enhancing the average gradient with a component of the gradient difference that is orthogonal to the average under the Fisher-metric.
Abstract (translated)
现代GPU配备了大量高带宽内存，能够支持多达数万个训练样本的小批量处理。然而，大多数现有的优化器在如此大的批量大小下难以有效运行。随着批处理量的增加，由于大量样本平均效应，梯度噪声减少，这限制了一阶方法（如随机梯度下降）从尖锐或次优极小值中逃脱并达到全局最小值的能力。同时，像自然梯度结合Kronecker-Factored Approximate Curvature (KFAC)这样的二阶方法，在处理大规模批量时往往需要过高的阻尼以保持稳定。这种高阻尼会抹去这些方法所依赖的曲率信息优势，使它们的表现与简单的梯度下降相差无几。 在本文中，我们介绍了Fisher-Orthogonal Projection（FOP），这是一种新颖的技术，能够在非常大的批处理大小下恢复二阶方法的有效性，从而实现具有改进泛化能力和更快收敛速度的大规模训练。FOP通过利用两个子批量的梯度来构造一个方差感知更新方向，并在Fisher度量下的平均梯度中增加了一个正交于平均值的梯度差异分量。
URL
https://arxiv.org/abs/2508.13898
PDF
