Gradient_Descent
-
Compiling Deterministic Structure into SLM Harnesses
2026-04-19 14:04:29Zan Kai Chong, Hiroyuki Ohsaki, Bryan NgAbstract
Enterprise deployment of small language models (SLMs) is constrained by epistemic asymmetry: SLMs cannot self-correct reasoning errors, while frontier LLMs are prohibitively costly and face data sovereignty limits for high-volume use. We propose Semantic Gradient Descent (SGDe), a teacher-student framework that compiles agentic workflows into discrete execution plans comprising DAG topologies, system prompts, and deterministic executable code. The trailing "e" distinguishes SGDe from stochastic gradient descent. SGDe operates in a discrete semantic space where a frontier teacher generates natural-language critiques acting as directional gradients to iteratively refine the SLM's workflow artefacts. We formalise SGDe within a PAC learning framework, establishing sample-complexity bounds that enable convergence with as few as three training examples on targeted synthetic tasks by leveraging the teacher as a statistical prior. On a GSM-Hard-derived test set built via adversarial synthesis, compiled workflows reach 91.3% accuracy at m=5 and 99.3% at m=3 within the small-m regime motivated by Corollary 1, a +26.3% to +34.3% absolute improvement over state-of-the-art prompt optimisers. In the emerging paradigm of harness engineering, SGDe treats placement of deterministic code (which subtasks to delegate to a Python runtime versus retain as LLM calls) as a trace-driven, per-node optimisation target, generalising the whole-problem offloading of PAL and PoT. The teacher compiles two complementary deterministic structures: capability offloading, which delegates subtasks to Python when the SLM cannot execute them reliably, and structural consensus, which wraps variance-limited reasoning steps in fan-out/fan-in subgraphs aggregated by deterministic voting.
Abstract (translated)
URL
https://arxiv.org/abs/2604.17450
PDF
-
Instant Colorization of Gaussian Splats
2026-04-18 21:56:52Daniel Lieber, Alexander Mock, Nils WandelAbstract
Gaussian Splatting has recently become one of the most popular frameworks for photorealistic 3D scene reconstruction and rendering. While current rasterizers allow for efficient mappings of 3D Gaussian splats onto 2D camera views, this work focuses on mapping 2D image information (e.g. color, neural features or segmentation masks) efficiently back onto an existing scene of Gaussian splats. This 'opposite' direction enables applications ranging from scene relighting and stylization to 3D semantic segmentation, but also introduces challenges, such as view-dependent colorization and occlusion handling. Our approach tackles these challenges using the normal equation to solve a visibility-weighted least squares problem for every Gaussian and can be implemented efficiently with existing differentiable rasterizers. We demonstrate the effectiveness of our approach on scene relighting, feature enrichment and 3D semantic segmentation tasks, achieving up to an order of magnitude speedup compared to gradient descent-based baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2604.17155
PDF
-
Stein Variational Black-Box Combinatorial Optimization
2026-04-17 08:40:17Thomas Landais, Olivier Goudet, Adrien Go\"effon, Fr\'ed\'eric Saubion, Sylvain LamprierAbstract
Combinatorial black-box optimization in high-dimensional settings demands a careful trade-off between exploiting promising regions of the search space and preserving sufficient exploration to identify multiple optima. Although Estimation-of-Distribution Algorithms (EDAs) provide a powerful model-based framework, they often concentrate on a single region of interest, which may result in premature convergence when facing complex or multimodal objective landscapes. In this work, we incorporate the Stein operator to introduce a repulsive mechanism among particles in the parameter space, thereby encouraging the population to disperse and jointly explore several modes of the fitness landscape. Empirical evaluations across diverse benchmark problems show that the proposed method achieves performance competitive with, and in several cases superior to, leading state-of-the-art approaches, particularly on large-scale instances. These findings highlight the potential of Stein variational gradient descent as a promising direction for addressing large, computationally expensive, discrete black-box optimization problems.
Abstract (translated)
URL
https://arxiv.org/abs/2604.15837
PDF
-
LP$^{2}$DH: A Locality-Preserving Pixel-Difference Hashing Framework for Dynamic Texture Recognition
2026-04-17 05:27:13Ruxin Ding, Jianfeng Ren, Heng Yu, Jiawei Li, Xudong JiangAbstract
Spatiotemporal Local Binary Pattern (STLBP) is a widely used dynamic texture descriptor, but it suffers from extremely high dimensionality. To tackle this, STLBP features are often extracted on three orthogonal planes, which sacrifice inter-plane correlation. In this work, we propose a Locality-Preserving Pixel-Difference Hashing (LP$^{2}$DH) framework that jointly encodes pixel differences in the full spatiotemporal neighbourhood. LP$^{2}$DH transforms Pixel-Difference Vectors (PDVs) into compact binary codes with maximal discriminative power. Furthermore, we incorporate a locality-preserving embedding to maintain the PDVs' local structure before and after hashing. Then, a curvilinear search strategy is utilized to jointly optimize the hashing matrix and binary codes via gradient descent on the Stiefel manifold. After hashing, dictionary learning is applied to encode the binary vectors into codewords, and the resulting histogram is utilized as the final feature representation. The proposed LP$^{2}$DH achieves state-of-the-art performance on three major dynamic texture recognition benchmarks: 99.80% against DT-GoogleNet's 98.93% on UCLA, 98.52% against HoGF$^{3D}$'s 97.63% on DynTex++, and 96.19% compared to STS's 95.00% on YUPENN. The source code is available at: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.15707
PDF
-
VoodooNet: Achieving Analytic Ground States via High-Dimensional Random Projections
Abstract
We present VoodooNet, a non-iterative neural architecture that replaces the stochastic gradient descent (SGD) paradigm with a closed-form analytic solution via Galactic Expansion. By projecting input manifolds into a high-dimensional, high-entropy "Galactic" space ($d \gg 784$), we demonstrate that complex features can be untangled without the thermodynamic cost of backpropagation. Utilizing the Moore-Penrose pseudoinverse to solve for the output layer in a single step, VoodooNet achieves a classification accuracy of \textbf{98.10\% on MNIST} and \textbf{86.63\% on Fashion-MNIST}. Notably, our results on Fashion-MNIST surpass a 10-epoch SGD baseline (84.41\%) while reducing the training time by orders of magnitude. We observe a near-logarithmic scaling law between dimensionality and accuracy, suggesting that performance is a function of "Galactic" volume rather than iterative refinement. This "Magic Hat" approach offers a new frontier for real-time Edge AI, where the traditional training phase is bypassed in favor of instantaneous manifold discovery.
Abstract (translated)
URL
https://arxiv.org/abs/2604.15613
PDF
-
Natural gradient descent with momentum
Abstract
We consider the problem of approximating a function by an element of a nonlinear manifold which admits a differentiable parametrization, typical examples being neural networks with differentiable activation functions or tensor networks. Natural gradient descent (NGD) for the optimization of a loss function can be seen as a preconditioned gradient descent where updates in the parameter space are driven by a functional perspective. In a spirit similar to Newton's method, a NGD step uses, instead of the Hessian, the Gram matrix of the generating system of the tangent space to the approximation manifold at the current iterate, with respect to a suitable metric. This corresponds to a locally optimal update in function space, following a projected gradient onto the tangent space to the manifold. Still, both gradient and natural gradient descent methods get stuck in local minima. Furthermore, when the model class is a nonlinear manifold or the loss function is not ideally conditioned (e.g., the KL-divergence for density estimation, or a norm of the residual of a partial differential equation in physics informed learning), even the natural gradient might yield non-optimal directions at each step. This work introduces a natural version of classical inertial dynamic methods like Heavy-Ball or Nesterov and show how it can improve the learning process when working with nonlinear model classes.
Abstract (translated)
URL
https://arxiv.org/abs/2604.15554
PDF
-
Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations
2026-04-14 17:01:36Tong Zhang, Jiangning Zhang, Zhucun Xue, Juntao Jiang, Yicheng Xu, Chengming Xu, Teng Hu, Xingyu Xie, Xiaobin Hu, Yabiao Wang, Yong Liu, Shuicheng YanAbstract
Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-order gradient descent methods, epitomized by stochastic gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines. However, large-scale model training, stringent differential privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency. To mitigate these bottlenecks, researchers explore second-order optimization techniques to surpass first-order performance ceilings, while zeroth-order methods reemerge to alleviate memory constraints inherent to large-scale training. Despite this proliferation of methodologies, the field lacks a cohesive framework that unifies underlying principles and delineates application scenarios for these disparate approaches. In this work, we retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstream optimizers across diverse model architectures and training scenarios. We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research. By synthesizing theoretical insights with extensive empirical evidence, we provide actionable guidance for designing next-generation highly efficient, robust, and trustworthy optimization methods. The code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.12968
PDF
-
Elastic Net Regularization and Gabor Dictionary for Classification of Heart Sound Signals using Deep Learning
2026-04-14 09:09:30Mahmoud Fakhry, Ascensi\'on Gallardo-Antol\'inAbstract
In this article, we propose the optimization of the resolution of time-frequency atoms and the regularization of fitting models to obtain better representations of heart sound signals. This is done by evaluating the classification performance of deep learning (DL) networks in discriminating five heart valvular conditions based on a new class of time-frequency feature matrices derived from the fitting models. We inspect several combinations of resolution and regularization, and the optimal one is that provides the highest performance. To this end, a fitting model is obtained based on a heart sound signal and an overcomplete dictionary of Gabor atoms using elastic net regularization of linear models. We consider two different DL architectures, the first mainly consisting of a 1D convolutional neural network (CNN) layer and a long short-term memory (LSTM) layer, while the second is composed of 1D and 2D CNN layers followed by an LSTM layer. The networks are trained with two algorithms, namely stochastic gradient descent with momentum (SGDM) and adaptive moment (ADAM). Extensive experimentation has been conducted using a database containing heart sound signals of five heart valvular conditions. The best classification accuracy of $98.95\%$ is achieved with the second architecture when trained with ADAM and feature matrices derived from optimal models obtained with a Gabor dictionary consisting of atoms with high-time low-frequency resolution and imposing sparsity on the models.
Abstract (translated)
URL
https://arxiv.org/abs/2604.12483
PDF
-
Analytical Modeling and Correction of Distance Error in Homography-Based Ground-Plane Mapping
2026-04-12 20:33:09Mateusz Szulc, Marcin IwanowskiAbstract
Accurate distance estimation from monocular cameras is essential for intelligent monitoring systems. In many deployments, image coordinates are mapped to ground positions using planar homographies initialized by manual selection of corresponding regions. Small inaccuracies in this initialization propagate into systematic distance distortions. This paper derives an explicit relationship between homography perturbations and the resulting distance error, showing that the error grows approximately quadratically with the true distance from the camera. Based on this model, two simple correction strategies are evaluated: regression-based estimation of the quadratic error function and direct optimization of the homography via coordinate-based gradient descent. A large-scale simulation study with more than 19 million test samples demonstrates that regression achieves higher peak accuracy when the model is reliably fitted, whereas gradient descent provides greater robustness against poor initial calibration. This suggests that improving geometric calibration may yield greater performance gains than increasing model complexity in many practical systems.
Abstract (translated)
URL
https://arxiv.org/abs/2604.10805
PDF
-
The Non-Optimality of Scientific Knowledge: Path Dependence, Lock-In, and The Local Minimum Trap
Abstract
Science is widely regarded as humanity's most reliable method for uncovering truths about the natural world. Yet the \emph{trajectory} of scientific discovery is rarely examined as an optimization problem in its own right. This paper argues that the body of scientific knowledge, at any given historical moment, represents a \emph{local optimum} rather than a global one--that the frameworks, formalisms, and paradigms through which we understand nature are substantially shaped by historical contingency, cognitive path dependence, and institutional lock-in. Drawing an analogy to gradient descent in machine learning, we propose that science follows the steepest local gradient of tractability, empirical accessibility, and institutional reward, and in doing so may bypass fundamentally superior descriptions of nature. We develop this thesis through detailed case studies spanning mathematics, physics, chemistry, biology, neuroscience, and statistical methodology. We identify three interlocking mechanisms of lock-in--cognitive, formal, and institutional--and argue that recognizing these mechanisms is a prerequisite for designing meta-scientific strategies capable of escaping local optima. We conclude by proposing concrete interventions and discussing the epistemological implications of our thesis for the philosophy of science.
Abstract (translated)
URL
https://arxiv.org/abs/2604.11828
PDF
-
EGLOCE: Training-Free Energy-Guided Latent Optimization for Concept Erasure
2026-04-10 15:19:02Junyeong Ahn, Seojin Yoon, Sungyong BaikAbstract
As text-to-image diffusion models grow increasingly prevalent, the ability to remove specific concepts-mostly explicit content and many copyrighted characters or styles-has become essential for safety and compliance. Existing unlearning approaches often require costly re-training, modify parameters at the cost of degradation of unrelated concept fidelity, or depend on indirect inference-time adjustment that compromise the effectiveness of concept erasure. Inspired by the success of energy-guided sampling for preservation of the condition of diffusion models, we introduce Energy-Guided Latent Optimization for Concept Erasure (EGLOCE), a training-free approach that removes unwanted concepts by re-directing noisy latent during inference. Our method employs a dual-objective framework: a repulsion energy that steers generation away from target concepts via gradient descent in latent space, and a retention energy that preserves semantic alignment to the original prompt. Combined with previous approaches that either require erroneous modified model weights or provide weak inference-time guidance, EGLOCE operates entirely at inference and enhances erasure performance, enabling plug-and-play integration. Extensive experiments demonstrate that EGLOCE improves concept removal while maintaining image quality and prompt alignment across baselines, even with adversarial attacks. To the best of our knowledge, our work is the first to establish a new paradigm for safe and controllable image generation through dual energy-based guidance during sampling.
Abstract (translated)
URL
https://arxiv.org/abs/2604.09405
PDF
-
Multi-Frequency Local Plasticity for Visual Representation Learning
2026-04-09 18:30:47Mehdi Fatan Serj, C. Alejandro Parraga, Xavier OtazuAbstract
We study how far structured architectural bias can compensate for the absence of end-to-end gradient-based representation learning in visual recognition. Building on the VisNet tradition, we introduce a modular hierarchical framework combining: (i) fixed multi-frequency Gabor decomposition into F=7 parallel streams; (ii) within-stream competitive learning with Hebbian and Oja updates and anti-Hebbian decorrelation; (iii) an associative memory module inspired by modern Hopfield retrieval; and (iv) iterative top-down modulation using local prediction and reconstruction signals. Representational layers are trained without end-to-end backpropagation through the full hierarchy; only the final linear readout and top-down projection matrices are optimized by gradient descent. We therefore interpret the model as a hybrid system that is predominantly locally trained but includes a small number of gradient-trained parameters. On CIFAR-10, the full model reaches 80.1% +/- 0.3% top-1 accuracy, linear probe), compared with 71.0% for a Hebbian-only baseline and 83.4% for a gradient-trained model on the same fixed Gabor basis. On CIFAR-100, performance is 54.8%. Factorial analysis indicates that multi-frequency streams, associative memory, and top-down feedback contribute largely additively, with a significant Streams x TopDown interaction (p=0.02). These results suggest that carefully chosen architectural priors can recover a substantial fraction of the performance typically associated with global gradient training, while leaving a measurable residual gap. Experiments are limited to CIFAR-10/100.
Abstract (translated)
URL
https://arxiv.org/abs/2604.09734
PDF
-
Soft-Quantum Algorithms
2026-04-07 23:30:40Basil Kyriacou, Mo Kordzanganeh, Maniraman Periyasamy, Alexey MelnikovAbstract
Quantum operations on pure states can be fully represented by unitary matrices. Variational quantum circuits, also known as quantum neural networks, embed data and trainable parameters into gate-based operations and optimize the parameters via gradient descent. The high cost of training and low fidelity of current quantum devices, however, restricts much of quantum machine learning to classical simulation. For few-qubit problems with large datasets, training the matrix elements directly, as is done with weight matrices in classical neural networks, can be faster than decomposing data and parameters into gates. We propose a method that trains matrices directly while maintaining unitarity through a single regularization term added to the loss function. A second training step, circuit alignment, then recovers a gate-based architecture from the resulting soft-unitary. On a five-qubit supervised classification task with 1000 datapoints, this two-step process produces a trained variational circuit in under four minutes, compared to over two hours for direct circuit training, while achieving lower binary cross-entropy loss. In a second experiment, soft-unitaries are embedded in a hybrid quantum-classical network for a reinforcement learning cartpole task, where the hybrid agent outperforms a purely classical baseline of comparable size.
Abstract (translated)
URL
https://arxiv.org/abs/2604.06523
PDF
-
The UNDO Flip-Flop: A Controlled Probe for Reversible Semantic State Management in State Space Model
Abstract
State space models (SSMs) have been shown to possess the theoretical capacity to model both star-free sequential tasks and bounded hierarchical structures Sarrof et al. (2024). However, formal expressivity results do not guarantee that gradient-based optimisation will reliably discover the corresponding solutions. Existing benchmarks probe either monotonic state tracking, as in the standard Flip-Flop task, or structural nesting, as in the Dyck languages, but neither isolates reversible semantic state retrieval. We introduce the UNDO Flip-Flop task to fill this gap. By extending the standard Flip-Flop with an UNDO, the task requires a model to maintain an implicit bounded stack and recover historical states under non-monotonic update sequences. We evaluate one-layer and two-layer Mamba-2 under this framework. Both variants fail to acquire the provably expressible stack-based rollback mechanism, converging instead on a local toggle heuristic that inverts the current state rather than retrieving stored history. Under an adversarial retraction pressure test held within the training length distribution, the two-layer model collapses to 41.10% accuracy, which is below random chance. The results confirm systematic rather than incidental failure. Causal ablation shows that the bottleneck lies in retrieval, not storage. These results draw a clear line between what an architecture can in principle represent and what gradient descent reliably learns, a distinction that theoretical expressivity analyses alone cannot capture.
Abstract (translated)
状态空间模型（SSMs）已被证明在理论上具备对无正则序列任务和有界分层结构进行建模的能力（Sarrof 等人，2024）。然而，理论表达能力并不能保证基于梯度的优化能可靠地发现相应解决方案。现有基准测试要么探测单调状态追踪（如标准 Flip-Flop 任务），要么探测结构嵌套（如 Dyck 语言），但两者均未孤立考察可逆语义状态检索。我们引入 UNDO Flip-Flop 任务以填补这一空白。通过在标准 Flip-Flop 基础上增加 UNDO 操作，该任务要求模型维持隐式有界栈，并在非单调更新序列下恢复历史状态。我们在此框架下评估了一层与两层 Mamba-2 模型。两种变体均未能习得理论上可表达的基于栈的回滚机制，反而收敛于一种仅反转当前状态而非检索存储历史的局部切换启发式策略。在训练长度分布内进行的对抗性回撤压力测试中，两层模型准确率降至 41.10%，低于随机概率。这些结果证实了系统性而非偶然性的失败。因果消融分析表明瓶颈在于检索而非存储。这些结果清晰划清了架构在原则上能表示什么与梯度下降能可靠学习什么之间的界限——这是单纯理论表达能力分析无法捕捉的区分。
URL
https://arxiv.org/abs/2604.05923
PDF
-
WRF4CIR: Weight-Regularized Fine-Tuning Network for Composed Image Retrieval
2026-04-07 08:23:39Yizhuo Xu, Chaojian Yu, Yuanjie Shao, Tongliang Liu, Qinmu Peng, Xinge YouAbstract
Composed Image Retrieval (CIR) task aims to retrieve target images based on reference images and modification texts. Current CIR methods primarily rely on fine-tuning vision-language pre-trained models. However, we find that these approaches commonly suffer from severe overfitting, posing challenges for CIR with limited triplet data. To better understand this issue, we present a systematic study of overfitting in VLP-based CIR, revealing a significant and previously overlooked generalization gap across different models and datasets. Motivated by these findings, we introduce WRF4CIR, a Weight-Regularized Fine-tuning network for CIR. Specifically, during the fine-tuning process, we apply adversarial perturbations to the model weights for regularization, where these perturbations are generated in the opposite direction of gradient descent. Intuitively, WRF4CIR increases the difficulty of fitting the training data, which helps mitigate overfitting in CIR under limited triplet supervision. Extensive experiments on benchmark datasets demonstrate that WRF4CIR significantly narrows the generalization gap and achieves substantial improvements over existing methods.
Abstract (translated)
组合图像检索（CIR）任务旨在基于参考图像和修改文本检索目标图像。当前CIR方法主要依赖对视觉-语言预训练模型进行微调。然而，我们发现这些方法普遍存在严重的过拟合问题，给有限三元组数据下的CIR带来挑战。为深入理解此问题，我们对基于VLP的CIR中的过拟合进行了系统性研究，揭示了一个显著且此前被忽视的跨模型与数据集的泛化差距。受此发现启发，我们提出了WRF4CIR，一种用于CIR的权重正则化微调网络。具体而言，在微调过程中，我们通过对模型权重施加对抗性扰动进行正则化，这些扰动在梯度下降的反方向生成。直观上，WRF4CIR增加了拟合训练数据的难度，从而有助于缓解有限三元组监督下CIR的过拟合问题。在基准数据集上的大量实验表明，WRF4CIR显著缩小了泛化差距，并相较于现有方法取得了实质性提升。
URL
https://arxiv.org/abs/2604.05583
PDF
-
Muon Dynamics as a Spectral Wasserstein Flow
Abstract
Gradient normalization is central in deep-learning optimization because it stabilizes training and reduces sensitivity to scale. For deep architectures, parameters are naturally grouped into matrices or blocks, so spectral normalizations are often more faithful than coordinatewise Euclidean ones; Muon is the main motivating example of this paper. More broadly, we study a family of spectral normalization rules, ranging from ordinary gradient descent to Muon and intermediate Schatten-type schemes, in a mean-field regime where parameters are modeled by probability measures. We introduce a family of Spectral Wasserstein distances indexed by a norm gamma on positive semidefinite matrices. The trace norm recovers the classical quadratic Wasserstein distance, the operator norm recovers the Muon geometry, and intermediate Schatten norms interpolate between them. We develop the static Kantorovich formulation, prove comparison bounds with W2, derive a max-min representation, and obtain a conditional Brenier theorem. For Gaussian marginals, the problem reduces to a constrained optimization on covariance matrices, extending the Bures formula and yielding a closed form for commuting covariances in the Schatten family. For monotone norms, including all Schatten cases, we prove the equivalence between the static and dynamic Benamou-Brenier formulations, deduce that the resulting transport cost is a genuine metric equivalent to W2 in fixed dimension, and show that the induced Gaussian covariance cost is also a metric. We then interpret the associated normalized continuity equation as a Spectral Wasserstein gradient flow, identify its exact finite-particle counterpart as a normalized matrix flow, obtain first geodesic-convexity results, and show how positively homogeneous mean-field models induce a spectral unbalanced transport on the sphere.
Abstract (translated)
梯度归一化在深度学习优化中至关重要，因为它能稳定训练过程并降低对尺度变化的敏感性。对于深度架构，参数自然按矩阵或分块分组，因此谱归一化通常比逐坐标欧几里得归一化更忠实；本文的主要动机案例Muon即体现了这一特性。更广泛地，我们在参数被建模为概率测度的平均场框架下，研究了一类谱归一化规则，涵盖从普通梯度下降到Muon及中间施瓦茨型方案的连续谱系。我们引入了一族以正半定矩阵上的范数γ为指标的谱Wasserstein距离。迹范数恢复经典二次Wasserstein距离，算子范数恢复Muon几何，中间施瓦茨范数则实现两者间的插值。我们发展了静态康托罗维奇公式，证明了与W₂的比较界，推导了极大极小表示，并获得了条件布伦耶定理。对于高斯边缘分布，问题简化为协方差矩阵上的约束优化，推广了伯斯公式，并给出了施瓦茨族中对易协方差的闭式解。对于单调范数（包括所有施瓦茨情形），我们证明了静态与动态贝纳穆-布伦耶公式的等价性，推导出所得输运成本是与W₂在固定维度下等价的真度量，并表明诱导的高斯协方差成本也是度量。随后，我们将相关的归一化连续性方程解释为谱Wasserstein梯度流，将其精确有限粒子对应识别为归一化矩阵流，获得了首个测地凸性结果，并展示了正齐次平均场模型如何在球面上诱导谱非平衡输运。
URL
https://arxiv.org/abs/2604.04891
PDF
-
Adversarial Robustness Analysis of Cloud-Assisted Autonomous Driving Systems
2026-04-06 01:54:41Maher Al Islam, Amr S. El-WakeelAbstract
Autonomous vehicles increasingly rely on deep learning-based perception and control, which impose substantial computational demands. Cloud-assisted architectures offload these functions to remote servers, enabling enhanced perception and coordinated decision-making through the Internet of Vehicles (IoV). However, this paradigm introduces cross-layer vulnerabilities, where adversarial manipulation of perception models and network impairments in the vehicle-cloud link can jointly undermine safety-critical autonomy. This paper presents a hardware-in-the-loop IoV testbed that integrates real-time perception, control, and communication to evaluate such vulnerabilities in cloud-assisted autonomous driving. A YOLOv8-based object detector deployed on the cloud is subjected to whitebox adversarial attacks using the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD), while network adversaries induce delay and packet loss in the vehicle-cloud loop. Results show that adversarial perturbations significantly degrade perception performance, with PGD reducing detection precision and recall from 0.73 and 0.68 in the clean baseline to 0.22 and 0.15 at epsilon= 0.04. Network delays of 150-250 ms, corresponding to transient losses of approximately 3-4 frames, and packet loss rates of 0.5-5 % further destabilize closed-loop control, leading to delayed actuation and rule violations. These findings highlight the need for cross-layer resilience in cloud-assisted autonomous driving systems.
Abstract (translated)
URL
https://arxiv.org/abs/2604.04349
PDF
-
Fine-grained Analysis of Stability and Generalization for Stochastic Bilevel Optimization
2026-04-05 12:12:58Xuelin Zhang, Hong Chen, Bin Gu, Tieliang Gong, Feng ZhengAbstract
Stochastic bilevel optimization (SBO) has been integrated into many machine learning paradigms recently, including hyperparameter optimization, meta learning, and reinforcement learning. Along with the wide range of applications, there have been numerous studies on the computational behavior of SBO. However, the generalization guarantees of SBO methods are far less understood from the lens of statistical learning theory. In this paper, we provide a systematic generalization analysis of the first-order gradient-based bilevel optimization methods. Firstly, we establish the quantitative connections between the on-average argument stability and the generalization gap of SBO methods. Then, we derive the upper bounds of on-average argument stability for single-timescale stochastic gradient descent (SGD) and two-timescale SGD, where three settings (nonconvex-nonconvex (NC-NC), convex-convex (C-C), and strongly-convex-strongly-convex (SC-SC)) are considered respectively. Experimental analysis validates our theoretical findings. Compared with the previous algorithmic stability analysis, our results do not require reinitializing the inner-level parameters at each iteration and are applicable to more general objective functions.
Abstract (translated)
随机双层优化（SBO）近期已被整合进多种机器学习范式，包括超参数优化、元学习与强化学习。随着应用范围的不断拓展，学界对SBO的计算行为展开了大量研究。然而，从统计学习理论视角审视，SBO方法的泛化保证机制仍远未被充分理解。本文对基于一阶梯度的双层优化方法进行了系统性泛化分析。首先，我们建立了SBO方法平均论证稳定性与泛化间隙之间的定量关联。随后，我们在非凸-非凸（NC-NC）、凸-凸（C-C）及强凸-强凸（SC-SC）三种设定下，分别推导了单时间尺度随机梯度下降（SGD）与双时间尺度SGD的平均论证稳定性上界。实验分析验证了我们的理论发现。相较于既有算法稳定性分析，本研究结果无需在每轮迭代中重新初始化内层参数，且适用于更广泛的目标函数形式。
URL
https://arxiv.org/abs/2604.04090
PDF
-
An Improved Last-Iterate Convergence Rate for Anchored Gradient Descent Ascent
2026-04-04 16:08:55Anja Surina, Arun Suggala, George Tsoukalas, Anton Kovsharov, Sergey Shirobokov, Francisco J. R. Ruiz, Pushmeet Kohli, Swarat ChaudhuriAbstract
We analyze the last-iterate convergence of the Anchored Gradient Descent Ascent algorithm for smooth convex-concave min-max problems. While previous work established a last-iterate rate of $\mathcal{O}(1/t^{2-2p})$ for the squared gradient norm, where $p \in (1/2, 1)$, it remained an open problem whether the improved exact $\mathcal{O}(1/t)$ rate is achievable. In this work, we resolve this question in the affirmative. This result was discovered autonomously by an AI system capable of writing formal proofs in Lean. The Lean proof can be accessed at this https URL
Abstract (translated)
我们分析了锚定梯度下降上升算法在光滑凸凹极小极大问题中的最后迭代收敛性。先前工作已证明，当 $p \in (1/2, 1)$ 时，该算法关于平方梯度范数的最后迭代收敛率为 $\mathcal{O}(1/t^{2-2p})$，但更优的精确 $\mathcal{O}(1/t)$ 收敛率是否可达仍是一个未解问题。本研究肯定地回答了这一问题。该成果由一个能够用 Lean 语言撰写形式化证明的 AI 系统自主发现。Lean 证明可在此 https 链接获取。
URL
https://arxiv.org/abs/2604.03782
PDF
-
Learning the Signature of Memorization in Autoregressive Language Models
2026-04-03 17:17:51David Ili\'c, Kostadin Cvejoski, David Stanojevi\'c, Evgeny GrigorenkoAbstract
All prior membership inference attacks for fine-tuned language models use hand-crafted heuristics (e.g., loss thresholding, Min-K\%, reference calibration), each bounded by the designer's intuition. We introduce the first transferable learned attack, enabled by the observation that fine-tuning any model on any corpus yields unlimited labeled data, since membership is known by construction. This removes the shadow model bottleneck and brings membership inference into the deep learning era: learning what matters rather than designing it, with generalization through training diversity and scale. We discover that fine-tuning language models produces an invariant signature of memorization detectable across architectural families and data domains. We train a membership inference classifier exclusively on transformer-based models. It transfers zero-shot to Mamba (state-space), RWKV-4 (linear attention), and RecurrentGemma (gated recurrence), achieving 0.963, 0.972, and 0.936 AUC respectively. Each evaluation combines an architecture and dataset never seen during training, yet all three exceed performance on held-out transformers (0.908 AUC). These four families share no computational mechanisms, their only commonality is gradient descent on cross-entropy loss. Even simple likelihood-based methods exhibit strong transfer, confirming the signature exists independently of the detection method. Our method, Learned Transfer MIA (LT-MIA), captures this signal most effectively by reframing membership inference as sequence classification over per-token distributional statistics. On transformers, LT-MIA achieves 2.8$\times$ higher TPR at 0.1\% FPR than the strongest baseline. The method also transfers to code (0.865 AUC) despite training only on natural language texts. Code and trained classifier available at this https URL.
Abstract (translated)
此前针对微调语言模型的所有成员推断攻击均依赖手工设计的启发式规则（例如损失阈值法、Min-K%、参考校准），其性能受设计者直觉的限制。我们提出了首个可迁移的习得攻击，其核心发现是：任何模型在任何语料库上微调时，由于成员身份在构造时已知，因此能产生无限标记数据。这消除了影子模型瓶颈，将成员推断带入深度学习时代——从“设计特征”转向“学习特征”，通过训练多样性和规模实现泛化。我们发现，微调语言模型会产生一个跨架构家族和数据域可检测的恒定记忆签名。我们仅在基于Transformer的模型上训练成员推断分类器，并实现零样本迁移至Mamba（状态空间模型）、RWKV-4（线性注意力）和RecurrentGemma（门控循环网络）， respective AUC分别达到0.963、0.972和0.936。每次评估均结合训练中未见的架构与数据集，但三者性能均超过在保留Transformer上的表现（0.908 AUC）。这四类架构共享无计算机制，其唯一共同点是基于交叉熵损失的梯度下降。即使简单基于似然的方法也展现出强迁移性，证实该签名独立于检测方法存在。我们的方法——习得迁移成员推断（LT-MIA）——通过将成员推断重构为基于每标记分布统计的序列分类任务，最有效地捕获该信号。在Transformer上，LT-MIA在0.1%假阳性率下实现比最强基线高2.8倍的真阳性率。该方法即使仅使用自然语言文本训练，也能迁移至代码领域（0.865 AUC）。代码与训练好的分类器可通过此链接获取。
URL
https://arxiv.org/abs/2604.03199
PDF
