Gradient_Descent
-
Does Flatness imply Generalization for Logistic Loss in Univariate Two-Layer ReLU Network?
Abstract
We consider the problem of generalization of arbitrarily overparameterized two-layer ReLU Neural Networks with univariate input. Recent work showed that under square loss, flat solutions (motivated by flat / stable minima and Edge of Stability phenomenon) provably cannot overfit, but it remains unclear whether the same phenomenon holds for logistic loss. This is a puzzling open problem because existing work on logistic loss shows that gradient descent with increasing step size converges to interpolating solutions (at infinity, for the margin-separable cases). In this paper, we prove that the \emph{flatness implied generalization} is more delicate under logistic loss. On the positive side, we show that flat solutions enjoy near-optimal generalization bounds within a region between the left-most and right-most \emph{uncertain} sets determined by each candidate solution. On the negative side, we show that there exist arbitrarily flat yet overfitting solutions at infinity that are (falsely) certain everywhere, thus certifying that flatness alone is insufficient for generalization in general. We demonstrate the effects predicted by our theory in a well-controlled simulation study.
Abstract (translated)
我们考虑了单变量输入的任意过参数化两层ReLU神经网络的泛化问题。最近的研究表明，在平方损失的情况下，平坦解（受稳定极小值和平稳性边界现象启发）可以被证明不会导致过拟合，但对于对数几率损失而言，这种情况是否仍然成立尚不清楚。这是一个令人困惑的未解决问题，因为现有的关于对数几率损失的工作显示，随着步长增加，梯度下降会收敛到插值解（在边际可分的情况下，在无穷远处）。在这篇论文中，我们证明了在对数几率损失下，“平坦性引起的泛化”更加微妙。从积极的一面来看，我们在由每个候选解确定的最左和最右“不确定集”之间的区域内展示了平坦解几乎具有最优的泛化界。另一方面，我们也表明存在一些解决方案，它们在无穷远处是任意平坦的且会导致过拟合，并且它们在整个范围内被认为是（错误地）确定性的，从而证明了仅仅依靠平坦性不足以保证泛化能力的一般情形。我们在一项严格控制的模拟研究中展示了我们理论预测的影响。
URL
https://arxiv.org/abs/2512.01473
PDF
-
Provable Benefit of Sign Descent: A Minimal Model Under Heavy-Tailed Class Imbalance
2025-11-30 07:21:02Robin Yadav, Shuo Xie, Tianhao Wang, Zhiyuan LiAbstract
Adaptive optimization methods (such as Adam) play a major role in LLM pretraining, significantly outperforming Gradient Descent (GD). Recent studies have proposed new smoothness assumptions on the loss function to explain the advantages of adaptive algorithms with structured preconditioners, e.g., coordinate-wise or layer-wise, and steepest descent methods w.r.t. non-euclidean norms, e.g., $\ell_\infty$ norm or spectral norm, over GD. However, it remains unclear how these smoothness assumptions manifest in language modelling tasks. In this work, we aim to analyze the benefit of $\ell_\infty$-norm descent (a.k.a. sign descent) directly from properties of the data distribution, namely, heavy-tailed class imbalance. We propose a minimal yet representative setting of next-token prediction, where we can provably show faster convergence of coordinate-wise algorithms such as Sign descent (steepest descent w.r.t. $\ell_\infty$ norm) over normalized GD (steepest descent w.r.t. to $\ell_2$ norm) in the presence of heavy tail class imbalance.
Abstract (translated)
自适应优化方法（如Adam）在大规模语言模型的预训练中扮演着重要角色，显著优于梯度下降法(GD)。近期研究提出了一些关于损失函数的新平滑假设，以解释具有结构化预条件器（例如坐标方向或层方向）的自适应算法以及相对于非欧几里得范数（如$\ell_\infty$范数或谱范数）最速下降方法的优势，相较于GD。然而，这些平滑性假设如何在语言建模任务中表现尚不清楚。 在这项工作中，我们的目标是从数据分布的特性，特别是长尾类别不平衡的角度分析$\ell_\infty$范数下降（即符号下降）带来的优势。我们提出了一种最小但具有代表性的下一个令牌预测设置，在这种设置下，当存在长尾类别不平衡时，我们可以严格证明坐标方向算法如符号下降（相对于$\ell_\infty$范数的最速下降方法）比归一化GD（相对于$\ell_2$范数的最速下降方法）收敛更快。
URL
https://arxiv.org/abs/2512.00763
PDF
-
A Perceptually Inspired Variational Framework for Color Enhancement
2025-11-28 16:32:12Rodrigo Palma-Amestoy, Edoardo Provenzi, Marcelo Bertalm\'io, Vicent CasellesAbstract
Basic phenomenology of human color vision has been widely taken as an inspiration to devise explicit color correction algorithms. The behavior of these models in terms of significative image features (such as contrast and dispersion) can be difficult to characterize. To cope with this, we propose to use a variational formulation of color contrast enhancement that is inspired by the basic phenomenology of color perception. In particular, we devise a set of basic requirements to be fulfilled by an energy to be considered as `perceptually inspired', showing that there is an explicit class of functionals satisfying all of them. We single out three explicit functionals that we consider of basic interest, showing similarities and differences with existing models. The minima of such functionals is computed using a gradient descent approach. We also present a general methodology to reduce the computational cost of the algorithms under analysis from ${\cal O}(N^2)$ to ${\cal O}(N\log N)$, being $N$ the number of input pixels.
Abstract (translated)
URL
https://arxiv.org/abs/2511.23329
PDF
-
GACELLE: GPU-accelerated tools for model parameter estimation and image reconstruction
2025-11-27 04:30:29Kwok-Shing Chan (Department of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, United States, Harvard Medical School, Boston, MA, United States), Hansol Lee (Department of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, United States, Harvard Medical School, Boston, MA, United States), Yixin Ma (Department of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, United States, Harvard Medical School, Boston, MA, United States), Berkin Bilgic (Department of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, United States, Harvard Medical School, Boston, MA, United States), Susie Y. Huang (Department of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, United States, Harvard Medical School, Boston, MA, United States), Hong-Hsi Lee (Department of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, United States, Harvard Medical School, Boston, MA, United States), Jos\'e P. Marques (Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands)Abstract
Quantitative MRI (qMRI) offers tissue-specific biomarkers that can be tracked over time or compared across populations; however, its adoption in clinical research is hindered by significant computational demands of parameter estimation. Images acquired at high spatial resolution or requiring fitting multiple parameters often require lengthy processing time, constraining their use in routine pipelines and slowing methodological innovation and clinical translation. We present GACELLE, an open source, GPU-accelerated framework for high-throughput qMRI analysis. GACELLE provides a stochastic gradient descent optimiser and a stochastic sampler in MATLAB, enabling fast parameter mapping, improved estimation robustness via spatial regularisation, and uncertainty quantification. GACELLE prioritises accessibility: users only need to provide a forward signal model, while GACELLE's backend manages computational parallelisation, automatic parameter updates, and memory-batching. The stochastic solver performs fully vectorised Markov chain Monte Carlo with identical likelihood on CPU and GPU, ensuring reproducibility across hardware. Benchmarking demonstrates up to 451-fold acceleration for the stochastic gradient descent solver and 14,380-fold acceleration for stochastic sampling compared to CPU-based estimation, without compromising accuracy. We demonstrated GACELLE's versatility on three representative qMRI models and on an image reconstruction task. Across these applications, GACELLE improves parameter precision, enhances test-retest reproducibility, and reduces noise in quantitative maps. By combining speed, usability and flexibility, GACELLE provides a generalisable optimisation framework for medical image analysis. It lowers the computational barrier for qMRI, paving the way for reproducible biomarker development, large-scale imaging studies, and clinical translation.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22094
PDF
-
Quantum Circuit Reasoning Models: A Variational Framework for Differentiable Logical Inference
2025-11-26 23:15:14Andrew KirulutaAbstract
This report introduces a novel class of reasoning architectures, termed Quantum Circuit Reasoning Models (QCRM), which extend the concept of Variational Quantum Circuits (VQC) from energy minimization and classification tasks to structured logical inference and reasoning. We posit that fundamental quantum mechanical operations, superposition, entanglement, interference, and measurement, naturally map to essential reasoning primitives such as hypothesis branching, constraint propagation, consistency enforcement, and decision making. The resulting framework combines quantum-inspired computation with differentiable optimization, enabling reasoning to emerge as a process of amplitude evolution and interference-driven selection of self-consistent states. We develop the mathematical foundation of QCRM, define its parameterized circuit architecture, and show how logical rules can be encoded as unitary transformations over proposition-qubit states. We further formalize a training objective grounded in classical gradient descent over circuit parameters and discuss simulation-based implementations on classical hardware. Finally, we propose the Quantum Reasoning Layer (QRL) as a differentiable hybrid component for composable reasoning models applicable to scientific, biomedical, and chemical inference domains.
Abstract (translated)
这份报告介绍了一类新的推理架构，称为量子电路推理模型（Quantum Circuit Reasoning Models，简称 QCRM），这类模型将变分量子线路（Variational Quantum Circuits，VQC）的概念从能量最小化和分类任务扩展到了结构化的逻辑推理和推理领域。我们认为基本的量子力学操作——叠加、纠缠、干涉以及测量——自然地映射到诸如假设分支、约束传播、一致性强制执行和决策制定等重要的推理原语上。由此产生的框架结合了灵感来自量子计算的过程与可微分优化，使得推理能够作为振幅演化和由干涉驱动的自洽状态选择过程而出现。 我们建立了QCRM的数学基础，并定义了它的参数化电路架构，展示了如何将逻辑规则编码为命题-量子比特状态上的幺正变换。此外，我们还形式化了一个基于经典梯度下降法训练模型的目标函数，并讨论了在经典硬件上进行模拟实施的方法。最后，我们提出了量子推理层（Quantum Reasoning Layer，QRL），这是一种可微分的混合组件，适用于科学、生物医学和化学推断领域中的组合式推理模型。
URL
https://arxiv.org/abs/2512.07871
PDF
-
Closed-Loop Transformers: Autoregressive Modeling as Iterative Latent Equilibrium
2025-11-26 20:02:59Akbar Anbar Jafari, Gholamreza AnbarjafariAbstract
Contemporary autoregressive transformers operate in open loop: each hidden state is computed in a single forward pass and never revised, causing errors to propagate uncorrected through the sequence. We identify this open-loop bottleneck as a fundamental architectural limitation underlying well-documented failures in long-range reasoning, factual consistency, and multi-step planning. To address this limitation, we introduce the closed-loop prediction principle, which requires that models iteratively refine latent representations until reaching a self-consistent equilibrium before committing to each token. We instantiate this principle as Equilibrium Transformers (EqT), which augment standard transformer layers with an Equilibrium Refinement Module that minimizes a learned energy function via gradient descent in latent space. The energy function enforces bidirectional prediction consistency, episodic memory coherence, and output confidence, all computed without external supervision. Theoretically, we prove that EqT performs approximate MAP inference in a latent energy-based model, establish linear convergence guarantees, and show that refinement improves predictions precisely on hard instances where one-shot inference is suboptimal. The framework unifies deep equilibrium models, diffusion language models, and test-time training as special cases. Preliminary experiments on the binary parity task demonstrate +3.28% average improvement on challenging sequences, with gains reaching +8.07% where standard transformers approach random performance, validating that the benefit of deliberation scales with task difficulty. Just as attention mechanisms resolved the sequential bottleneck of recurrent networks, we propose that closed-loop equilibrium may resolve the commitment bottleneck of open-loop autoregression, representing a foundational step toward language models.
Abstract (translated)
URL
https://arxiv.org/abs/2511.21882
PDF
-
Gradient Descent Algorithm Survey
2025-11-25 09:30:44Deng Fucheng, Wang Wanjie, Gong Ao, Wang Xiaoqi, Wang FanAbstract
Focusing on the practical configuration needs of optimization algorithms in deep learning, this article concentrates on five major algorithms: SGD, Mini-batch SGD, Momentum, Adam, and Lion. It systematically analyzes the core advantages, limitations, and key practical recommendations of each algorithm. The research aims to gain an in-depth understanding of these algorithms and provide a standardized reference for the reasonable selection, parameter tuning, and performance improvement of optimization algorithms in both academic research and engineering practice, helping to solve optimization challenges in different scales of models and various training scenarios.
Abstract (translated)
URL
https://arxiv.org/abs/2511.20725
PDF
-
The Metaphysics We Train: A Heideggerian Reading of Machine Learning
Abstract
This paper offers a phenomenological reading of contemporary machine learning through Heideggerian concepts, aimed at enriching practitioners' reflexive understanding of their own practice. We argue that this philosophical lens reveals three insights invisible to purely technical analysis. First, the algorithmic Entwurf (projection) is distinctive in being automated, opaque, and emergent--a metaphysics that operates without explicit articulation or debate, crystallizing implicitly through gradient descent rather than theoretical argument. Second, even sophisticated technical advances remain within the regime of Gestell (Enframing), improving calculation without questioning the primacy of calculation itself. Third, AI's lack of existential structure, specifically the absence of Care (Sorge), is genuinely explanatory: it illuminates why AI systems have no internal resources for questioning their own optimization imperatives, and why they optimize without the anxiety (Angst) that signals, in human agents, the friction between calculative absorption and authentic existence. We conclude by exploring the pedagogical value of this perspective, arguing that data science education should cultivate not only technical competence but ontological literacy--the capacity to recognize what worldviews our tools enact and when calculation itself may be the wrong mode of engagement.
Abstract (translated)
本文通过海德格尔的概念提供了一种现象学视角，解读当代机器学习，并旨在加深实践者的反思理解。我们主张这种哲学透镜揭示了三个技术分析无法察觉的洞察。 首先，算法设计（Entwurf）的独特之处在于其自动化、不透明和涌现性——这是一种无需明确表述或讨论就运作的形而上学体系，在于通过梯度下降而非理论论证结晶形成。 其次，即使是高度先进的技术进步仍停留在Gestell（框架化）的领域内，这改善了计算能力，但并未质疑计算本身至高无上的地位。 第三，人工智能缺乏存在结构，特别是关心（Sorge）的缺失，这一点具有真正的解释力：它揭示了为什么AI系统没有内在资源来质疑自身的优化目标，并且它们在优化过程中不会产生焦虑（Angst），而这种焦虑是人类代理者在计算吸收与真实存在的冲突中产生的信号。 本文最后探讨了这一视角的教学价值，主张数据科学教育不仅要培养技术能力，还要培育本体论素养——即认识到我们的工具正在实施何种世界观，并能够在何种情况下计算本身并非合适的参与方式。
URL
https://arxiv.org/abs/2602.19028
PDF
-
Frequency-Invariant Beamforming in Elevation and Azimuth via Autograd and Concentric Circular Microphone Arrays
2025-11-24 18:42:03Jorge Ortigoso-Narro, Jose A. Belloch, Maximo Morales-Cespedes, Maximo CobosAbstract
The use of planar and concentric circular microphone arrays in beamforming has gained attention due to their ability to optimize both azimuth and elevation angles, making them ideal for spatial audio tasks like sound source localization and noise suppression. Unlike linear arrays, which restrict steering to a single axis, 2D arrays offer dual-axis optimization, although elevation control remains challenging. This study explores the integration of autograd, an automatic differentiation tool, with concentric circular arrays to impose beamwidth and frequency invariance constraints. This enables continuous optimization over both angles while maintaining performance across a wide frequency range. We evaluate our method through simulations of beamwidth, white noise gain, and directivity across multiple frequencies. A comparative analysis is presented against standard and advanced beamformers, including delay-and-sum, modified delay-and-sum, a Jacobi-Anger expansion-based method, and a Gaussian window-based gradient descent approach. Our method achieves superior spatial selectivity and narrower mainlobes, particularly in the elevation axis at lower frequencies. These results underscore the effectiveness of our approach in enhancing beamforming performance for acoustic sensing and spatial audio applications requiring precise dual-axis control.
Abstract (translated)
URL
https://arxiv.org/abs/2511.19403
PDF
-
Local Entropy Search over Descent Sequences for Bayesian Optimization
2025-11-24 15:52:17David Stenger, Armin Lindicke, Alexander von Rohr, Sebastian TrimpeAbstract
Searching large and complex design spaces for a global optimum can be infeasible and unnecessary. A practical alternative is to iteratively refine the neighborhood of an initial design using local optimization methods such as gradient descent. We propose local entropy search (LES), a Bayesian optimization paradigm that explicitly targets the solutions reachable by the descent sequences of iterative optimizers. The algorithm propagates the posterior belief over the objective through the optimizer, resulting in a probability distribution over descent sequences. It then selects the next evaluation by maximizing mutual information with that distribution, using a combination of analytic entropy calculations and Monte-Carlo sampling of descent sequences. Empirical results on high-complexity synthetic objectives and benchmark problems show that LES achieves strong sample efficiency compared to existing local and global Bayesian optimization methods.
Abstract (translated)
URL
https://arxiv.org/abs/2511.19241
PDF
-
LRDUN: A Low-Rank Deep Unfolding Network for Efficient Spectral Compressive Imaging
2025-11-23 16:08:34He Huang, Yujun Guo, Wei HeAbstract
Deep unfolding networks (DUNs) have achieved remarkable success and become the mainstream paradigm for spectral compressive imaging (SCI) reconstruction. Existing DUNs are derived from full-HSI imaging models, where each stage operates directly on the high-dimensional HSI, refining the entire data cube based on the single 2D coded measurement. However, this paradigm leads to computational redundancy and suffers from the ill-posed nature of mapping 2D residuals back to 3D space of HSI. In this paper, we propose two novel imaging models corresponding to the spectral basis and subspace image by explicitly integrating low-rank (LR) decomposition with the sensing model. Compared to recovering the full HSI, estimating these compact low-dimensional components significantly mitigates the ill-posedness. Building upon these novel models, we develop the Low-Rank Deep Unfolding Network (LRDUN), which jointly solves the two subproblems within an unfolded proximal gradient descent (PGD) framework. Furthermore, we introduce a Generalized Feature Unfolding Mechanism (GFUM) that decouples the physical rank in the data-fidelity term from the feature dimensionality in the prior module, enhancing the representational capacity and flexibility of the network. Extensive experiments on simulated and real datasets demonstrate that the proposed LRDUN achieves state-of-the-art (SOTA) reconstruction quality with significantly reduced computational cost.
Abstract (translated)
URL
https://arxiv.org/abs/2511.18513
PDF
-
UnfoldLDM: Deep Unfolding-based Blind Image Restoration with Latent Diffusion Priors
2025-11-22 18:44:01Chunming He, Rihan Zhang, Zheng Chen, Bowen Yang, CHengyu Fang, Yunlong Lin, Fengyang Xiao, Sina FarsiuAbstract
Deep unfolding networks (DUNs) combine the interpretability of model-based methods with the learning ability of deep networks, yet remain limited for blind image restoration (BIR). Existing DUNs suffer from: (1) \textbf{Degradation-specific dependency}, as their optimization frameworks are tied to a known degradation model, making them unsuitable for BIR tasks; and (2) \textbf{Over-smoothing bias}, resulting from the direct feeding of gradient descent outputs, dominated by low-frequency content, into the proximal term, suppressing fine textures. To overcome these issues, we propose UnfoldLDM to integrate DUNs with latent diffusion model (LDM) for BIR. In each stage, UnfoldLDM employs a multi-granularity degradation-aware (MGDA) module as the gradient descent step. MGDA models BIR as an unknown degradation estimation problem and estimates both the holistic degradation matrix and its decomposed forms, enabling robust degradation removal. For the proximal step, we design a degradation-resistant LDM (DR-LDM) to extract compact degradation-invariant priors from the MGDA output. Guided by this prior, an over-smoothing correction transformer (OCFormer) explicitly recovers high-frequency components and enhances texture details. This unique combination ensures the final result is degradation-free and visually rich. Experiments show that our UnfoldLDM achieves a leading place on various BIR tasks and benefits downstream tasks. Moreover, our design is compatible with existing DUN-based methods, serving as a plug-and-play framework. Code will be released.
Abstract (translated)
URL
https://arxiv.org/abs/2511.18152
PDF
-
Neural Networks Learn Generic Multi-Index Models Near Information-Theoretic Limit
2025-11-19 04:46:47Bohan Zhang, Zihao Wang, Hengyu Fu, Jason D. LeeAbstract
In deep learning, a central issue is to understand how neural networks efficiently learn high-dimensional features. To this end, we explore the gradient descent learning of a general Gaussian Multi-index model $f(\boldsymbol{x})=g(\boldsymbol{U}\boldsymbol{x})$ with hidden subspace $\boldsymbol{U}\in \mathbb{R}^{r\times d}$, which is the canonical setup to study representation learning. We prove that under generic non-degenerate assumptions on the link function, a standard two-layer neural network trained via layer-wise gradient descent can agnostically learn the target with $o_d(1)$ test error using $\widetilde{\mathcal{O}}(d)$ samples and $\widetilde{\mathcal{O}}(d^2)$ time. The sample and time complexity both align with the information-theoretic limit up to leading order and are therefore optimal. During the first stage of gradient descent learning, the proof proceeds via showing that the inner weights can perform a power-iteration process. This process implicitly mimics a spectral start for the whole span of the hidden subspace and eventually eliminates finite-sample noise and recovers this span. It surprisingly indicates that optimal results can only be achieved if the first layer is trained for more than $\mathcal{O}(1)$ steps. This work demonstrates the ability of neural networks to effectively learn hierarchical functions with respect to both sample and time efficiency.
Abstract (translated)
URL
https://arxiv.org/abs/2511.15120
PDF
-
Implicit Bias of the JKO Scheme
Abstract
Wasserstein gradient flow provides a general framework for minimizing an energy functional $J$ over the space of probability measures on a Riemannian manifold $(M,g)$. Its canonical time-discretization, the Jordan-Kinderlehrer-Otto (JKO) scheme, produces for any step size $\eta>0$ a sequence of probability distributions $\rho_k^\eta$ that approximate to first order in $\eta$ Wasserstein gradient flow on $J$. But the JKO scheme also has many other remarkable properties not shared by other first order integrators, e.g. it preserves energy dissipation and exhibits unconditional stability for $\lambda$-geodesically convex functionals $J$. To better understand the JKO scheme we characterize its implicit bias at second order in $\eta$. We show that $\rho_k^\eta$ are approximated to order $\eta^2$ by Wasserstein gradient flow on a \emph{modified} energy \[ J^{\eta}(\rho) = J(\rho) - \frac{\eta}{4}\int_M \Big\lVert \nabla_g \frac{\delta J}{\delta \rho} (\rho) \Big\rVert_{2}^{2} \,\rho(dx), \] obtained by subtracting from $J$ the squared metric curvature of $J$ times $\eta/4$. The JKO scheme therefore adds at second order in $\eta$ a \textit{deceleration} in directions where the metric curvature of $J$ is rapidly changing. This corresponds to canonical implicit biases for common functionals: for entropy the implicit bias is the Fisher information, for KL-divergence it is the Fisher-Hyv{ä}rinen divergence, and for Riemannian gradient descent it is the kinetic energy in the metric $g$. To understand the differences between minimizing $J$ and $J^\eta$ we study \emph{JKO-Flow}, Wasserstein gradient flow on $J^\eta$, in several simple numerical examples. These include exactly solvable Langevin dynamics on the Bures-Wasserstein space and Langevin sampling from a quartic potential in 1D.
Abstract (translated)
URL
https://arxiv.org/abs/2511.14827
PDF
-
Autonomous Vehicle Path Planning by Searching With Differentiable Simulation
2025-11-14 07:56:34Asen Nachkov, Jan-Nico Zaech, Danda Pani Paudel, Xi Wang, Luc Van GoolAbstract
Planning allows an agent to safely refine its actions before executing them in the real world. In autonomous driving, this is crucial to avoid collisions and navigate in complex, dense traffic scenarios. One way to plan is to search for the best action sequence. However, this is challenging when all necessary components - policy, next-state predictor, and critic - have to be learned. Here we propose Differentiable Simulation for Search (DSS), a framework that leverages the differentiable simulator Waymax as both a next state predictor and a critic. It relies on the simulator's hardcoded dynamics, making state predictions highly accurate, while utilizing the simulator's differentiability to effectively search across action sequences. Our DSS agent optimizes its actions using gradient descent over imagined future trajectories. We show experimentally that DSS - the combination of planning gradients and stochastic search - significantly improves tracking and path planning accuracy compared to sequence prediction, imitation learning, model-free RL, and other planning methods.
Abstract (translated)
URL
https://arxiv.org/abs/2511.11043
PDF
-
Fast Neural Tangent Kernel Alignment, Norm and Effective Rank via Trace Estimation
Abstract
The Neural Tangent Kernel (NTK) characterizes how a model's state evolves over Gradient Descent. Computing the full NTK matrix is often infeasible, especially for recurrent architectures. Here, we introduce a matrix-free perspective, using trace estimation to rapidly analyze the empirical, finite-width NTK. This enables fast computation of the NTK's trace, Frobenius norm, effective rank, and alignment. We provide numerical recipes based on the Hutch++ trace estimator with provably fast convergence guarantees. In addition, we show that, due to the structure of the NTK, one can compute the trace using only forward- or reverse-mode automatic differentiation, not requiring both modes. We show these so-called one-sided estimators can outperform Hutch++ in the low-sample regime, especially when the gap between the model state and parameter count is large. In total, our results demonstrate that matrix-free randomized approaches can yield speedups of many orders of magnitude, leading to faster analysis and applications of the NTK.
Abstract (translated)
URL
https://arxiv.org/abs/2511.10796
PDF
-
Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalization
2025-11-10 18:40:24Yu Huang, Zixin Wen, Aarti Singh, Yuejie Chi, Yuxin ChenAbstract
The ability to reason lies at the core of artificial intelligence (AI), and challenging problems usually call for deeper and longer reasoning to tackle. A crucial question about AI reasoning is whether models can extrapolate learned reasoning patterns to solve harder tasks with longer chain-of-thought (CoT). In this work, we present a theoretical analysis of transformers learning on synthetic state-tracking tasks with gradient descent. We mathematically prove how the algebraic structure of state-tracking problems governs the degree of extrapolation of the learned CoT. Specifically, our theory characterizes the length generalization of transformers through the mechanism of attention concentration, linking the retrieval robustness of the attention layer to the state-tracking task structure of long-context reasoning. Moreover, for transformers with limited reasoning length, we prove that a recursive self-training scheme can progressively extend the range of solvable problem lengths. To our knowledge, we provide the first optimization guarantee that constant-depth transformers provably learn $\mathsf{NC}^1$-complete problems with CoT, significantly going beyond prior art confined in $\mathsf{TC}^0$, unless the widely held conjecture $\mathsf{TC}^0 \neq \mathsf{NC}^1$ fails. Finally, we present a broad set of experiments supporting our theoretical results, confirming the length generalization behaviors and the mechanism of attention concentration.
Abstract (translated)
URL
https://arxiv.org/abs/2511.07378
PDF
-
Sparse4DGS: 4D Gaussian Splatting for Sparse-Frame Dynamic Scene Reconstruction
2025-11-10 14:10:43Changyue Shi, Chuxiao Yang, Xinyuan Hu, Minghao Chen, Wenwen Pan, Yan Yang, Jiajun Ding, Zhou Yu, Jun YuAbstract
Dynamic Gaussian Splatting approaches have achieved remarkable performance for 4D scene reconstruction. However, these approaches rely on dense-frame video sequences for photorealistic reconstruction. In real-world scenarios, due to equipment constraints, sometimes only sparse frames are accessible. In this paper, we propose Sparse4DGS, the first method for sparse-frame dynamic scene reconstruction. We observe that dynamic reconstruction methods fail in both canonical and deformed spaces under sparse-frame settings, especially in areas with high texture richness. Sparse4DGS tackles this challenge by focusing on texture-rich areas. For the deformation network, we propose Texture-Aware Deformation Regularization, which introduces a texture-based depth alignment loss to regulate Gaussian deformation. For the canonical Gaussian field, we introduce Texture-Aware Canonical Optimization, which incorporates texture-based noise into the gradient descent process of canonical Gaussians. Extensive experiments show that when taking sparse frames as inputs, our method outperforms existing dynamic or few-shot techniques on NeRF-Synthetic, HyperNeRF, NeRF-DS, and our iPhone-4D datasets.
Abstract (translated)
URL
https://arxiv.org/abs/2511.07122
PDF
-
FoCLIP: A Feature-Space Misalignment Framework for CLIP-Based Image Manipulation and Detection
2025-11-10 10:54:35Yulin Chen, Zeyuan Wang, Tianyuan Yu, Yingmei Wei, Liang BaiAbstract
The well-aligned attribute of CLIP-based models enables its effective application like CLIPscore as a widely adopted image quality assessment metric. However, such a CLIP-based metric is vulnerable for its delicate multimodal alignment. In this work, we propose \textbf{FoCLIP}, a feature-space misalignment framework for fooling CLIP-based image quality metric. Based on the stochastic gradient descent technique, FoCLIP integrates three key components to construct fooling examples: feature alignment as the core module to reduce image-text modality gaps, the score distribution balance module and pixel-guard regularization, which collectively optimize multimodal output equilibrium between CLIPscore performance and image quality. Such a design can be engineered to maximize the CLIPscore predictions across diverse input prompts, despite exhibiting either visual unrecognizability or semantic incongruence with the corresponding adversarial prompts from human perceptual perspectives. Experiments on ten artistic masterpiece prompts and ImageNet subsets demonstrate that optimized images can achieve significant improvement in CLIPscore while preserving high visual fidelity. In addition, we found that grayscale conversion induces significant feature degradation in fooling images, exhibiting noticeable CLIPscore reduction while preserving statistical consistency with original images. Inspired by this phenomenon, we propose a color channel sensitivity-driven tampering detection mechanism that achieves 91% accuracy on standard benchmarks. In conclusion, this work establishes a practical pathway for feature misalignment in CLIP-based multimodal systems and the corresponding defense method.
Abstract (translated)
URL
https://arxiv.org/abs/2511.06947
PDF
-
Sampling and Loss Weights in Multi-Domain Training
2025-11-10 10:08:53Mahdi Salmani, Pratik Worah, Meisam Razaviyayn, Vahab MirrokniAbstract
In the training of large deep neural networks, there is a need for vast amounts of training data. To meet this need, data is collected from multiple domains, such as Wikipedia and GitHub. These domains are heterogeneous in both data quality and the diversity of information they provide. This raises the question of how much we should rely on each domain. Several methods have attempted to address this issue by assigning sampling weights to each data domain using heuristics or approximations. As a first step toward a deeper understanding of the role of data mixing, this work revisits the problem by studying two kinds of weights: sampling weights, which control how much each domain contributes in a batch, and loss weights, which scale the loss from each domain during training. Through a rigorous study of linear regression, we show that these two weights play complementary roles. First, they can reduce the variance of gradient estimates in iterative methods such as stochastic gradient descent (SGD). Second, they can improve generalization performance by reducing the generalization gap. We provide both theoretical and empirical support for these claims. We further study the joint dynamics of sampling weights and loss weights, examining how they can be combined to capture both contributions.
Abstract (translated)
URL
https://arxiv.org/abs/2511.06913
PDF
