Gradient_Descent
-
PoGO: A Scalable Proof of Useful Work via Quantized Gradient Descent and Merkle Proofs
2025-04-10 08:09:34Jos\'e I. OrlickiAbstract
We present a design called \emph{Proof of Gradient Optimization} (PoGO) for blockchain consensus, where miners produce verifiable evidence of training large-scale machine-learning models. Building on previous work, we incorporate \emph{quantized gradients} (4-bit precision) to reduce storage and computation requirements, while still preserving the ability of verifiers to check that real progress has been made on lowering the model's loss. Additionally, we employ Merkle proofs over the full 32-bit model to handle large parameter sets and to enable random leaf checks with minimal on-chain data. We illustrate these ideas using GPT-3 (175B parameters) as a reference example and also refer to smaller but high-performance models (e.g., \emph{Gemma~3} with 27B parameters). We provide an empirical cost analysis showing that verification is significantly cheaper than training, thanks in part to quantization and sampling. We also discuss the necessity of longer block times (potentially hours) when incorporating meaningful training steps, the trade-offs when using specialized GPU hardware, and how binary diffs may incrementally optimize updates. Finally, we note that fine-tuning can be handled in a similar manner, merely changing the dataset and the manner of sampling but preserving the overall verification flow. Our protocol allows verifiers to issue either \emph{positive} or \emph{negative} attestations; these are aggregated at finalization to either confirm the update or slash the miner.
Abstract (translated)
我们提出了一种名为“梯度优化证明”(PoGO)的区块链共识设计，其中矿工生成训练大规模机器学习模型的有效证据。在此前工作的基础上，我们采用4位精度的量化梯度来减少存储和计算需求，同时仍然保持验证者检查实际损失降低进展的能力。此外，我们在整个32位模型上使用Merkle证明处理大量参数，并允许进行随机叶节点检查，仅需极少量的链上数据。我们以GPT-3（1750亿参数）作为参考示例来阐述这些想法，并提及一些较小但高性能的模型（如具有270亿参数的Gemma 3）。我们提供了一个经验成本分析，表明由于量化和采样的原因，验证比训练要便宜得多。我们还讨论了在包含有意义的训练步骤时需要更长的区块时间（可能长达数小时），使用专用GPU硬件时的成本与收益权衡，以及二进制差分如何逐步优化更新的问题。最后，我们注意到微调可以以类似的方式处理，只需更改数据集和采样的方式即可保持整体验证流程不变。我们的协议允许验证者发布“积极”或“消极”的认证；这些在最终确定时进行汇总，确认更新或削减矿工的奖励。
URL
https://arxiv.org/abs/2504.07540
PDF
-
Architecture independent generalization bounds for overparametrized deep ReLU networks
2025-04-08 05:37:38Thomas Chen, Chun-Kai Kevin Chien, Patricia Mu\~noz Ewald, Andrew G. MooreAbstract
We prove that overparametrized neural networks are able to generalize with a test error that is independent of the level of overparametrization, and independent of the Vapnik-Chervonenkis (VC) dimension. We prove explicit bounds that only depend on the metric geometry of the test and training sets, on the regularity properties of the activation function, and on the operator norms of the weights and norms of biases. For overparametrized deep ReLU networks with a training sample size bounded by the input space dimension, we explicitly construct zero loss minimizers without use of gradient descent, and prove that the generalization error is independent of the network architecture.
Abstract (translated)
我们证明了过度参数化的神经网络能够在测试误差上实现泛化，该误差与过度参数化的程度无关，并且不受Vapnik-Chervonenkis (VC) 维度的影响。我们给出了具体依赖于测试集和训练集的测地几何、激活函数的正则性性质以及权重算子范数及偏置范数的显式界。对于输入空间维度受限的过度参数化深度ReLU网络，我们在不使用梯度下降的情况下明确构造出零损失最小值，并证明了泛化误差与网络架构无关。
URL
https://arxiv.org/abs/2504.05695
PDF
-
Lattice: Learning to Efficiently Compress the Memory
2025-04-08 03:48:43Mahdi Karami, Vahab MirrokniAbstract
Attention mechanisms have revolutionized sequence learning but suffer from quadratic computational complexity. This paper introduces Lattice, a novel recurrent neural network (RNN) mechanism that leverages the inherent low-rank structure of K-V matrices to efficiently compress the cache into a fixed number of memory slots, achieving sub-quadratic complexity. We formulate this compression as an online optimization problem and derive a dynamic memory update rule based on a single gradient descent step. The resulting recurrence features a state- and input-dependent gating mechanism, offering an interpretable memory update process. The core innovation is the orthogonal update: each memory slot is updated exclusively with information orthogonal to its current state hence incorporation of only novel, non-redundant data, which minimizes the interference with previously stored information. The experimental results show that Lattice achieves the best perplexity compared to all baselines across diverse context lengths, with performance improvement becoming more pronounced as the context length increases.
Abstract (translated)
注意力机制已彻底改变了序列学习，但其计算复杂度呈二次方增长。本文介绍了一种名为Lattice的新颖递归神经网络（RNN）机制，该机制利用K-V矩阵的固有低秩结构来高效地将缓存压缩为固定数量的记忆槽，从而实现次线性复杂度。我们将这种压缩形式化为一个在线优化问题，并基于单步梯度下降推导出一种动态记忆更新规则。由此产生的递归特征包含了一个依赖于状态和输入的门控机制，提供了可解释的记忆更新过程。 Lattice的核心创新在于正交更新：每个记忆槽仅通过与其当前状态正交的信息进行更新，从而只吸收新颖且非冗余的数据，最大限度地减少对先前存储信息的干扰。实验结果表明，无论上下文长度如何，Lattice在所有基准测试中都达到了最佳困惑度（perplexity），并且随着上下文长度的增加，其性能改进愈发明显。
URL
https://arxiv.org/abs/2504.05646
PDF
-
BC-ADMM: An Efficient Non-convex Constrained Optimizer with Robotic Applications
2025-04-07 19:56:06Zherong Pan, Kui WuAbstract
Non-convex constrained optimizations are ubiquitous in robotic applications such as multi-agent navigation, UAV trajectory optimization, and soft robot simulation. For this problem class, conventional optimizers suffer from small step sizes and slow convergence. We propose BC-ADMM, a variant of Alternating Direction Method of Multiplier (ADMM), that can solve a class of non-convex constrained optimizations with biconvex constraint relaxation. Our algorithm allows larger step sizes by breaking the problem into small-scale sub-problems that can be easily solved in parallel. We show that our method has both theoretical convergence speed guarantees and practical convergence guarantees in the asymptotic sense. Through numerical experiments in a row of four robotic applications, we show that BC-ADMM has faster convergence than conventional gradient descent and Newton's method in terms of wall clock time.
Abstract (translated)
非凸约束优化在机器人应用中非常普遍，例如多智能体导航、无人机轨迹优化和软机器人模拟。对于这类问题，传统优化器由于步长小且收敛慢而存在局限性。我们提出了BC-ADMM，这是一种交替方向乘子法（ADMM）的变体，能够解决一类带有双凸约束松弛的非凸约束优化问题。我们的算法通过将大问题分解为可以并行求解的小规模子问题来允许使用更大的步长。我们证明了该方法在理论上的收敛速度保证以及渐近意义下的实际收敛性保证。通过一系列四个机器人应用中的数值实验，我们展示了BC-ADMM在实时时钟时间上比传统梯度下降和牛顿法的收敛速度快。
URL
https://arxiv.org/abs/2504.05465
PDF
-
Explaining Low Perception Model Competency with High-Competency Counterfactuals
2025-04-07 16:46:52Sara Pohland, Claire TomlinAbstract
There exist many methods to explain how an image classification model generates its decision, but very little work has explored methods to explain why a classifier might lack confidence in its prediction. As there are various reasons the classifier might lose confidence, it would be valuable for this model to not only indicate its level of uncertainty but also explain why it is uncertain. Counterfactual images have been used to visualize changes that could be made to an image to generate a different classification decision. In this work, we explore the use of counterfactuals to offer an explanation for low model competency--a generalized form of predictive uncertainty that measures confidence. Toward this end, we develop five novel methods to generate high-competency counterfactual images, namely Image Gradient Descent (IGD), Feature Gradient Descent (FGD), Autoencoder Reconstruction (Reco), Latent Gradient Descent (LGD), and Latent Nearest Neighbors (LNN). We evaluate these methods across two unique datasets containing images with six known causes for low model competency and find Reco, LGD, and LNN to be the most promising methods for counterfactual generation. We further evaluate how these three methods can be utilized by pre-trained Multimodal Large Language Models (MLLMs) to generate language explanations for low model competency. We find that the inclusion of a counterfactual image in the language model query greatly increases the ability of the model to generate an accurate explanation for the cause of low model competency, thus demonstrating the utility of counterfactual images in explaining low perception model competency.
Abstract (translated)
有许多方法可以解释图像分类模型如何生成其决策，但很少有研究探索了分类器为何对其预测缺乏信心的方法。由于造成分类器自信度下降的原因多种多样，该模型不仅需要表明它的不确定性水平，还需要解释为什么它会感到不确定。 反事实图像被用来展示对图像进行哪些改变可以产生不同的分类结果。在这项工作中，我们探讨使用反事实来提供低模型能力（一种衡量信心的预测不确定性的泛化形式）的解释的方法。为此，我们开发了五种新的生成高能力反事实图像的方法：图像梯度下降（IGD）、特征梯度下降（FGD）、自动编码器重建（Reco）、潜在梯度下降（LGD）和潜在最近邻（LNN）。我们在包含六种已知导致低模型能力的图像的两个独特数据集上评估了这些方法，并发现Reco、LGD和LNN是生成反事实最有前景的方法。我们进一步研究了这三种方法如何被预训练的多模态大型语言模型（MLLM）用来为低模型能力产生文本解释。我们发现，在语言模型查询中包含一张反事实图像大大提高了其生成准确解释的能力，即解释造成低模型能力的原因，从而证明了反事实图像在解释低感知模型能力方面的实用性。
URL
https://arxiv.org/abs/2504.05254
PDF
-
Comparative analysis of Realistic EMF Exposure Estimation from Low Density Sensor Network by Finite & Infinite Neural Networks
2025-04-07 12:31:53Mohammed Mallik, Laurent Clavier, Davy P. GaillotAbstract
Understanding the spatial and temporal patterns of environmental exposure to radio-frequency electromagnetic fields (RF-EMF) is essential for conducting risk assessments. These assessments aim to explore potential connections between RF-EMF exposure and its effects on human health, as well as on wildlife and plant life. Existing research has used different machine learning tools for EMF exposure estimation; however, a comparative analysis of these techniques is required to better understand their performance for real-world datasets. In this work, we present both finite and infinite-width convolutional network-based methods to estimate and assess EMF exposure levels from 70 real-world sensors in Lille, France. A comparative analysis has been conducted to analyze the performance of the methods' execution time and estimation accuracy. To improve estimation accuracy for higher-resolution grids, we utilized a preconditioned gradient descent method for kernel estimation. Root Mean Square Error (RMSE) is used as the evaluation criterion for comparing the performance of these deep learning models.
Abstract (translated)
理解射频电磁场（RF-EMF）的环境暴露在时空模式中的规律对于进行风险评估至关重要。这些评估旨在探索RF-EMF暴露与人类健康、野生动物和植物生命之间潜在联系的影响。现有的研究已经使用了不同的机器学习工具来估算电磁场（EMF）暴露，但是需要对这些技术进行全面分析以更好地了解它们处理实际数据集的表现。在这项工作中，我们提出了基于有限宽度和无限宽度卷积网络的方法来估计和评估法国里尔70个真实世界传感器的EMF暴露水平。进行了一次比较性分析来分析方法执行时间和估算准确性的表现。为了提高更高分辨率网格中的估算准确性，我们在核函数估计中使用了预处理梯度下降法。均方根误差（RMSE）被用作比较这些深度学习模型性能的标准。
URL
https://arxiv.org/abs/2504.07990
PDF
-
Constrained Gaussian Process Motion Planning via Stein Variational Newton Inference
2025-04-07 11:20:11Jiayun Li, Kay Pompetzki, An Thai Le, Haolei Tong, Jan Peters, Georgia ChalvatzakiAbstract
Gaussian Process Motion Planning (GPMP) is a widely used framework for generating smooth trajectories within a limited compute time--an essential requirement in many robotic applications. However, traditional GPMP approaches often struggle with enforcing hard nonlinear constraints and rely on Maximum a Posteriori (MAP) solutions that disregard the full Bayesian posterior. This limits planning diversity and ultimately hampers decision-making. Recent efforts to integrate Stein Variational Gradient Descent (SVGD) into motion planning have shown promise in handling complex constraints. Nonetheless, these methods still face persistent challenges, such as difficulties in strictly enforcing constraints and inefficiencies when the probabilistic inference problem is poorly conditioned. To address these issues, we propose a novel constrained Stein Variational Gaussian Process Motion Planning (cSGPMP) framework, incorporating a GPMP prior specifically designed for trajectory optimization under hard constraints. Our approach improves the efficiency of particle-based inference while explicitly handling nonlinear constraints. This advancement significantly broadens the applicability of GPMP to motion planning scenarios demanding robust Bayesian inference, strict constraint adherence, and computational efficiency within a limited time. We validate our method on standard benchmarks, achieving an average success rate of 98.57% across 350 planning tasks, significantly outperforming competitive baselines. This demonstrates the ability of our method to discover and use diverse trajectory modes, enhancing flexibility and adaptability in complex environments, and delivering significant improvements over standard baselines without incurring major computational costs.
Abstract (translated)
高斯过程运动规划（GPMP）是一种广泛应用于生成平滑轨迹的框架，这对于许多机器人应用中的计算时间限制是一个基本要求。然而，传统的GPMP方法在处理硬非线性约束时往往面临困难，并且依赖于最大后验概率（MAP）解，这忽略了完整的贝叶斯后验分布。这种做法限制了规划多样性并最终影响决策制定能力。最近将Stein变分梯度下降法（SVGD）集成到运动规划中的尝试，在处理复杂约束方面显示出潜力。尽管如此，这些方法仍然面临持续的挑战，例如难以严格强制执行约束和在概率推理问题条件不佳时效率低下。 为了解决这些问题，我们提出了一种新的带有非线性约束的Stein变分高斯过程运动规划（cSGPMP）框架，该框架结合了专门用于硬约束轨迹优化的GPMP先验。我们的方法通过显式处理非线性约束提高了基于粒子的推理效率。这一改进大大扩展了GPMP在需要稳健贝叶斯推断、严格遵守约束和有限时间内计算高效的运动规划场景中的适用范围。 我们在标准基准上验证了我们提出的方法，实现了350个规划任务的平均成功率98.57%，显著优于竞争性基线方法。这表明我们的方法能够发现并利用多种轨迹模式，提高在复杂环境下的灵活性和适应性，并且在不大幅增加计算成本的情况下对标准基线进行了重大改进。 这种方法为机器人系统中的运动规划提供了一种强大的新工具，尤其是在需要遵守硬约束条件的动态环境中显得尤为重要。
URL
https://arxiv.org/abs/2504.04936
PDF
-
Provable Failure of Language Models in Learning Majority Boolean Logic via Gradient Descent
2025-04-07 03:08:12Bo Chen, Zhenmei Shi, Zhao Song, Jiahao ZhangAbstract
Recent advancements in Transformer-based architectures have led to impressive breakthroughs in natural language processing tasks, with models such as GPT-4, Claude, and Gemini demonstrating human-level reasoning abilities. However, despite their high performance, concerns remain about the inherent limitations of these models, especially when it comes to learning basic logical functions. While complexity-theoretic analyses indicate that Transformers can represent simple logic functions (e.g., $\mathsf{AND}$, $\mathsf{OR}$, and majority gates) by its nature of belonging to the $\mathsf{TC}^0$ class, these results assume ideal parameter settings and do not account for the constraints imposed by gradient descent-based training methods. In this work, we investigate whether Transformers can truly learn simple majority functions when trained using gradient-based methods. We focus on a simplified variant of the Transformer architecture and consider both $n=\mathrm{poly}(d)$ and $n=\exp(\Omega(d))$ number of training samples, where each sample is a $d$-size binary string paired with the output of a basic majority function. Our analysis demonstrates that even after $\mathrm{poly}(d)$ gradient queries, the generalization error of the Transformer model still remains substantially large, growing exponentially with $d$. This work highlights fundamental optimization challenges in training Transformers for the simplest logical reasoning tasks and provides new insights into their theoretical limitations.
Abstract (translated)
最近基于Transformer架构的进展在自然语言处理任务中取得了显著突破，模型如GPT-4、Claude和Gemini展示了接近人类水平的理解能力。然而，尽管这些模型表现出色，人们依然对其内在局限性表示担忧，特别是在学习基本逻辑功能方面的问题。虽然复杂性理论分析表明，Transformer由于属于$\mathsf{TC}^0$类（理论上能够代表简单的逻辑函数如$\mathsf{AND}$、$\mathsf{OR}$和多数门），可以自然地表达这些简单逻辑函数，但这些结果假设了理想化的参数设置，并未考虑基于梯度下降的训练方法所施加的实际约束。本研究探讨了使用基于梯度的方法训练时，Transformer是否能够真正学会简单的多数表决函数（majority functions）。 我们专注于一个简化的Transformer架构变体，并考虑两种不同的样本数量：$n=\mathrm{poly}(d)$和$n=\exp(\Omega(d))$，其中每个样本由长度为$d$的二进制字符串与其对应的简单多数表决函数输出组成。我们的分析表明，在进行了$\mathrm{poly}(d)$次梯度查询之后，Transformer模型的泛化误差仍然显著较大，并且随着$d$的增长呈指数上升。 这项工作突显了在最简单的逻辑推理任务中训练Transformer时面临的根本性优化挑战，并为理解这些模型的理论局限提供了新的视角。
URL
https://arxiv.org/abs/2504.04702
PDF
-
Gating is Weighting: Understanding Gated Linear Attention through In-context Learning
2025-04-06 00:37:36Yingcong Li, Davoud Ataee Tarzanagh, Ankit Singh Rawat, Maryam Fazel, Samet OymakAbstract
Linear attention methods offer a compelling alternative to softmax attention due to their efficiency in recurrent decoding. Recent research has focused on enhancing standard linear attention by incorporating gating while retaining its computational benefits. Such Gated Linear Attention (GLA) architectures include competitive models such as Mamba and RWKV. In this work, we investigate the in-context learning capabilities of the GLA model and make the following contributions. We show that a multilayer GLA can implement a general class of Weighted Preconditioned Gradient Descent (WPGD) algorithms with data-dependent weights. These weights are induced by the gating mechanism and the input, enabling the model to control the contribution of individual tokens to prediction. To further understand the mechanics of this weighting, we introduce a novel data model with multitask prompts and characterize the optimization landscape of learning a WPGD algorithm. Under mild conditions, we establish the existence and uniqueness (up to scaling) of a global minimum, corresponding to a unique WPGD solution. Finally, we translate these findings to explore the optimization landscape of GLA and shed light on how gating facilitates context-aware learning and when it is provably better than vanilla linear attention.
Abstract (translated)
线性注意力方法由于其在递归解码中的效率，为传统的softmax注意力提供了一种有吸引力的替代方案。最近的研究集中在通过引入门控机制来增强标准线性注意力，同时保持计算效益不变。这种门控线性注意（GLA）架构包括竞争性的模型如Mamba和RWKV。在这项工作中，我们探讨了GLA模型在情境学习中的能力，并作出了以下贡献： 1. 我们展示了多层的GLA可以实现一类具有数据依赖权重的加权预处理梯度下降(WPGD)算法。这些权重是由门控机制和输入共同诱导出来的，使模型能够控制各个令牌对预测的贡献。 2. 为进一步理解这种加权机制的工作原理，我们引入了一种新的数据模型，该模型使用多任务提示，并描述了学习WPGD算法时优化景观的特点。在适度条件下，我们证明了一个全局最小值的存在性和唯一性（直到缩放为止），这对应于一个唯一的WPGD解决方案。 3. 最后，我们将这些发现转化为探讨GLA的优化景观，并阐明门控机制如何促进上下文感知学习以及何时可以严格证明其优于传统的线性注意力。 这项研究提供了关于门控线性注意模型内部运作的新见解，并为进一步改进语言模型中的注意力机制指明了方向。
URL
https://arxiv.org/abs/2504.04308
PDF
-
Compressing 3D Gaussian Splatting by Noise-Substituted Vector Quantization
2025-04-03 22:19:34Haishan Wang, Mohammad Hassan Vali, Arno SolinAbstract
3D Gaussian Splatting (3DGS) has demonstrated remarkable effectiveness in 3D reconstruction, achieving high-quality results with real-time radiance field rendering. However, a key challenge is the substantial storage cost: reconstructing a single scene typically requires millions of Gaussian splats, each represented by 59 floating-point parameters, resulting in approximately 1~GB of memory. To address this challenge, we propose a compression method by building separate attribute codebooks and storing only discrete code indices. Specifically, we employ noise-substituted vector quantization technique to jointly train the codebooks and model features, ensuring consistency between gradient descent optimization and parameter discretization. Our method reduces the memory consumption efficiently (around $45\times$) while maintaining competitive reconstruction quality on standard 3D benchmark scenes. Experiments on different codebook sizes show the trade-off between compression ratio and image quality. Furthermore, the trained compressed model remains fully compatible with popular 3DGS viewers and enables faster rendering speed, making it well-suited for practical applications.
Abstract (translated)
3D高斯点阵（3DGS）在三维重建中表现出显著的有效性，能够实现高质量的结果，并且支持实时辐射场渲染。然而，一个关键的挑战在于其巨大的存储成本：重构单个场景通常需要数百万个高斯点阵，每个点阵由59个浮点参数表示，总计大约需要1GB的内存空间。为了解决这一问题，我们提出了一种压缩方法，即构建独立的属性代码本并仅存储离散化的代码索引。具体来说，我们采用噪声替代向量量化技术来联合训练代码本和模型特征，确保梯度下降优化与参数离散化之间的一致性。我们的方法能够有效地减少内存消耗（约45倍），同时在标准的3D基准场景上保持竞争性的重建质量。不同大小代码本上的实验表明了压缩比率与图像质量之间的权衡关系。此外，训练好的压缩模型完全兼容流行的3DGS查看器，并且支持更快的渲染速度，使其非常适合实际应用。
URL
https://arxiv.org/abs/2504.03059
PDF
-
Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
2025-04-03 21:35:22Hanping Zhang, Yuhong GuoAbstract
Safe Reinforcement Learning (Safe RL) aims to train an RL agent to maximize its performance in real-world environments while adhering to safety constraints, as exceeding safety violation limits can result in severe consequences. In this paper, we propose a novel safe RL approach called Safety Modulated Policy Optimization (SMPO), which enables safe policy function learning within the standard policy optimization framework through safety modulated rewards. In particular, we consider safety violation costs as feedback from the RL environments that are parallel to the standard awards, and introduce a Q-cost function as safety critic to estimate expected future cumulative costs. Then we propose to modulate the rewards using a cost-aware weighting function, which is carefully designed to ensure the safety limits based on the estimation of the safety critic, while maximizing the expected rewards. The policy function and the safety critic are simultaneously learned through gradient descent during online interactions with the environment. We conduct experiments using multiple RL environments and the experimental results demonstrate that our method outperforms several classic and state-of-the-art comparison methods in terms of overall safe RL performance.
Abstract (translated)
安全强化学习（Safe Reinforcement Learning，简称 Safe RL）旨在训练一个强化学习代理在遵守安全性约束的前提下，在现实世界环境中最大化其性能。超出安全违规限制可能会导致严重后果。本文提出了一种名为“安全调制策略优化”（Safety Modulated Policy Optimization，SMPO）的新方法，该方法通过引入基于安全性的奖励调节机制，在标准策略优化框架内实现安全策略函数学习。 具体来说，我们将安全性违规成本视为来自平行于标准奖赏的强化学习环境中的反馈，并引入一个 Q-成本函数作为安全评论家来估计未来的累计预期成本。然后，我们提出使用一种成本感知加权函数对奖励进行调制，该函数经过精心设计，在确保基于安全评论家估算的安全限制的同时，最大化预期收益。 策略函数和安全评论家通过与环境的在线交互期间同时利用梯度下降法学习。我们在多个强化学习环境中进行了实验，并且实验证明了我们的方法在总体安全性强化学习性能方面优于几种经典及最新的比较方法。
URL
https://arxiv.org/abs/2504.03040
PDF
-
Learning from Streaming Video with Orthogonal Gradients
2025-04-02 17:59:57Tengda Han, Dilara Gokay, Joseph Heyward, Chuhan Zhang, Daniel Zoran, Viorica P\u{a}tr\u{a}ucean, Jo\~ao Carreira, Dima Damen, Andrew ZissermanAbstract
We address the challenge of representation learning from a continuous stream of video as input, in a self-supervised manner. This differs from the standard approaches to video learning where videos are chopped and shuffled during training in order to create a non-redundant batch that satisfies the independently and identically distributed (IID) sample assumption expected by conventional training paradigms. When videos are only available as a continuous stream of input, the IID assumption is evidently broken, leading to poor performance. We demonstrate the drop in performance when moving from shuffled to sequential learning on three tasks: the one-video representation learning method DoRA, standard VideoMAE on multi-video datasets, and the task of future video prediction. To address this drop, we propose a geometric modification to standard optimizers, to decorrelate batches by utilising orthogonal gradients during training. The proposed modification can be applied to any optimizer -- we demonstrate it with Stochastic Gradient Descent (SGD) and AdamW. Our proposed orthogonal optimizer allows models trained from streaming videos to alleviate the drop in representation learning performance, as evaluated on downstream tasks. On three scenarios (DoRA, VideoMAE, future prediction), we show our orthogonal optimizer outperforms the strong AdamW in all three scenarios.
Abstract (translated)
我们解决的是从连续视频流中以自监督方式学习表示的挑战。这与标准的视频学习方法不同，后者在训练过程中将视频片段打乱并重新排序，以便创建一个不冗余且满足独立同分布（IID）样本假设的小批量数据集。当仅能获得连续输入视频时，IID假设显然被破坏，导致性能下降。我们通过三个任务展示了从打乱学习到顺序学习的性能下降：单个视频表示学习方法DoRA、标准VideoMAE在多视频数据集上的应用以及未来视频预测任务。 为了应对这一问题，我们提出了对标准优化器进行几何修改的方法，利用正交梯度在训练过程中去相关化批次。所提出的修改可以应用于任何优化器——我们在随机梯度下降（SGD）和AdamW上进行了演示。我们的正交优化器使从流式视频中训练的模型能够缓解表示学习性能下降的问题，并通过下游任务进行评估。在三个场景（DoRA、VideoMAE、未来预测）下，我们展示了我们的正交优化器在所有三种情况下均优于强大的AdamW。
URL
https://arxiv.org/abs/2504.01961
PDF
-
Bridge the Gap between SNN and ANN for Image Restoration
2025-04-02 14:12:06Xin Su, Chen Wu, Zhuoran ZhengAbstract
Models of dense prediction based on traditional Artificial Neural Networks (ANNs) require a lot of energy, especially for image restoration tasks. Currently, neural networks based on the SNN (Spiking Neural Network) framework are beginning to make their mark in the field of image restoration, especially as they typically use less than 10\% of the energy of ANNs with the same architecture. However, training an SNN is much more expensive than training an ANN, due to the use of the heuristic gradient descent strategy. In other words, the process of SNN's potential membrane signal changing from sparse to dense is very slow, which affects the convergence of the whole this http URL tackle this problem, we propose a novel distillation technique, called asymmetric framework (ANN-SNN) distillation, in which the teacher is an ANN and the student is an SNN. Specifically, we leverage the intermediate features (feature maps) learned by the ANN as hints to guide the training process of the SNN. This approach not only accelerates the convergence of the SNN but also improves its final performance, effectively bridging the gap between the efficiency of the SNN and the superior learning capabilities of ANN. Extensive experimental results show that our designed SNN-based image restoration model, which has only 1/300 the number of parameters of the teacher network and 1/50 the energy consumption of the teacher network, is as good as the teacher network in some denoising tasks.
Abstract (translated)
基于传统人工神经网络（ANN）的密集预测模型在执行如图像恢复等任务时需要消耗大量能源。目前，基于脉冲神经网络（SNN）框架的神经网络开始在图像恢复领域崭露头角，特别是因为它们通常使用的能量是具有相同架构的人工神经网络的不到10%。然而，由于采用了启发式梯度下降策略，训练一个SNN比训练ANN要昂贵得多。换句话说，在SNN中，潜在膜信号从稀疏到密集的变化过程非常缓慢，这影响了整个训练过程的收敛性。 为了解决这个问题，我们提出了一种新颖的知识蒸馏技术，称为不对称框架（ANN-SNN）蒸馏，在这种方法中，教师模型是ANN而学生模型是SNN。具体来说，我们将通过ANN学习到的中间特征（特征图）作为线索来指导SNN的训练过程。这种做法不仅加速了SNN的收敛速度，还提升了其最终性能，有效地弥合了SNN在效率方面与ANN在学习能力方面的差距。 大量的实验结果表明，我们设计的基于SNN的图像恢复模型，在参数数量上仅为教师网络的1/300，并且能耗为教师网络的1/50，但在某些去噪任务中的性能可以媲美甚至超过教师网络。
URL
https://arxiv.org/abs/2504.01755
PDF
-
From Easy to Hard: Building a Shortcut for Differentially Private Image Synthesis
2025-04-02 06:30:55Kecen Li, Chen Gong, Xiaochen Li, Yuzhong Zhao, Xinwen Hou, Tianhao WangAbstract
Differentially private (DP) image synthesis aims to generate synthetic images from a sensitive dataset, alleviating the privacy leakage concerns of organizations sharing and utilizing synthetic images. Although previous methods have significantly progressed, especially in training diffusion models on sensitive images with DP Stochastic Gradient Descent (DP-SGD), they still suffer from unsatisfactory performance. In this work, inspired by curriculum learning, we propose a two-stage DP image synthesis framework, where diffusion models learn to generate DP synthetic images from easy to hard. Unlike existing methods that directly use DP-SGD to train diffusion models, we propose an easy stage in the beginning, where diffusion models learn simple features of the sensitive images. To facilitate this easy stage, we propose to use `central images', simply aggregations of random samples of the sensitive dataset. Intuitively, although those central images do not show details, they demonstrate useful characteristics of all images and only incur minimal privacy costs, thus helping early-phase model training. We conduct experiments to present that on the average of four investigated image datasets, the fidelity and utility metrics of our synthetic images are 33.1% and 2.1% better than the state-of-the-art method.
Abstract (translated)
差分隐私（DP）图像合成的目标是从敏感数据集生成合成图像，以减轻组织在共享和利用这些合成图像时的隐私泄露担忧。尽管先前的方法已取得了显著进步，尤其是在使用差分隐私随机梯度下降法（DP-SGD）对敏感图像训练扩散模型方面，它们仍然存在性能不理想的状况。在这项工作中，受课程学习的启发，我们提出了一种两阶段的DP图像合成框架，在此框架中，扩散模型从易到难逐步学习生成DP合成图像。不同于现有的直接使用DP-SGD来训练扩散模型的方法，我们在初始阶段提出了一个“简单”阶段，使扩散模型能够学习敏感图像的基本特征。为了促进这一简单的学习阶段，我们提议采用“中心图像”，即敏感数据集中随机样本的简单聚合物。直观地讲，尽管这些中心图并没有展示细节信息，但它们展示了所有图像的重要特性，并且只带来极小的隐私成本，从而有助于早期模型训练。我们的实验表明，在四个研究的图像数据集上的平均表现中，我们生成的合成图像在保真度和实用性指标上分别比现有最佳方法高出33.1%和2.1%。
URL
https://arxiv.org/abs/2504.01395
PDF
-
PixelCAM: Pixel Class Activation Mapping for Histology Image Classification and ROI Localization
2025-03-31 14:18:01Alexis Guichemerre, Soufiane Belharbi, Mohammadhadi Shateri, Luke McCaffrey, Eric GrangerAbstract
Weakly supervised object localization (WSOL) methods allow training models to classify images and localize ROIs. WSOL only requires low-cost image-class annotations yet provides a visually interpretable classifier, which is important in histology image analysis. Standard WSOL methods rely on class activation mapping (CAM) methods to produce spatial localization maps according to a single- or two-step strategy. While both strategies have made significant progress, they still face several limitations with histology images. Single-step methods can easily result in under- or over-activation due to the limited visual ROI saliency in histology images and the limited localization cues. They also face the well-known issue of asynchronous convergence between classification and localization tasks. The two-step approach is sub-optimal because it is tied to a frozen classifier, limiting the capacity for localization. Moreover, these methods also struggle when applied to out-of-distribution (OOD) datasets. In this paper, a multi-task approach for WSOL is introduced for simultaneous training of both tasks to address the asynchronous convergence problem. In particular, localization is performed in the pixel-feature space of an image encoder that is shared with classification. This allows learning discriminant features and accurate delineation of foreground/background regions to support ROI localization and image classification. We propose PixelCAM, a cost-effective foreground/background pixel-wise classifier in the pixel-feature space that allows for spatial object localization. PixelCAM is trained using pixel pseudo-labels collected from a pretrained WSOL model. Both image and pixel-wise classifiers are trained simultaneously using standard gradient descent. In addition, our pixel classifier can easily be integrated into CNN- and transformer-based architectures without any modifications.
Abstract (translated)
弱监督目标定位（WSOL）方法允许在对图像进行分类的同时，训练模型来定位感兴趣区域（ROI）。与标准的完全标注数据集相比，WSOL仅需要低成本的图像类别标签，这为医学组织学图像分析提供了重要的视觉可解释性。传统的WSOL方法依赖于类激活图（CAM）技术，通过单步或两步策略生成空间定位地图。 虽然这两种策略都取得了显著进展，但在处理组织学图像时仍面临诸多局限。单步法由于组织学图像中ROI的可视性有限和局部化线索不足，容易导致过强或过弱的激活现象，并且也存在分类任务与定位任务之间异步收敛的问题。两步方法则因为依赖于固定的分类器而难以优化，限制了其在定位方面的表现能力。此外，这些传统方法当应用于分布外（OOD）数据集时也会遇到困难。 本文介绍了一种用于WSOL的多任务方法，通过同时训练两个任务以解决异步收敛问题。具体来说，在一个与分类共享权重的图像编码器像素特征空间中执行定位操作，从而允许学习区分性特征并精确地划分前景/背景区域，支持ROI定位和图像分类。 我们提出了一种名为PixelCAM的方法，它是在像素特征空间中的成本效益型前景/背景像素级分类器，可以用于实现空间物体的定位。PixelCAM通过使用从预训练WSOL模型收集到的像素伪标签进行训练，并且图像和像素级分类器同时采用标准梯度下降法训练。此外，我们的像素分类器能够轻松集成到基于CNN或transformer架构中，无需任何修改。 这种方法不仅在组织学图像上表现良好，在处理分布外数据集时也显示出更优的鲁棒性和适用性。
URL
https://arxiv.org/abs/2503.24135
PDF
-
Bayesian Predictive Coding
2025-03-31 12:40:50Alexander Tschantz, Magnus Koudahl, Hampus Linander, Lancelot Da Costa, Conor Heins, Jeff Beck, Christopher BuckleyAbstract
Predictive coding (PC) is an influential theory of information processing in the brain, providing a biologically plausible alternative to backpropagation. It is motivated in terms of Bayesian inference, as hidden states and parameters are optimised via gradient descent on variational free energy. However, implementations of PC rely on maximum \textit{a posteriori} (MAP) estimates of hidden states and maximum likelihood (ML) estimates of parameters, limiting their ability to quantify epistemic uncertainty. In this work, we investigate a Bayesian extension to PC that estimates a posterior distribution over network parameters. This approach, termed Bayesian Predictive coding (BPC), preserves the locality of PC and results in closed-form Hebbian weight updates. Compared to PC, our BPC algorithm converges in fewer epochs in the full-batch setting and remains competitive in the mini-batch setting. Additionally, we demonstrate that BPC offers uncertainty quantification comparable to existing methods in Bayesian deep learning, while also improving convergence properties. Together, these results suggest that BPC provides a biologically plausible method for Bayesian learning in the brain, as well as an attractive approach to uncertainty quantification in deep learning.
Abstract (translated)
预测编码（Predictive Coding，简称PC）是大脑信息处理的一个有影响力的理论模型，提供了一种生物上合理的反向传播算法替代方案。它基于贝叶斯推理原理，在变分自由能量的梯度下降过程中优化隐含状态和参数。然而，现有的预测编码实现主要依赖于最大后验概率（MAP）来估计隐藏状态，并使用最大似然法（ML）来估计参数，这限制了它们量化知识不确定性（epistemic uncertainty）的能力。 在这项工作中，我们探讨了一种贝叶斯扩展的预测编码方法，该方法可以对网络参数进行后验分布的估计。这种方法被称为贝叶斯预测编码（Bayesian Predictive Coding, 简称BPC），它保持了预测编码的局部性，并且生成了封闭形式的赫布式权重更新公式。 相比传统的预测编码算法，在全批次设置下，我们的BPC算法在更少的迭代周期内收敛；而在小批量设置下，该方法仍然具有竞争力。此外，我们还展示了BPC能够在不确定性量化方面与现有的贝叶斯深度学习方法相媲美，并且还能改进其收敛性能。 这些结果表明，BPC为大脑中的贝叶斯学习提供了一种生物上合理的方案，同时在深度学习中提供了评估不确定性的有吸引力的方法。
URL
https://arxiv.org/abs/2503.24016
PDF
-
Towards Understanding the Optimization Mechanisms in Deep Learning
2025-03-29 08:46:13Binchuan Qi, Wei Gong, Li LiAbstract
In this paper, we adopt a probability distribution estimation perspective to explore the optimization mechanisms of supervised classification using deep neural networks. We demonstrate that, when employing the Fenchel-Young loss, despite the non-convex nature of the fitting error with respect to the model's parameters, global optimal solutions can be approximated by simultaneously minimizing both the gradient norm and the structural error. The former can be controlled through gradient descent algorithms. For the latter, we prove that it can be managed by increasing the number of parameters and ensuring parameter independence, thereby providing theoretical insights into mechanisms such as over-parameterization and random initialization. Ultimately, the paper validates the key conclusions of the proposed method through empirical results, illustrating its practical effectiveness.
Abstract (translated)
在这篇论文中，我们从概率分布估计的角度出发，探讨了使用深度神经网络进行监督分类的优化机制。我们展示了当采用Fenchel-Young损失函数时，尽管相对于模型参数拟合误差是非凸的，但仍可以通过同时最小化梯度范数和结构误差来近似全局最优解。前者可通过梯度下降算法加以控制。对于后者，我们证明通过增加参数数量并确保参数独立性可以进行管理，从而为诸如过度参数化和随机初始化等机制提供了理论见解。最终，本文通过实证结果验证了所提出方法的关键结论，并展示了其实际有效性。
URL
https://arxiv.org/abs/2503.23016
PDF
-
DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimation
2025-03-29 06:27:22Chengkun Wei, Weixian Li, Gong Chen, Wenzhi ChenAbstract
Differentially Private Stochastic Gradient Descent (DP-SGD) is a widely adopted technique for privacy-preserving deep learning. A critical challenge in DP-SGD is selecting the optimal clipping threshold C, which involves balancing the trade-off between clipping bias and noise magnitude, incurring substantial privacy and computing overhead during hyperparameter tuning. In this paper, we propose Dynamic Clipping DP-SGD (DC-SGD), a framework that leverages differentially private histograms to estimate gradient norm distributions and dynamically adjust the clipping threshold C. Our framework includes two novel mechanisms: DC-SGD-P and DC-SGD-E. DC-SGD-P adjusts the clipping threshold based on a percentile of gradient norms, while DC-SGD-E minimizes the expected squared error of gradients to optimize C. These dynamic adjustments significantly reduce the burden of hyperparameter tuning C. The extensive experiments on various deep learning tasks, including image classification and natural language processing, show that our proposed dynamic algorithms achieve up to 9 times acceleration on hyperparameter tuning than DP-SGD. And DC-SGD-E can achieve an accuracy improvement of 10.62% on CIFAR10 than DP-SGD under the same privacy budget of hyperparameter tuning. We conduct rigorous theoretical privacy and convergence analyses, showing that our methods seamlessly integrate with the Adam optimizer. Our results highlight the robust performance and efficiency of DC-SGD, offering a practical solution for differentially private deep learning with reduced computational overhead and enhanced privacy guarantees.
Abstract (translated)
差分隐私随机梯度下降（DP-SGD）是用于保护隐私的深度学习中广泛采用的技术。在DP-SGD中，选择最优剪裁阈值C是一个关键挑战，它需要在剪裁偏差和噪声大小之间找到平衡点，在超参数调整过程中这会产生大量的计算开销和隐私成本。本文提出了动态剪裁差分隐私随机梯度下降（DC-SGD），这是一种利用差分隐私直方图来估计梯度范数分布，并动态调整剪裁阈值C的框架。我们的框架包括两种新的机制：DC-SGD-P 和 DC-SGD-E。其中，DC-SGD-P 基于梯度范数的某个百分位数值来调整剪裁阈值，而DC-SGD-E 则通过最小化梯度平方误差的期望值来优化C值。这些动态调整显著减少了超参数调优过程中关于选择最优C值的工作量。 在各种深度学习任务上的广泛实验（包括图像分类和自然语言处理）表明，我们的动态算法比DP-SGD快9倍。特别是在给定相同的隐私预算的情况下，在CIFAR10数据集上，DC-SGD-E 达到了比 DP-SGD 高出10.62%的准确性。 我们进行了严格的理论分析，证明了我们的方法可以与Adam优化器无缝集成，并展示了 DC-SGD 在隐私保护和收敛性方面的稳健性能及高效性。这些结果表明，在减少计算开销的同时增强隐私保证方面，DC-SGD 提供了一种实用解决方案，适用于差分隐私深度学习的广泛应用场景。
URL
https://arxiv.org/abs/2503.22988
PDF
-
Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent
Abstract
We show that the behavior of stochastic gradient descent is related to Bayesian statistics by showing that SGD is effectively diffusion on a fractal landscape, where the fractal dimension can be accounted for in a purely Bayesian way. By doing this we show that SGD can be regarded as a modified Bayesian sampler which accounts for accessibility constraints induced by the fractal structure of the loss landscape. We verify our results experimentally by examining the diffusion of weights during training. These results offer insight into the factors which determine the learning process, and seemingly answer the question of how SGD and purely Bayesian sampling are related.
Abstract (translated)
我们展示了随机梯度下降的行为与贝叶斯统计有关，通过证明SGD在分形景观上的扩散过程，其中分形维度可以通过纯粹的贝叶斯方法来解释。通过这种方式，我们表明SGD可以被视为一种修改后的贝叶斯采样器，该采样器考虑了由损失景观的分形结构引起的可达性限制。我们通过实验验证了这一结果，方法是观察训练过程中权重的扩散情况。这些成果提供了对影响学习过程的因素的理解，并似乎回答了SGD与纯粹的贝叶斯采样之间关系的问题。
URL
https://arxiv.org/abs/2503.22478
PDF
-
SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representations
2025-03-28 14:14:19Krispin Wandel, Hesheng WangAbstract
Semantic correspondence made tremendous progress through the recent advancements of large vision models (LVM). While these LVMs have been shown to reliably capture local semantics, the same can currently not be said for capturing global geometric relationships between semantic object regions. This problem leads to unreliable performance for semantic correspondence between images with extreme view variation. In this work, we aim to leverage monocular depth estimates to capture these geometric relationships for more robust and data-efficient semantic correspondence. First, we introduce a simple but effective method to build 3D object-class representations from monocular depth estimates and LVM features using a sparsely annotated image correspondence dataset. Second, we formulate an alignment energy that can be minimized using gradient descent to obtain an alignment between the 3D object-class representation and the object-class instance in the input RGB-image. Our method achieves state-of-the-art matching accuracy in multiple categories on the challenging SPair-71k dataset, increasing the PCK@0.1 score by more than 10 points on three categories and overall by 3.3 points from 85.6% to 88.9%. Additional resources and code are available at this https URL.
Abstract (translated)
语义对应关系通过最近大型视觉模型（LVM）的进步取得了巨大进展。尽管这些LVM已被证明能够可靠地捕捉局部语义，但目前还不能说它们能有效地捕捉不同语义对象区域之间的全局几何关系。这个问题导致在处理视角变化极大的图像时，语义对应的性能变得不可靠。在这项工作中，我们旨在利用单目深度估计来捕捉这些几何关系，以实现更稳健且数据效率更高的语义对应。 首先，我们介绍了一种简单但有效的方法，可以使用稀疏标注的图像对应数据集从单目深度估计和LVM特征构建3D对象类别表示。其次，我们提出了一个对齐能量公式，该公式可以通过梯度下降法最小化来获得3D对象类表示与输入RGB图像中对象实例之间的对齐。 我们的方法在具有挑战性的SPair-71k数据集的多个类别上实现了最先进的匹配准确性，在三个类别上的PCK@0.1分数提高了超过10个百分点，并且整体从85.6%提高到了88.9%，增加了3.3个百分点。更多资源和代码可以在[这里](https://this_https_URL.com/)找到。 请注意，最后一个URL应被替换为实际提供的链接地址。
URL
https://arxiv.org/abs/2503.22462
PDF
