Gradient_Descent
-
Towards the Resistance of Neural Network Watermarking to Fine-tuning
2025-05-02 05:11:17Ling Tang, Yuefeng Chen, Hui Xue, Quanshi ZhangAbstract
This paper proves a new watermarking method to embed the ownership information into a deep neural network (DNN), which is robust to fine-tuning. Specifically, we prove that when the input feature of a convolutional layer only contains low-frequency components, specific frequency components of the convolutional filter will not be changed by gradient descent during the fine-tuning process, where we propose a revised Fourier transform to extract frequency components from the convolutional filter. Additionally, we also prove that these frequency components are equivariant to weight scaling and weight permutations. In this way, we design a watermark module to encode the watermark information to specific frequency components in a convolutional filter. Preliminary experiments demonstrate the effectiveness of our method.
Abstract (translated)
本文提出了一种新的水印技术，可以将所有权信息嵌入到深度神经网络（DNN）中，并且这种水印在微调过程中是鲁棒的。具体而言，我们证明了当卷积层的输入特征仅包含低频分量时，在梯度下降过程中的微调阶段，该层的卷积滤波器的特定频率成分不会发生变化。为此，我们提出了一种修订版的傅里叶变换方法来从卷积滤波器中提取这些频率成分。此外，我们也证明了这些频率成分对于权重缩放和权重置换是等变（equivariant）的。通过这种方式，我们设计了一个水印模块，可以将水印信息编码到卷积滤波器的特定频率分量中。初步实验表明我们的方法的有效性。
URL
https://arxiv.org/abs/2505.01007
PDF
-
How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias
2025-05-02 00:07:35Ruiquan Huang, Yingbin Liang, Jing YangAbstract
Language recognition tasks are fundamental in natural language processing (NLP) and have been widely used to benchmark the performance of large language models (LLMs). These tasks also play a crucial role in explaining the working mechanisms of transformers. In this work, we focus on two representative tasks in the category of regular language recognition, known as `even pairs' and `parity check', the aim of which is to determine whether the occurrences of certain subsequences in a given sequence are even. Our goal is to explore how a one-layer transformer, consisting of an attention layer followed by a linear layer, learns to solve these tasks by theoretically analyzing its training dynamics under gradient descent. While even pairs can be solved directly by a one-layer transformer, parity check need to be solved by integrating Chain-of-Thought (CoT), either into the inference stage of a transformer well-trained for the even pairs task, or into the training of a one-layer transformer. For both problems, our analysis shows that the joint training of attention and linear layers exhibits two distinct phases. In the first phase, the attention layer grows rapidly, mapping data sequences into separable vectors. In the second phase, the attention layer becomes stable, while the linear layer grows logarithmically and approaches in direction to a max-margin hyperplane that correctly separates the attention layer outputs into positive and negative samples, and the loss decreases at a rate of $O(1/t)$. Our experiments validate those theoretical results.
Abstract (translated)
语言识别任务是自然语言处理（NLP）中的基本任务，广泛用于评估大型语言模型（LLMs）的性能。这些任务在解释变压器的工作机制方面也扮演着重要角色。在这项工作中，我们专注于两类常规语言识别任务的代表性任务：即“偶数对”和“奇偶校验”，其目的是确定给定序列中某些子序列出现次数是否为偶数。我们的目标是通过理论分析单层变压器（由注意力层和线性层组成）在梯度下降训练过程中的学习动态，来探索它如何解决这些任务。“偶数对”可以直接使用单层变压器解决，“奇偶校验”则需要结合“链式思维”(Chain-of-Thought, CoT) 方法来解决。对于已经经过“偶数对”任务良好训练的变压器推理阶段中集成CoT，或是在单层变压器的训练过程中直接整合。 针对这两个问题，我们的分析表明注意力和线性层的联合训练展示了两个不同的阶段。在第一阶段，注意力层迅速增长，并将数据序列映射为可分向量。而在第二阶段，注意力层变得稳定，而线性层则以对数方式增长，在方向上逐渐接近于能够正确分离注意力层输出（将样本分为正负两类）的最大间隔超平面，并且损失函数的减少速度是$O(1/t)$。 实验结果验证了这些理论成果。
URL
https://arxiv.org/abs/2505.00926
PDF
-
The Dark Side of Digital Twins: Adversarial Attacks on AI-Driven Water Forecasting
2025-04-28 22:34:11Mohammadhossein Homaei, Victor Gonzalez Morales, Oscar Mogollon-Gutierrez, Andres CaroAbstract
Digital twins (DTs) are improving water distribution systems by using real-time data, analytics, and prediction models to optimize operations. This paper presents a DT platform designed for a Spanish water supply network, utilizing Long Short-Term Memory (LSTM) networks to predict water consumption. However, machine learning models are vulnerable to adversarial attacks, such as the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD). These attacks manipulate critical model parameters, injecting subtle distortions that degrade forecasting accuracy. To further exploit these vulnerabilities, we introduce a Learning Automata (LA) and Random LA-based approach that dynamically adjusts perturbations, making adversarial attacks more difficult to detect. Experimental results show that this approach significantly impacts prediction reliability, causing the Mean Absolute Percentage Error (MAPE) to rise from 26% to over 35%. Moreover, adaptive attack strategies amplify this effect, highlighting cybersecurity risks in AI-driven DTs. These findings emphasize the urgent need for robust defenses, including adversarial training, anomaly detection, and secure data pipelines.
Abstract (translated)
数字孪生（DT）通过使用实时数据、分析和预测模型来优化运营，正在改善水分配系统。本文介绍了一个专为西班牙供水网络设计的数字孪生平台，并利用长短期记忆（LSTM）网络来预测用水量。然而，机器学习模型容易受到诸如快速梯度符号法（FGSM）和投影梯度下降（PGD）等对抗性攻击的影响。这些攻击操纵关键的模型参数，在不影响数据外观的情况下注入细微扰动以降低预测准确性。为了进一步利用这些漏洞，我们引入了一种基于学习自动机（LA）和随机学习自动机的方法，该方法能够动态调整扰动，使对抗性攻击更难被检测到。实验结果表明，这种方法显著影响了预测的可靠性，导致平均绝对百分比误差（MAPE）从26%上升至超过35%。此外，适应性的攻击策略放大了这种效果，突显了AI驱动的数字孪生中的网络安全风险。这些发现强调了建立强大的防御措施的迫切需要，包括对抗性训练、异常检测和安全的数据管道。
URL
https://arxiv.org/abs/2504.20295
PDF
-
FedAvgen: Metadata for Model Aggregation In Communication Systems
2025-04-28 13:11:32Anthony Kiggundu, Dennis Krummacker, Hans D. SchottenAbstract
To improve business efficiency and minimize costs, Artificial Intelligence (AI) practitioners have adopted a shift from formulating models from scratch towards sharing pretrained models. The pretrained models are then aggregated into a global model with higher generalization capabilities, which is afterwards distributed to the client devices. This approach is known as federated learning and inherently utilizes different techniques to select the candidate client models averaged to obtain the global model. This approach, in the case of communication systems, faces challenges arising from the existential diversity in device profiles. The multiplicity in profiles motivates our conceptual assessment of a metaheuristic algorithm (FedAvgen), which relates each pretrained model with its weight space as metadata, to a phenotype and genotype, respectively. This parent-child genetic evolution characterizes the global averaging step in federated learning. We then compare the results of our approach to two widely adopted baseline federated learning algorithms like Federated Averaging (FedAvg) and Federated Stochastic Gradient Descent (FedSGD).
Abstract (translated)
为了提高业务效率并减少成本，人工智能（AI）从业者已从零开始构建模型转向共享预训练模型。这些预训练模型会被聚合为一个具有更强泛化能力的全局模型，然后分发到客户端设备上。这种方法被称为联邦学习，并且内在地利用不同的技术来选择用于获得全局模型的候选客户端模型的平均值。在通信系统中，该方法面临着由于设备配置文件存在多样性而产生的挑战。这种配置文件的多样化促使我们对一种元启发式算法（FedAvgen）进行概念评估，该算法将每个预训练模型与其权重空间分别视为元数据中的表型和基因型。这种父代-子代遗传进化特征化了联邦学习中的全局平均步骤。然后，我们将我们的方法的结果与两个广泛采用的基线联邦学习算法，如联邦平均（FedAvg）和联邦随机梯度下降（FedSGD），进行比较。
URL
https://arxiv.org/abs/2505.05486
PDF
-
Smart Placement, Faster Robots -- A Comparison of Algorithms for Robot Base-Pose Optimization
2025-04-28 08:36:17Matthias Mayer, Matthias AlthoffAbstract
Robotic automation is a key technology that increases the efficiency and flexibility of manufacturing processes. However, one of the challenges in deploying robots in novel environments is finding the optimal base pose for the robot, which affects its reachability and deployment cost. Yet, the existing research for automatically optimizing the base pose of robots has not been compared. We address this problem by optimizing the base pose of industrial robots with Bayesian optimization, exhaustive search, genetic algorithms, and stochastic gradient descent and find that all algorithms can reduce the cycle time for various evaluated tasks in synthetic and real-world environments. Stochastic gradient descent shows superior performance with regard to success rate solving over 90% of our real-world tasks, while genetic algorithms show the lowest final costs. All benchmarks and implemented methods are available as baselines against which novel approaches can be compared.
Abstract (translated)
机器人自动化是提高制造过程效率和灵活性的关键技术。然而，在新的环境中部署机器人时，一个挑战在于找到机器人的最优基座姿态（base pose），这会影响其可达性和部署成本。不过，现有的关于自动优化机器人基座姿态的研究尚未进行过比较研究。我们通过使用贝叶斯优化、穷举搜索、遗传算法和随机梯度下降来优化工业机器人的基座姿态，并发现所有这些算法都能在合成环境和真实世界环境中减少各种任务的循环时间。随机梯度下降法表现出色，成功解决了超过90%的真实世界任务；而遗传算法则显示出最低的最终成本。所有的基准测试和实现方法都可以作为新方法进行比较的基础。
URL
https://arxiv.org/abs/2504.19577
PDF
-
Sharp higher order convergence rates for the Adam optimizer
2025-04-28 02:17:50Steffen Dereich, Arnulf Jentzen, Adrian RiekertAbstract
Gradient descent based optimization methods are the methods of choice to train deep neural networks in machine learning. Beyond the standard gradient descent method, also suitable modified variants of standard gradient descent involving acceleration techniques such as the momentum method and/or adaptivity techniques such as the RMSprop method are frequently considered optimization methods. These days the most popular of such sophisticated optimization schemes is presumably the Adam optimizer that has been proposed in 2014 by Kingma and Ba. A highly relevant topic of research is to investigate the speed of convergence of such optimization methods. In particular, in 1964 Polyak showed that the standard gradient descent method converges in a neighborhood of a strict local minimizer with rate (x - 1)(x + 1)^{-1} while momentum achieves the (optimal) strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} where x \in (1,\infty) is the condition number (the ratio of the largest and the smallest eigenvalue) of the Hessian of the objective function at the local minimizer. It is the key contribution of this work to reveal that Adam also converges with the strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} while RMSprop only converges with the convergence rate (x - 1)(x + 1)^{-1}.
Abstract (translated)
基于梯度下降的优化方法是训练深度神经网络在机器学习中的首选方法。除了标准的梯度下降方法外，还包括适合加速技术（如动量法）和适应性技术（如RMSprop方法）的标准梯度下降的各种变体，这些技术也常被考虑作为优化方法。近年来，最流行的此类复杂优化方案可能是Kingma和Ba在2014年提出的Adam优化器。研究这样一个优化方法的收敛速度是一个非常重要的课题。特别地，在1964年Polyak证明了标准梯度下降方法在一个严格局部极小值附近的收敛率为(x - 1)(x + 1)^{-1}，而动量法则实现了最优且更快速的收敛率(\sqrt{x} - 1)(\sqrt{x} + 1)^{-1}，其中x \in (1,\infty)是目标函数在局部极小值点处Hessian矩阵的最大特征值与最小特征值比（即条件数）。这项工作的关键贡献在于揭示了Adam优化器也以严格更快的收敛率(\sqrt{x} - 1)(\sqrt{x} + 1)^{-1}进行收敛，而RMSprop仅以(x - 1)(x + 1)^{-1}的收敛速率进行收敛。
URL
https://arxiv.org/abs/2504.19426
PDF
-
NANO-SLAM : Natural Gradient Gaussian Approximation for Vehicle SLAM
2025-04-27 11:05:15Tianyi Zhang, Wenhan Cao, Chang Liu, Feihong Zhang, Wei Wu, Shengbo Eben LiAbstract
Accurate localization is a challenging task for autonomous vehicles, particularly in GPS-denied environments such as urban canyons and tunnels. In these scenarios, simultaneous localization and mapping (SLAM) offers a more robust alternative to GPS-based positioning, enabling vehicles to determine their position using onboard sensors and surrounding environment's landmarks. Among various vehicle SLAM approaches, Rao-Blackwellized particle filter (RBPF) stands out as one of the most widely adopted methods due to its efficient solution with logarithmic complexity relative to the map size. RBPF approximates the posterior distribution of the vehicle pose using a set of Monte Carlo particles through two main steps: sampling and importance weighting. The key to effective sampling lies in solving a distribution that closely approximates the posterior, known as the sampling distribution, to accelerate convergence. Existing methods typically derive this distribution via linearization, which introduces significant approximation errors due to the inherent nonlinearity of the system. To address this limitation, we propose a novel vehicle SLAM method called \textit{N}atural Gr\textit{a}dient Gaussia\textit{n} Appr\textit{o}ximation (NANO)-SLAM, which avoids linearization errors by modeling the sampling distribution as the solution to an optimization problem over Gaussian parameters and solving it using natural gradient descent. This approach improves the accuracy of the sampling distribution and consequently enhances localization performance. Experimental results on the long-distance Sydney Victoria Park vehicle SLAM dataset show that NANO-SLAM achieves over 50\% improvement in localization accuracy compared to the most widely used vehicle SLAM algorithms, with minimal additional computational cost.
Abstract (translated)
准确的定位是自主车辆面临的挑战，特别是在没有GPS信号的环境中，如城市峡谷和隧道。在这种情况下，同时定位与地图构建（SLAM）提供了一个比基于GPS的位置服务更为稳健的选择，使车辆能够利用车载传感器和周围环境中的地标来确定自己的位置。在众多车辆SLAM方法中，Rao-Blackwellized粒子滤波器（RBPF）因其对地图大小具有对数复杂度的高效解决方案而被广泛采用。RBPF通过两个主要步骤——采样和重要性加权——利用一组蒙特卡洛粒子来近似车辆姿态的后验分布。有效采样的关键在于求解一个能够紧密逼近后验分布的分布，即采样分布，从而加速收敛过程。现有方法通常通过线性化导出该分布，但由于系统固有的非线性特性，这会导致显著的近似误差。 为了解决这一局限性，我们提出了一种新的车辆SLAM方法——自然梯度高斯逼近（NANO）-SLAM，这种方法通过将采样分布建模为一个关于高斯参数优化问题的解，并利用自然梯度下降法进行求解来避免线性化误差。这种做法提高了采样分布的准确性，从而提升了定位性能。 在长距离悉尼维多利亚公园车辆SLAM数据集上的实验结果显示，NANO-SLAM相较于最广泛使用的车辆SLAM算法，在定位精度上实现了超过50%的改进，并且几乎不需要增加额外的计算成本。
URL
https://arxiv.org/abs/2504.19195
PDF
-
Generalization Capability for Imitation Learning
2025-04-25 17:59:59Yixiao WangAbstract
Imitation learning holds the promise of equipping robots with versatile skills by learning from expert demonstrations. However, policies trained on finite datasets often struggle to generalize beyond the training distribution. In this work, we present a unified perspective on the generalization capability of imitation learning, grounded in both information theorey and data distribution property. We first show that the generalization gap can be upper bounded by (i) the conditional information bottleneck on intermediate representations and (ii) the mutual information between the model parameters and the training dataset. This characterization provides theoretical guidance for designing effective training strategies in imitation learning, particularly in determining whether to freeze, fine-tune, or train large pretrained encoders (e.g., vision-language models or vision foundation models) from scratch to achieve better generalization. Furthermore, we demonstrate that high conditional entropy from input to output induces a flatter likelihood landscape, thereby reducing the upper bound on the generalization gap. In addition, it shortens the stochastic gradient descent (SGD) escape time from sharp local minima, which may increase the likelihood of reaching global optima under fixed optimization budgets. These insights explain why imitation learning often exhibits limited generalization and underscore the importance of not only scaling the diversity of input data but also enriching the variability of output labels conditioned on the same input.
Abstract (translated)
模仿学习通过从专家演示中学习，承诺赋予机器人多种技能。然而，基于有限数据集训练的策略往往难以超越训练分布进行泛化。在这项工作中，我们提出了一种统一的观点来探讨模仿学习的泛化能力，该观点既以信息理论为依据，也考虑了数据分布特性。首先，我们表明泛化差距可以被上界限定为（i）中间表示上的条件信息瓶颈和（ii）模型参数与训练数据集之间的互信息。这一特征提供了设计有效训练策略的理论指导，在模仿学习中特别有助于确定是否冻结、微调或从头开始训练大规模预训练编码器（例如，视觉-语言模型或视觉基础模型），以实现更好的泛化能力。 此外，我们还证明了输入到输出方向上的高条件熵会诱导出更平坦的概率景观，从而减少泛化差距的上界。同时，它缩短了随机梯度下降（SGD）从尖锐局部最小值逃逸的时间，在固定优化预算下可能增加达到全局最优解的可能性。 这些见解解释了为什么模仿学习通常表现出有限的泛化能力，并强调不仅需要扩大输入数据的多样性，而且还需要丰富给定相同输入时输出标签的变化性。
URL
https://arxiv.org/abs/2504.18538
PDF
-
Gradient Descent as a Shrinkage Operator for Spectral Bias
Abstract
We generalize the connection between activation function and spline regression/smoothing and characterize how this choice may influence spectral bias within a 1D shallow network. We then demonstrate how gradient descent (GD) can be reinterpreted as a shrinkage operator that masks the singular values of a neural network's Jacobian. Viewed this way, GD implicitly selects the number of frequency components to retain, thereby controlling the spectral bias. An explicit relationship is proposed between the choice of GD hyperparameters (learning rate & number of iterations) and bandwidth (the number of active components). GD regularization is shown to be effective only with monotonic activation functions. Finally, we highlight the utility of non-monotonic activation functions (sinc, Gaussian) as iteration-efficient surrogates for spectral bias.
Abstract (translated)
我们扩展了激活函数与样条回归/平滑之间的联系，并描述了这种选择如何可能影响一维浅层网络中的谱偏差（spectral bias）。然后，我们展示了梯度下降（GD）可以重新解释为一种收缩算子，该算子屏蔽神经网络雅可比矩阵的奇异值。从这个角度来看，GD隐式地选择了要保留的频率分量的数量，从而控制了谱偏差。提出了梯度下降超参数的选择（学习率和迭代次数）与带宽（活动组件数量）之间的显式关系。证明了GD正则化仅在使用单调激活函数时有效。最后，我们强调非单调激活函数（如sinc和高斯）作为具有较高迭代效率的谱偏差替代方案的实用性。
URL
https://arxiv.org/abs/2504.18207
PDF
-
Gradient-Optimized Fuzzy Classifier: A Benchmark Study Against State-of-the-Art Models
2025-04-22 20:47:06Magnus Sieverding, Nathan Steffen, Kelly CohenAbstract
This paper presents a performance benchmarking study of a Gradient-Optimized Fuzzy Inference System (GF) classifier against several state-of-the-art machine learning models, including Random Forest, XGBoost, Logistic Regression, Support Vector Machines, and Neural Networks. The evaluation was conducted across five datasets from the UCI Machine Learning Repository, each chosen for their diversity in input types, class distributions, and classification complexity. Unlike traditional Fuzzy Inference Systems that rely on derivative-free optimization methods, the GF leverages gradient descent to significantly improving training efficiency and predictive performance. Results demonstrate that the GF model achieved competitive, and in several cases superior, classification accuracy while maintaining high precision and exceptionally low training times. In particular, the GF exhibited strong consistency across folds and datasets, underscoring its robustness in handling noisy data and variable feature sets. These findings support the potential of gradient optimized fuzzy systems as interpretable, efficient, and adaptable alternatives to more complex deep learning models in supervised learning tasks.
Abstract (translated)
本文介绍了一项针对梯度优化模糊推理系统（GF）分类器的性能基准研究，将其与几种最先进的机器学习模型进行了对比，包括随机森林、XGBoost、逻辑回归、支持向量机和神经网络。评估在来自UCI机器学习仓库的五个不同数据集上进行，这些数据集因输入类型多样、类别分布各异及分类难度的不同而被选中。 不同于依赖无导数优化方法的传统模糊推理系统，GF利用梯度下降显著提升了训练效率和预测性能。实验结果显示，在保持高精度的同时，GF模型实现了与其它基准模型相当甚至更优的分类准确率，并且具备极低的训练时间。此外，GF在各个折集（folds）和数据集上表现出了强一致性，这突显了其处理噪声数据及变化特征集合时的鲁棒性。 这些发现支持了梯度优化模糊系统作为解释性强、高效且适应性强的替代方案，可应用于监督学习任务中复杂的深度学习模型。
URL
https://arxiv.org/abs/2504.16263
PDF
-
Efficient Temporal Consistency in Diffusion-Based Video Editing with Adaptor Modules: A Theoretical Framework
2025-04-22 16:28:35Xinyuan Song, Yangfan He, Sida Li, Jianhui Wang, Hongyang He, Xinhang Yuan, Ruoyu Wang, Jiaqi Chen, Keqin Li, Kuan Lu, Menghao Huo, Binxu Li, Pei LiuAbstract
Adapter-based methods are commonly used to enhance model performance with minimal additional complexity, especially in video editing tasks that require frame-to-frame consistency. By inserting small, learnable modules into pretrained diffusion models, these adapters can maintain temporal coherence without extensive retraining. Approaches that incorporate prompt learning with both shared and frame-specific tokens are particularly effective in preserving continuity across frames at low training cost. In this work, we want to provide a general theoretical framework for adapters that maintain frame consistency in DDIM-based models under a temporal consistency loss. First, we prove that the temporal consistency objective is differentiable under bounded feature norms, and we establish a Lipschitz bound on its gradient. Second, we show that gradient descent on this objective decreases the loss monotonically and converges to a local minimum if the learning rate is within an appropriate range. Finally, we analyze the stability of modules in the DDIM inversion procedure, showing that the associated error remains controlled. These theoretical findings will reinforce the reliability of diffusion-based video editing methods that rely on adapter strategies and provide theoretical insights in video generation tasks.
Abstract (translated)
基于适配器的方法通常用于在最小增加复杂度的情况下增强模型性能，特别是在需要帧到帧一致性的视频编辑任务中。通过将小型可学习模块插入预训练的扩散模型中，这些适配器可以在无需大量重新训练的情况下保持时间一致性。结合提示学习（使用共享和特定于每一帧的标记）的方法特别有效，在低训练成本下跨帧维持连续性。 在这项工作中，我们希望为在DDIM（Denoising Diffusion Implicit Models）基于模型中保持帧一致性的适配器提供一个通用理论框架，并在此基础上引入时间一致性损失。首先，我们将证明当特征范数有界时，时间一致性目标是可微的，并且我们建立了其梯度上的Lipschitz边界。其次，我们展示在适当的学习率范围内进行梯度下降会使该目标的损失单调减少并最终收敛到局部最小值。最后，我们分析了DDIM反演过程中的模块稳定性，证明相关误差保持受控。 这些理论发现将增强基于扩散方法依赖适配器策略进行视频编辑技术的可靠性，并为视频生成任务提供理论见解。
URL
https://arxiv.org/abs/2504.16016
PDF
-
Cascade IPG Observer for Underwater Robot State Estimation
2025-04-21 17:10:10Kaustubh Joshi, Tianchen Liu, Nikhil ChopraAbstract
This paper presents a novel cascade nonlinear observer framework for inertial state estimation. It tackles the problem of intermediate state estimation when external localization is unavailable or in the event of a sensor outage. The proposed observer comprises two nonlinear observers based on a recently developed iteratively preconditioned gradient descent (IPG) algorithm. It takes the inputs via an IMU preintegration model where the first observer is a quaternion-based IPG. The output for the first observer is the input for the second observer, estimating the velocity and, consequently, the position. The proposed observer is validated on a public underwater dataset and a real-world experiment using our robot platform. The estimation is compared with an extended Kalman filter (EKF) and an invariant extended Kalman filter (InEKF). Results demonstrate that our method outperforms these methods regarding better positional accuracy and lower variance.
Abstract (translated)
本文提出了一种新的级联非线性观测器框架，用于惯性状态估计。该方法解决了在外部定位不可用或传感器故障情况下中间状态估计的问题。所提出的观测器由两个基于最近开发的迭代预条件梯度下降（IPG）算法的非线性观测器组成。它通过IMU（惯性测量单元）预积分模型接收输入，其中第一个观测器是基于四元数的IPG。第一个观测器的输出作为第二个观测器的输入，用于估计速度和位置。该提出的观测器在公开的水下数据集和我们机器人平台上的真实世界实验中进行了验证，并与扩展卡尔曼滤波（EKF）和不变扩展卡尔曼滤波（InEKF）方法进行比较。结果显示，我们的方法在位置精度和方差方面优于这些方法。
URL
https://arxiv.org/abs/2504.15235
PDF
-
HyperFlow: Gradient-Free Emulation of Few-Shot Fine-Tuning
2025-04-21 03:04:38Donggyun Kim, Chanwoo Kim, Seunghoon HongAbstract
While test-time fine-tuning is beneficial in few-shot learning, the need for multiple backpropagation steps can be prohibitively expensive in real-time or low-resource scenarios. To address this limitation, we propose an approach that emulates gradient descent without computing gradients, enabling efficient test-time adaptation. Specifically, we formulate gradient descent as an Euler discretization of an ordinary differential equation (ODE) and train an auxiliary network to predict the task-conditional drift using only the few-shot support set. The adaptation then reduces to a simple numerical integration (e.g., via the Euler method), which requires only a few forward passes of the auxiliary network -- no gradients or forward passes of the target model are needed. In experiments on cross-domain few-shot classification using the Meta-Dataset and CDFSL benchmarks, our method significantly improves out-of-domain performance over the non-fine-tuned baseline while incurring only 6\% of the memory cost and 0.02\% of the computation time of standard fine-tuning, thus establishing a practical middle ground between direct transfer and fully fine-tuned approaches.
Abstract (translated)
虽然测试时的微调在少量样本学习中是有益的，但在实时或资源受限场景下，需要进行多次反向传播步骤的成本可能过高。为了解决这一限制，我们提出了一种方法，该方法能够在不计算梯度的情况下模拟梯度下降，从而实现高效的测试时适应性调整。具体而言，我们将梯度下降表述为普通微分方程（ODE）的欧拉离散化，并训练一个辅助网络仅使用少量样本支持集来预测任务条件下的漂移。随后的适应过程简化为简单的数值积分（例如通过欧拉方法），这只需要进行几次辅助网络的前向传递——不需要计算梯度或目标模型的前向传递。 在跨域少量样本分类实验中，我们使用了Meta-Dataset和CDFSL基准测试，并且我们的方法显著提高了跨域性能，相比于不进行微调的基础线而言，在仅占用标准微调内存成本6%和0.02%计算时间的情况下实现了这一改进。因此，该方法在直接迁移与完全微调之间建立了一种实用的中间途径。 简言之，这项研究提供了一个有效的方法来优化少量样本学习中的测试时适应性调整过程，在减少资源消耗的同时提高跨域模型性能。
URL
https://arxiv.org/abs/2504.15323
PDF
-
Visual Prompting for One-shot Controllable Video Editing without Inversion
2025-04-19 16:00:47Zhengbo Zhang, Yuxi Zhou, Duo Peng, Joo-Hwee Lim, Zhigang Tu, De Wen Soh, Lin Geng FooAbstract
One-shot controllable video editing (OCVE) is an important yet challenging task, aiming to propagate user edits that are made -- using any image editing tool -- on the first frame of a video to all subsequent frames, while ensuring content consistency between edited frames and source frames. To achieve this, prior methods employ DDIM inversion to transform source frames into latent noise, which is then fed into a pre-trained diffusion model, conditioned on the user-edited first frame, to generate the edited video. However, the DDIM inversion process accumulates errors, which hinder the latent noise from accurately reconstructing the source frames, ultimately compromising content consistency in the generated edited frames. To overcome it, our method eliminates the need for DDIM inversion by performing OCVE through a novel perspective based on visual prompting. Furthermore, inspired by consistency models that can perform multi-step consistency sampling to generate a sequence of content-consistent images, we propose a content consistency sampling (CCS) to ensure content consistency between the generated edited frames and the source frames. Moreover, we introduce a temporal-content consistency sampling (TCS) based on Stein Variational Gradient Descent to ensure temporal consistency across the edited frames. Extensive experiments validate the effectiveness of our approach.
Abstract (translated)
一次性可控视频编辑（OCVE）是一项重要但具有挑战性的任务，旨在将用户在视频第一帧上使用任何图像编辑工具进行的修改传播到所有后续帧，并确保编辑后的帧与原始帧之间保持内容一致性。为了实现这一目标，先前的方法采用DDIM逆向过程将源帧转换为潜在噪声，然后将其输入预训练的扩散模型，在此模型中以用户编辑的第一帧作为条件来生成编辑后的视频。然而，DDIM逆向过程中会累积错误，这阻碍了潜在噪声准确重建原始帧的能力，最终影响了生成的编辑帧的内容一致性。 为了克服这一问题，我们的方法通过一种新的基于视觉提示的方法消除了对DDIM逆向过程的需求，并在此基础上执行OCVE。此外，受到能够进行多步一致性采样以生成一系列内容一致图像的一致性模型的启发，我们提出了内容一致性采样（CCS），以确保生成的编辑帧与源帧之间的内容一致性。另外，我们引入了一种基于Stein变分梯度下降的时间-内容一致性采样（TCS）方法，以确保编辑后的帧之间的时间连续性。大量的实验验证了我们这种方法的有效性。
URL
https://arxiv.org/abs/2504.14335
PDF
-
A mean teacher algorithm for unlearning of language models
2025-04-18 00:34:19Yegor KlochkovAbstract
One of the goals of language model unlearning is to reduce memorization of selected text instances while retaining the model's general abilities. Despite various proposed methods, reducing memorization of large datasets without noticeable degradation in model utility remains challenging. In this paper, we investigate the mean teacher algorithm (Tarvainen & Valpola, 2017), a simple proximal optimization method from continual learning literature that gradually modifies the teacher model. We show that the mean teacher can approximate a trajectory of a slow natural gradient descent (NGD), which inherently seeks low-curvature updates that are less likely to degrade the model utility. While slow NGD can suffer from vanishing gradients, we introduce a new unlearning loss called "negative log-unlikelihood" (NLUL) that avoids this problem. We show that the combination of mean teacher and NLUL improves some metrics on the MUSE benchmarks (Shi et al., 2024).
Abstract (translated)
语言模型“遗忘”（unlearning）的一个目标是减少对选定文本实例的记忆，同时保持模型的一般能力。尽管已提出了多种方法，但在不显著降低模型实用性的情况下减少大型数据集的内存仍是一个挑战。在本文中，我们研究了均值教师算法（Tarvainen & Valpola, 2017），这是一种源自连续学习文献中的简单邻近优化方法，它逐步修改教师模型。我们展示了均值教师可以逼近慢速自然梯度下降（NGD）的轨迹，而这种更新本质上寻求低曲率更新，这些更新不太可能降低模型实用性。虽然慢速NGD可能会遇到消失的梯度问题，但我们引入了一种新的遗忘损失方法，称为“负对数非可能性”（NLUL），它可以避免这个问题。我们展示了均值教师与NLUL结合后，在MUSE基准测试（Shi et al., 2024）中的某些指标有所提高。
URL
https://arxiv.org/abs/2504.13388
PDF
-
Recursive Deep Inverse Reinforcement Learning
2025-04-17 17:39:35Paul Ghanem, Michael Potter, Owen Howell, Pau Closas, Alireza Ramezani, Deniz Erdogmus, Robert Platt, Tales ImbiribaAbstract
Inferring an adversary's goals from exhibited behavior is crucial for counterplanning and non-cooperative multi-agent systems in domains like cybersecurity, military, and strategy games. Deep Inverse Reinforcement Learning (IRL) methods based on maximum entropy principles show promise in recovering adversaries' goals but are typically offline, require large batch sizes with gradient descent, and rely on first-order updates, limiting their applicability in real-time scenarios. We propose an online Recursive Deep Inverse Reinforcement Learning (RDIRL) approach to recover the cost function governing the adversary actions and goals. Specifically, we minimize an upper bound on the standard Guided Cost Learning (GCL) objective using sequential second-order Newton updates, akin to the Extended Kalman Filter (EKF), leading to a fast (in terms of convergence) learning algorithm. We demonstrate that RDIRL is able to recover cost and reward functions of expert agents in standard and adversarial benchmark tasks. Experiments on benchmark tasks show that our proposed approach outperforms several leading IRL algorithms.
Abstract (translated)
从对手的行为推断其目标对于网络安全、军事和策略游戏中的反规划及非合作多智能体系统至关重要。基于最大熵原则的深度逆向强化学习（IRL）方法在恢复对手的目标方面展现出潜力，但这些方法通常为离线操作，需要大量批次大小以及使用梯度下降，并且依赖于一阶更新，这限制了它们在实时场景中的应用。我们提出了一种在线递归深度逆向强化学习（RDIRL）的方法来恢复控制对手行动和目标的成本函数。具体来说，我们通过类似扩展卡尔曼滤波器（EKF）的顺序二阶牛顿更新最小化标准引导成本学习（GCL）目标上的上限，从而获得一个快速收敛的学习算法。我们在标准和对抗性基准任务中展示了RDIRL能够恢复专家代理的成本和奖励函数的能力。在基准任务上的实验表明，我们提出的方法优于几种领先的IRL算法。 总结来说，这项研究提供了一种新的方法来实时地理解和预测对手的行为，在网络安全、军事策略以及多智能体系统等领域具有重要应用价值。
URL
https://arxiv.org/abs/2504.13241
PDF
-
Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Prediction
2025-04-17 14:05:33Dubing Chen, Huan Zheng, Jin Fang, Xingping Dong, Xianfei Li, Wenlong Liao, Tao He, Pai Peng, Jianbing ShenAbstract
We present GDFusion, a temporal fusion method for vision-based 3D semantic occupancy prediction (VisionOcc). GDFusion opens up the underexplored aspects of temporal fusion within the VisionOcc framework, focusing on both temporal cues and fusion strategies. It systematically examines the entire VisionOcc pipeline, identifying three fundamental yet previously overlooked temporal cues: scene-level consistency, motion calibration, and geometric complementation. These cues capture diverse facets of temporal evolution and make distinct contributions across various modules in the VisionOcc framework. To effectively fuse temporal signals across heterogeneous representations, we propose a novel fusion strategy by reinterpreting the formulation of vanilla RNNs. This reinterpretation leverages gradient descent on features to unify the integration of diverse temporal information, seamlessly embedding the proposed temporal cues into the network. Extensive experiments on nuScenes demonstrate that GDFusion significantly outperforms established baselines. Notably, on Occ3D benchmark, it achieves 1.4\%-4.8\% mIoU improvements and reduces memory consumption by 27\%-72\%.
Abstract (translated)
我们提出了GDFusion，这是一种用于基于视觉的三维语义占用预测（VisionOcc）的时间融合方法。GDFusion 开拓了在 VisionOcc 框架内未充分探索的时间融合方面，专注于时间线索和融合策略。它系统地审查了整个 VisionOcc 管道，并识别出三个基础但以前被忽视的时间线索：场景级一致性、运动校准和几何补充。这些线索捕捉到时间演变的各个方面，在 VisionOcc 框架的不同模块中做出了独特的贡献。 为了有效融合异构表示中的时序信号，我们提出了一种新的融合策略，通过重新解释标准 RNN 的公式来实现这一点。这种重新解读利用了特征上的梯度下降，以统一不同类型的时序信息的集成，并将提出的时序线索无缝地嵌入到网络中。 在 nuScenes 数据集上进行的广泛实验表明，GDFusion 显著优于现有的基准方法。特别是在 Occ3D 基准测试上，它实现了 1.4%-4.8% 的 mIoU 改进，并减少了 27%-72% 的内存消耗。
URL
https://arxiv.org/abs/2504.12959
PDF
-
Quantum Computing Supported Adversarial Attack-Resilient Autonomous Vehicle Perception Module for Traffic Sign Classification
2025-04-17 05:08:08Reek Majumder, Mashrur Chowdhury, Sakib Mahmud Khan, Zadid Khan, Fahim Ahmad, Frank Ngeni, Gurcan Comert, Judith Mwakalonge, Dimitra MichalakaAbstract
Deep learning (DL)-based image classification models are essential for autonomous vehicle (AV) perception modules since incorrect categorization might have severe repercussions. Adversarial attacks are widely studied cyberattacks that can lead DL models to predict inaccurate output, such as incorrectly classified traffic signs by the perception module of an autonomous vehicle. In this study, we create and compare hybrid classical-quantum deep learning (HCQ-DL) models with classical deep learning (C-DL) models to demonstrate robustness against adversarial attacks for perception modules. Before feeding them into the quantum system, we used transfer learning models, alexnet and vgg-16, as feature extractors. We tested over 1000 quantum circuits in our HCQ-DL models for projected gradient descent (PGD), fast gradient sign attack (FGSA), and gradient attack (GA), which are three well-known untargeted adversarial approaches. We evaluated the performance of all models during adversarial attacks and no-attack scenarios. Our HCQ-DL models maintain accuracy above 95\% during a no-attack scenario and above 91\% for GA and FGSA attacks, which is higher than C-DL models. During the PGD attack, our alexnet-based HCQ-DL model maintained an accuracy of 85\% compared to C-DL models that achieved accuracies below 21\%. Our results highlight that the HCQ-DL models provide improved accuracy for traffic sign classification under adversarial settings compared to their classical counterparts.
Abstract (translated)
基于深度学习（DL）的图像分类模型对于自动驾驶汽车（AV）感知模块至关重要，因为错误的分类可能会导致严重后果。对抗性攻击是广泛研究的网络攻击之一，可以导致DL模型预测出不准确的结果，例如自动驾驶车辆感知模块中交通标志被误分类的情况。在这项研究中，我们创建并比较了混合经典-量子深度学习（HCQ-DL）模型与传统深度学习（C-DL）模型，以展示其在对抗性攻击下为感知模块提供的鲁棒性。为了将它们输入到量子系统之前，我们使用迁移学习模型alexnet和vgg-16作为特征提取器。我们在我们的HCQ-DL模型中测试了超过1000个量子电路，针对投影梯度下降（PGD）、快速梯度符号攻击（FGSA）和梯度攻击（GA），这三种著名的非目标对抗性方法进行了测试。我们评估了所有模型在遭遇对抗性和无攻击场景下的性能表现。我们的HCQ-DL模型在无攻击场景下保持超过95%的准确率，在面对GA和FGSA攻击时准确率维持在91%以上，这一数值高于传统DL模型的表现。在PGD攻击期间，我们基于alexnet的HCQ-DL模型能够保持85%的准确性，而C-DL模型的准确率则低于21%。我们的研究结果表明，在对抗性设置下，HCQ-DL模型为交通标志分类提供的精度优于其传统DL模型对应者。
URL
https://arxiv.org/abs/2504.12644
PDF
-
PG-DPIR: An efficient plug-and-play method for high-count Poisson-Gaussian inverse problems
2025-04-14 16:23:15Maud Biquard, Marie Chabert, Florence Genin, Christophe Latry, Thomas OberlinAbstract
Poisson-Gaussian noise describes the noise of various imaging systems thus the need of efficient algorithms for Poisson-Gaussian image restoration. Deep learning methods offer state-of-the-art performance but often require sensor-specific training when used in a supervised setting. A promising alternative is given by plug-and-play (PnP) methods, which consist in learning only a regularization through a denoiser, allowing to restore images from several sources with the same network. This paper introduces PG-DPIR, an efficient PnP method for high-count Poisson-Gaussian inverse problems, adapted from DPIR. While DPIR is designed for white Gaussian noise, a naive adaptation to Poisson-Gaussian noise leads to prohibitively slow algorithms due to the absence of a closed-form proximal operator. To address this, we adapt DPIR for the specificities of Poisson-Gaussian noise and propose in particular an efficient initialization of the gradient descent required for the proximal step that accelerates convergence by several orders of magnitude. Experiments are conducted on satellite image restoration and super-resolution problems. High-resolution realistic Pleiades images are simulated for the experiments, which demonstrate that PG-DPIR achieves state-of-the-art performance with improved efficiency, which seems promising for on-ground satellite processing chains.
Abstract (translated)
泊松-高斯噪声描述了各种成像系统中的噪声特性，因此需要高效的算法来解决泊松-高斯图像恢复问题。深度学习方法提供了最先进的性能，但当在监督设置下使用时通常需要特定传感器的训练。一种有前景的替代方案是由插件播放（PnP）方法提供的，这些方法仅通过去噪器学习正则化项，从而能够利用同一网络从多个来源恢复图像。本文介绍了PG-DPIR，这是一种高效的PnP方法，用于解决高计数泊松-高斯逆问题，基于DPIR进行了改进。虽然DPIR是为白色高斯噪声设计的，但直接将其应用于泊松-高斯噪声会导致算法运行速度极其缓慢，因为缺乏封闭形式的近似算子。为了应对这一挑战，我们针对Poisson-Gaussian噪声的特点对DPIR进行了调整，并特别提出了一种高效的梯度下降初始化方法，用于加速近似步骤中的收敛速度，提高了几个数量级的速度。实验在卫星图像恢复和超分辨率问题上进行。利用高分辨率的现实Pleiades图像模拟了实验数据，结果表明PG-DPIR实现了最先进的性能并提高了效率，这似乎对于地面卫星处理链来说前景广阔。
URL
https://arxiv.org/abs/2504.10375
PDF
-
Differentially Private 2D Human Pose Estimation
2025-04-14 12:50:37Kaushik Bhargav Sivangi, Idris Zakariyya, Paul Henderson, Fani DeligianniAbstract
Human pose estimation (HPE) has become essential in numerous applications including healthcare, activity recognition, and human-computer interaction. However, the privacy implications of processing sensitive visual data present significant deployment barriers in critical domains. While traditional anonymization techniques offer limited protection and often compromise data utility for broader motion analysis, Differential Privacy (DP) provides formal privacy guarantees but typically degrades model performance when applied naively. In this work, we present the first differentially private 2D human pose estimation (2D-HPE) by applying Differentially Private Stochastic Gradient Descent (DP-SGD) to this task. To effectively balance privacy with performance, we adopt Projected DP-SGD (PDP-SGD), which projects the noisy gradients to a low-dimensional subspace. Additionally, we adapt TinyViT, a compact and efficient vision transformer for coordinate classification in HPE, providing a lightweight yet powerful backbone that enhances privacy-preserving deployment feasibility on resource-limited devices. Our approach is particularly valuable for multimedia interpretation tasks, enabling privacy-safe analysis and understanding of human motion across diverse visual media while preserving the semantic meaning required for downstream applications. Comprehensive experiments on the MPII Human Pose Dataset demonstrate significant performance enhancement with PDP-SGD achieving 78.48% PCKh@0.5 at a strict privacy budget ($\epsilon=0.2$), compared to 63.85% for standard DP-SGD. This work lays foundation for privacy-preserving human pose estimation in real-world, sensitive applications.
Abstract (translated)
人体姿态估计（HPE）在医疗保健、活动识别和人机交互等多个领域变得至关重要。然而，处理敏感视觉数据的隐私问题构成了关键应用领域的重大部署障碍。传统的匿名化技术提供的保护有限，并且常常会损害用于广泛运动分析的数据效用。相比之下，差分隐私（DP）提供了正式的隐私保证，但当直接应用于模型训练时通常会导致性能下降。在这项工作中，我们首次提出了一个具有差分隐私保障的二维人体姿态估计方法（2D-HPE），通过将差分私有随机梯度下降法（DP-SGD）应用到这个任务中来实现。为了有效地平衡隐私与性能之间的关系，我们采用了投影式差分私有随机梯度下降法（PDP-SGD），这种方法将带有噪声的梯度投影到了一个低维子空间内。 此外，我们将TinyViT这种紧凑且高效的视觉变换器应用到坐标分类中，并将其用于人体姿态估计，提供了一个轻量级但功能强大的骨干网络，增强了资源受限设备上隐私保护部署的可能性。我们的方法对于多媒体解读任务特别有价值，它能够在不损害下游应用程序所需语义意义的情况下，在各种视觉媒体中实现对人类运动的隐私安全分析与理解。 在MPII人体姿态数据集上的全面实验表明，使用PDP-SGD的方法取得了显著的性能提升，当严格的隐私预算为$\epsilon=0.2$时，实现了78.48%的PCKh@0.5指标，而标准DP-SGD仅达到63.85%。这项工作为现实世界中敏感应用中的差分私有人体姿态估计奠定了基础。
URL
https://arxiv.org/abs/2504.10190
PDF
