Gradient_Descent
-
Exploring Adversarial Watermarking in Transformer-Based Models: Transferability and Robustness Against Defense Mechanism for Medical Images
2025-06-05 15:03:38Rifat Sadik, Tanvir Rahman, Arpan Bhattacharjee, Bikash Chandra Halder, Ismail HossainAbstract
Deep learning models have shown remarkable success in dermatological image analysis, offering potential for automated skin disease diagnosis. Previously, convolutional neural network(CNN) based architectures have achieved immense popularity and success in computer vision (CV) based task like skin image recognition, generation and video analysis. But with the emergence of transformer based models, CV tasks are now are nowadays carrying out using these models. Vision Transformers (ViTs) is such a transformer-based models that have shown success in computer vision. It uses self-attention mechanisms to achieve state-of-the-art performance across various tasks. However, their reliance on global attention mechanisms makes them susceptible to adversarial perturbations. This paper aims to investigate the susceptibility of ViTs for medical images to adversarial watermarking-a method that adds so-called imperceptible perturbations in order to fool models. By generating adversarial watermarks through Projected Gradient Descent (PGD), we examine the transferability of such attacks to CNNs and analyze the performance defense mechanism -- adversarial training. Results indicate that while performance is not compromised for clean images, ViTs certainly become much more vulnerable to adversarial attacks: an accuracy drop of as low as 27.6%. Nevertheless, adversarial training raises it up to 90.0%.
Abstract (translated)
深度学习模型在皮肤病图像分析中表现出显著的成功，为自动化皮肤疾病诊断提供了潜在可能。此前，基于卷积神经网络（CNN）的架构在计算机视觉（CV）任务如皮肤图像识别、生成和视频分析方面取得了巨大的流行度和成功。然而，随着基于变换器（Transformer）模型的出现，如今许多CV任务正在使用这些模型来完成。Vision Transformers (ViTs) 是一种基于变换器的模型，在计算机视觉领域表现出色。它通过自注意力机制在各种任务中达到了最先进的性能。但是，它们依赖于全局注意机制使得其容易受到对抗性扰动的影响。 本文旨在研究医学图像中的ViT对对抗性水印（即添加所谓的不显眼干扰以误导模型的方法）的敏感性。通过使用Projected Gradient Descent (PGD) 生成对抗性水印，我们考察了此类攻击在CNN上的转移性和分析防御机制——对抗训练的表现。 结果表明，在处理干净图像时性能并未受损，但ViT确实对对抗性攻击更加脆弱：准确率下降至低至27.6%。然而，通过对抗训练可以将其提升到90.0%。
URL
https://arxiv.org/abs/2506.06389
PDF
-
Comparative performance of ensemble models in predicting dental provider types: insights from fee-for-service data
2025-06-04 21:55:27Mohammad Subhi Al-Batah, Muhyeeddin Alqaraleh, Mowafaq Salem Alzboon, Abdullah AlouraniAbstract
Dental provider classification plays a crucial role in optimizing healthcare resource allocation and policy planning. Effective categorization of providers, such as standard rendering providers and safety net clinic (SNC) providers, enhances service delivery to underserved populations. This study aimed to evaluate the performance of machine learning models in classifying dental providers using a 2018 dataset. A dataset of 24,300 instances with 20 features was analyzed, including beneficiary and service counts across fee-for-service (FFS), Geographic Managed Care, and Pre-Paid Health Plans. Providers were categorized by delivery system and patient age groups (0-20 and 21+). Despite 38.1% missing data, multiple machine learning algorithms were tested, including k-Nearest Neighbors (kNN), Decision Trees, Support Vector Machines (SVM), Stochastic Gradient Descent (SGD), Random Forest, Neural Networks, and Gradient Boosting. A 10-fold cross-validation approach was applied, and models were evaluated using AUC, classification accuracy (CA), F1-score, precision, and recall. Neural Networks achieved the highest AUC (0.975) and CA (94.1%), followed by Random Forest (AUC: 0.948, CA: 93.0%). These models effectively handled imbalanced data and complex feature interactions, outperforming traditional classifiers like Logistic Regression and SVM. Advanced machine learning techniques, particularly ensemble and deep learning models, significantly enhance dental workforce classification. Their integration into healthcare analytics can improve provider identification and resource distribution, benefiting underserved populations.
Abstract (translated)
牙科提供者分类在优化卫生资源分配和政策规划中起着关键作用。有效的提供者分类，如标准服务提供者和服务网诊所（SNC）提供者，增强了向边缘化人群提供的服务质量。本研究旨在评估机器学习模型在使用2018年数据集对牙科提供者进行分类时的性能表现。分析的数据集中包含了24,300个实例和20个特征，包括按费用报销（FFS）、地理管理医疗护理以及预付健康计划的服务受益人和服务数量统计。根据服务系统和患者年龄组（0-20岁及21岁以上）对提供者进行了分类。 尽管有高达38.1%的数据缺失，研究中测试了多种机器学习算法，包括k最近邻(kNN)、决策树、支持向量机(SVM)、随机梯度下降(SGD)、随机森林、神经网络和梯度提升。采用了一种十倍交叉验证的方法，并使用AUC（曲线下面积）、分类准确率(CA)、F1分数、精确度和召回率对模型进行评估。 在所有模型中，神经网络获得了最高的AUC值（0.975）和最高分类准确率（CA：94.1%），其次是随机森林（AUC: 0.948, CA: 93.0%）。这些模型能够有效处理不平衡数据及复杂的特征交互，优于传统的分类器如逻辑回归和支持向量机。先进的机器学习技术，尤其是集成和深度学习模型，在牙科工作力量的分类中表现出色。 将这些高级方法整合到医疗保健数据分析中，可以提高提供者的识别效率和服务资源分配的合理性，从而更好地服务于边缘化群体。
URL
https://arxiv.org/abs/2506.04479
PDF
-
Classifying Dental Care Providers Through Machine Learning with Features Ranking
2025-06-04 21:45:40Mohammad Subhi Al-Batah, Mowafaq Salem Alzboon, Muhyeeddin Alqaraleh, Mohammed Hasan Abu-Arqoub, Rashiq Rafiq MarieAbstract
This study investigates the application of machine learning (ML) models for classifying dental providers into two categories - standard rendering providers and safety net clinic (SNC) providers - using a 2018 dataset of 24,300 instances with 20 features. The dataset, characterized by high missing values (38.1%), includes service counts (preventive, treatment, exams), delivery systems (FFS, managed care), and beneficiary demographics. Feature ranking methods such as information gain, Gini index, and ANOVA were employed to identify critical predictors, revealing treatment-related metrics (TXMT_USER_CNT, TXMT_SVC_CNT) as top-ranked features. Twelve ML models, including k-Nearest Neighbors (kNN), Decision Trees, Support Vector Machines (SVM), Stochastic Gradient Descent (SGD), Random Forest, Neural Networks, and Gradient Boosting, were evaluated using 10-fold cross-validation. Classification accuracy was tested across incremental feature subsets derived from rankings. The Neural Network achieved the highest accuracy (94.1%) using all 20 features, followed by Gradient Boosting (93.2%) and Random Forest (93.0%). Models showed improved performance as more features were incorporated, with SGD and ensemble methods demonstrating robustness to missing data. Feature ranking highlighted the dominance of treatment service counts and annotation codes in distinguishing provider types, while demographic variables (AGE_GROUP, CALENDAR_YEAR) had minimal impact. The study underscores the importance of feature selection in enhancing model efficiency and accuracy, particularly in imbalanced healthcare datasets. These findings advocate for integrating feature-ranking techniques with advanced ML algorithms to optimize dental provider classification, enabling targeted resource allocation for underserved populations.
Abstract (translated)
这项研究探讨了使用机器学习（ML）模型将牙科提供者分类为两类——标准服务提供商和安全网诊所（SNC）提供者的方法。该研究基于2018年的数据集进行，包含24,300个实例和20个特征，其中包括高缺失值率（38.1%）。数据集中包含了各种服务计数（预防、治疗、检查）、交付系统（FFS、管理护理）以及受益人的人口统计信息。 为了识别关键预测因素，研究使用了诸如信息增益、Gini指数和ANOVA等特征排名方法。结果显示，与治疗相关的指标（如TXMT_USER_CNT和TXMT_SVC_CNT）是最重要的特征。 研究评估了12种机器学习模型，包括k-近邻算法（kNN）、决策树、支持向量机（SVM）、随机梯度下降（SGD）、随机森林、神经网络以及梯度提升等。这些模型通过十折交叉验证进行了测试，并在不同特征子集上评估了分类准确性。 实验结果表明，当使用所有20个特征时，神经网络实现了最高的准确率（94.1%），其次是梯度提升算法（93.2%）和随机森林（93.0%）。随着纳入更多特征，模型表现有所改善。随机梯度下降和支持向量机等方法在处理缺失数据方面表现出较强的鲁棒性。 此外，特征排名表明治疗服务计数以及注释代码是区分不同提供商类型的关键因素，而人口统计变量（如AGE_GROUP和CALENDAR_YEAR）影响较小。 该研究强调了特征选择对于提高模型效率和准确性的关键作用，特别是在不平衡的医疗保健数据集中。这些发现支持将特征排名技术与高级机器学习算法相结合以优化牙科提供者分类，并促进对服务不足人群进行针对性资源分配的方法。
URL
https://arxiv.org/abs/2506.04474
PDF
-
Target Semantics Clustering via Text Representations for Robust Universal Domain Adaptation
2025-06-04 03:11:53Weinan He, Zilei Wang, Yixin ZhangAbstract
Universal Domain Adaptation (UniDA) focuses on transferring source domain knowledge to the target domain under both domain shift and unknown category shift. Its main challenge lies in identifying common class samples and aligning them. Current methods typically obtain target domain semantics centers from an unconstrained continuous image representation space. Due to domain shift and the unknown number of clusters, these centers often result in complex and less robust alignment algorithm. In this paper, based on vision-language models, we search for semantic centers in a semantically meaningful and discrete text representation space. The constrained space ensures almost no domain bias and appropriate semantic granularity for these centers, enabling a simple and robust adaptation algorithm. Specifically, we propose TArget Semantics Clustering (TASC) via Text Representations, which leverages information maximization as a unified objective and involves two stages. First, with the frozen encoders, a greedy search-based framework is used to search for an optimal set of text embeddings to represent target semantics. Second, with the search results fixed, encoders are refined based on gradient descent, simultaneously achieving robust domain alignment and private class clustering. Additionally, we propose Universal Maximum Similarity (UniMS), a scoring function tailored for detecting open-set samples in UniDA. Experimentally, we evaluate the universality of UniDA algorithms under four category shift scenarios. Extensive experiments on four benchmarks demonstrate the effectiveness and robustness of our method, which has achieved state-of-the-art performance.
Abstract (translated)
通用领域适应（UniDA）专注于在域偏移和未知类别迁移的情况下，将源领域的知识转移到目标领域。其主要挑战在于识别共同类别的样本并进行对齐。目前的方法通常从一个不受限制的连续图像表示空间中获取目标域语义中心，由于域偏移以及簇的数量未知，这些中心往往导致复杂的、鲁棒性较差的对齐算法。 在本文中，基于视觉-语言模型，我们在一个具有意义且离散的文本表示空间中搜索语义中心。这种受约束的空间确保了几乎没有领域偏差，并为这些中心提供了适当的语义粒度，从而能够实现简单且稳健的适应算法。具体来说，我们提出了通过文本表示进行目标语义聚类（TASC），该方法利用信息最大化作为统一的目标，并涉及两个阶段。首先，在冻结编码器的情况下，使用基于贪心搜索的框架来寻找最优的一组文本嵌入以表征目标领域的语义。其次，在固定了搜索结果之后，根据梯度下降对编码器进行细化，同时实现稳健的域对齐和私有类别聚类。 此外，我们提出了为检测UniDA中的开放集样本量身定制的评分函数——通用最大相似性（UniMS）。 在实验中，我们在四种类别迁移场景下评估了UniDA算法的普适性。广泛的基准测试表明，我们的方法具有有效性和鲁棒性，并取得了最先进的性能表现。
URL
https://arxiv.org/abs/2506.03521
PDF
-
EWGN: Elastic Weight Generation and Context Switching in Deep Learning
2025-06-01 21:59:53Shriraj P. Sawant, Krishna P. MiyapuramAbstract
The ability to learn and retain a wide variety of tasks is a hallmark of human intelligence that has inspired research in artificial general intelligence. Continual learning approaches provide a significant step towards achieving this goal. It has been known that task variability and context switching are challenging for learning in neural networks. Catastrophic forgetting refers to the poor performance on retention of a previously learned task when a new task is being learned. Switching between different task contexts can be a useful approach to mitigate the same by preventing the interference between the varying task weights of the network. This paper introduces Elastic Weight Generative Networks (EWGN) as an idea for context switching between two different tasks. The proposed EWGN architecture uses an additional network that generates the weights of the primary network dynamically while consolidating the weights learned. The weight generation is input-dependent and thus enables context switching. Using standard computer vision datasets, namely MNIST and fashion-MNIST, we analyse the retention of previously learned task representations in Fully Connected Networks, Convolutional Neural Networks, and EWGN architectures with Stochastic Gradient Descent and Elastic Weight Consolidation learning algorithms. Understanding dynamic weight generation and context-switching ability can be useful in enabling continual learning for improved performance.
Abstract (translated)
学习和保留广泛任务的能力是人类智能的一个标志，这激发了对通用人工智能的研究。连续学习的方法为实现这一目标提供了重要的步骤。已知任务多样性和上下文切换对于神经网络中的学习构成了挑战。“灾难性遗忘”指的是当学习新任务时，以前所学任务的表现会变得糟糕的现象。在不同任务的背景下进行切换可以通过防止网络中变化的任务权重之间的干扰来减轻这种问题。 本文介绍了一种名为弹性权重生成网络（Elastic Weight Generative Networks, EWGN）的方法，用于在两个不同的任务之间实现上下文切换。提出的EWGN架构使用一个附加网络动态地生成主网络的权重，并同时巩固已经学习到的权重。该权重生成是基于输入依赖的，因此可以实现上下文切换。 我们利用标准计算机视觉数据集（如MNIST和时尚-MNIST），分析在全连接网络、卷积神经网络以及采用随机梯度下降法和弹性权重巩固算法的EWGN架构中之前学习到的任务表示的保留情况。理解动态权重生成和上下文转换能力对于实现连续学习，提高性能很有帮助。
URL
https://arxiv.org/abs/2506.02065
PDF
-
Constrained Stein Variational Gradient Descent for Robot Perception, Planning, and Identification
2025-05-31 14:52:34Griffin Tabor, Tucker HermansAbstract
Many core problems in robotics can be framed as constrained optimization problems. Often on these problems, the robotic system has uncertainty, or it would be advantageous to identify multiple high quality feasible solutions. To enable this, we present two novel frameworks for applying principles of constrained optimization to the new variational inference algorithm Stein variational gradient descent. Our general framework supports multiple types of constrained optimizers and can handle arbitrary constraints. We demonstrate on a variety of problems that we are able to learn to approximate distributions without violating constraints. Specifically, we show that we can build distributions of: robot motion plans that exactly avoid collisions, robot arm joint angles on the SE(3) manifold with exact table placement constraints, and object poses from point clouds with table placement constraints.
Abstract (translated)
机器人技术中许多核心问题可以被表述为约束优化问题。通常，在这些问题上，机器人系统存在不确定性，或者识别多个高质量的可行解会更有优势。为了实现这一目标，我们提出了两个新颖的框架，将约束优化原则应用于新的变分推理算法Stein变分梯度下降法中。我们的通用框架支持多种类型的约束优化器，并且可以处理任意类型约束条件。我们在各种问题上展示了能够学习到在不违反约束条件下近似分布的能力。具体来说，我们证明了我们可以构建以下分布： - 精确避免碰撞的机器人运动计划 - 在SE(3)流形上的机械臂关节角度，具有精确的桌面放置约束条件 - 具有桌面放置约束的来自点云的对象姿态 这些框架使得在处理包含复杂约束条件的问题时更加灵活和高效。
URL
https://arxiv.org/abs/2506.00589
PDF
-
Scaling Textual Gradients via Sampling-Based Momentum
2025-05-31 05:35:45Zixin Ding, Junyuan Hong, Jiachen T. Wang, Zinan Lin, Zhangyang Wang, Yuxin ChenAbstract
As prompts play an increasingly critical role in large language models (LLMs), optimizing textual prompts has become a crucial challenge. The Textual Gradient Descent (TGD) framework has emerged as a promising data-driven approach that iteratively refines textual prompts using LLM - suggested updates (or textual gradients) over minibatches of training samples. In this paper, we empirically demonstrate that scaling the number of training examples initially improves but later degrades TGD's performance across multiple downstream NLP tasks. However, while data scaling improves results for most tasks, it also significantly increases the computational cost when leveraging LLMs. To address this, we draw inspiration from numerical gradient descent and propose Textual Stochastic Gradient Descent with Momentum (TSGD-M) - a method that facilitates scalable in-context learning by reweighting prompt sampling based on past batch distributions. Across nine NLP tasks spanning three domains - including BIG-Bench Hard (BBH), natural language understanding tasks, and reasoning tasks - TSGD-M significantly outperforms TGD baselines that do not incorporate reweighted sampling, while also reducing variance in most tasks.
Abstract (translated)
随着提示在大型语言模型（LLM）中扮演越来越重要的角色，优化文本提示已成为一个关键挑战。文本梯度下降（TGD）框架作为一种有前景的数据驱动方法应运而生，该方法通过使用LLM提供的更新建议（或称为文本梯度），在小批量训练样本上迭代地精炼文本提示。在这篇论文中，我们实证证明了增加初始训练示例的数量可以改善但最终会降低TGD框架在多个下游NLP任务上的性能表现。然而，在大多数情况下，数据量的扩展虽然提高了结果质量，但也显著增加了利用LLM进行计算的成本。为了解决这个问题，我们借鉴数值梯度下降的方法，并提出了带有动量的文本随机梯度下降（TSGD-M）——这种方法通过根据过去的批次分布重新加权提示采样来促进可扩展的上下文学习。在涵盖三大领域的九个NLP任务上（包括BIG-Bench Hard (BBH)、自然语言理解任务和推理任务），TSGD-M显著优于不使用重加权采样的TGD基线方法，并且在大多数任务中也降低了方差。 此段落概述了改进文本提示优化技术的研究成果，提出了新的TSGD-M方法来提高大型语言模型的性能，同时降低计算成本。
URL
https://arxiv.org/abs/2506.00400
PDF
-
3D Gaussian Splat Vulnerabilities
2025-05-30 22:21:22Matthew Hull, Haoyang Yang, Pratham Mehta, Mansi Phute, Aeree Cho, Haoran Wang, Matthew Lau, Wenke Lee, Willian T. Lunardi, Martin Andreoni, Polo ChauAbstract
With 3D Gaussian Splatting (3DGS) being increasingly used in safety-critical applications, how can an adversary manipulate the scene to cause harm? We introduce CLOAK, the first attack that leverages view-dependent Gaussian appearances - colors and textures that change with viewing angle - to embed adversarial content visible only from specific viewpoints. We further demonstrate DAGGER, a targeted adversarial attack directly perturbing 3D Gaussians without access to underlying training data, deceiving multi-stage object detectors e.g., Faster R-CNN, through established methods such as projected gradient descent. These attacks highlight underexplored vulnerabilities in 3DGS, introducing a new potential threat to robotic learning for autonomous navigation and other safety-critical 3DGS applications.
Abstract (translated)
随着3D高斯点阵（3D Gaussian Splatting，简称3DGS）在安全关键应用中的使用日益增多，对手如何才能通过操纵场景来造成危害？我们介绍了CLOAK，这是首个利用视图依赖的高斯外观——即随观察角度变化的颜色和纹理——将对抗性内容嵌入其中的技术。这种对抗性内容仅从特定视角可见。此外，我们还展示了DAGGER，这是一种直接扰动3D高斯分布而无需访问底层训练数据的定向对抗攻击，通过诸如投影梯度下降等方法欺骗多阶段物体检测器（例如Faster R-CNN）。这些攻击揭示了在3DGS中尚未被充分探索的安全漏洞，并为自主导航和其他安全关键型3DGS应用中的机器人学习引入了一个新的潜在威胁。
URL
https://arxiv.org/abs/2506.00280
PDF
-
Optimal Density Functions for Weighted Convolution in Learning Models
2025-05-30 12:36:36Simone Cammarasana, Giuseppe Patan\`eAbstract
The paper introduces the weighted convolution, a novel approach to the convolution for signals defined on regular grids (e.g., 2D images) through the application of an optimal density function to scale the contribution of neighbouring pixels based on their distance from the central pixel. This choice differs from the traditional uniform convolution, which treats all neighbouring pixels equally. Our weighted convolution can be applied to convolutional neural network problems to improve the approximation accuracy. Given a convolutional network, we define a framework to compute the optimal density function through a minimisation model. The framework separates the optimisation of the convolutional kernel weights (using stochastic gradient descent) from the optimisation of the density function (using DIRECT-L). Experimental results on a learning model for an image-to-image task (e.g., image denoising) show that the weighted convolution significantly reduces the loss (up to 53% improvement) and increases the test accuracy compared to standard convolution. While this method increases execution time by 11%, it is robust across several hyperparameters of the learning model. Future work will apply the weighted convolution to real-case 2D and 3D image convolutional learning problems.
Abstract (translated)
该论文介绍了加权卷积，这是一种针对在规则网格（例如二维图像）上定义的信号的新颖卷积方法。通过应用最优密度函数来调整邻近像素对中心像素贡献的比例，其依据是这些像素与中心像素的距离。这一选择不同于传统的均匀卷积法，后者对所有相邻像素一视同仁。我们的加权卷积可以应用于卷积神经网络问题中以提高逼近精度。 对于给定的卷积网络，我们通过最小化模型定义了一个框架来计算最优密度函数。该框架将卷积核权重的优化（使用随机梯度下降法）与密度函数的优化（使用DIRECT-L方法）分开进行。在学习模型处理图像到图像任务（例如图像去噪）时所取得的实验结果表明，加权卷积显著降低了损失（最高可达53%的改进），并提高了测试精度，相比标准卷积而言。 虽然此方法增加了11%的执行时间，但它对于学习模型中的多个超参数都是稳健的。未来的研究将把加权卷积应用于实际案例中的二维和三维图像卷积学习问题。
URL
https://arxiv.org/abs/2505.24527
PDF
-
Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
2025-05-29 09:55:29Chengli Tan, Yubo Zhou, Haishan Ye, Guang Dai, Junmin Liu, Zengjie Song, Jiangshe Zhang, Zixiang Zhao, Yunda Hao, Yong XuAbstract
Deep neural networks have been increasingly used in safety-critical applications such as medical diagnosis and autonomous driving. However, many studies suggest that they are prone to being poorly calibrated and have a propensity for overconfidence, which may have disastrous consequences. In this paper, unlike standard training such as stochastic gradient descent, we show that the recently proposed sharpness-aware minimization (SAM) counteracts this tendency towards overconfidence. The theoretical analysis suggests that SAM allows us to learn models that are already well-calibrated by implicitly maximizing the entropy of the predictive distribution. Inspired by this finding, we further propose a variant of SAM, coined as CSAM, to ameliorate model calibration. Extensive experiments on various datasets, including ImageNet-1K, demonstrate the benefits of SAM in reducing calibration error. Meanwhile, CSAM performs even better than SAM and consistently achieves lower calibration error than other approaches
Abstract (translated)
深度神经网络在诸如医疗诊断和自动驾驶等安全关键型应用中的使用越来越广泛。然而，许多研究表明这些模型容易出现校准不良的问题，并且倾向于过度自信，这可能会导致灾难性后果。在这篇论文中，不同于标准的训练方法如随机梯度下降，我们展示了最近提出的敏锐度感知最小化（SAM）可以对抗这种过度自信的趋势。理论分析表明，SAM允许我们学习出已经具有良好校准性的模型，通过隐式地最大化预测分布的熵来实现这一点。受到这一发现的启发，我们进一步提出了一种名为CSAM的SAM变体，以改进模型校准。在包括ImageNet-1K在内的多个数据集上的大量实验表明，SAM能够减少校准误差。与此同时，CSAM的表现甚至比SAM还要好，并且始终优于其他方法，在所有情况下均实现了更低的校准误差。
URL
https://arxiv.org/abs/2505.23866
PDF
-
On the performance of machine-learning assisted Monte Carlo in sampling from simple statistical physics models
2025-05-28 17:13:11Luca Maria Del Bono, Federico Ricci-Tersenghi, Francesco ZamponiAbstract
Recent years have seen a rise in the application of machine learning techniques to aid the simulation of hard-to-sample systems that cannot be studied using traditional methods. Despite the introduction of many different architectures and procedures, a wide theoretical understanding is still lacking, with the risk of suboptimal implementations. As a first step to address this gap, we provide here a complete analytic study of the widely-used Sequential Tempering procedure applied to a shallow MADE architecture for the Curie-Weiss model. The contribution of this work is twofold: firstly, we give a description of the optimal weights and of the training under Gradient Descent optimization. Secondly, we compare what happens in Sequential Tempering with and without the addition of local Metropolis Monte Carlo steps. We are thus able to give theoretical predictions on the best procedure to apply in this case. This work establishes a clear theoretical basis for the integration of machine learning techniques into Monte Carlo sampling and optimization.
Abstract (translated)
近年来，机器学习技术被广泛应用于模拟难以采样的系统，这些系统的研究无法通过传统方法进行。尽管已经引入了许多不同的架构和程序，但仍然缺乏广泛的理论理解，存在次优实现的风险。作为弥补这一缺口的第一步，我们在这里对广泛应用的顺序退火（Sequential Tempering）过程进行了完整分析，该过程应用于浅层MADE（Masked Autoencoder for Distribution Estimation）架构来模拟Curie-Weiss模型。 这项工作的贡献有两方面：首先，我们描述了最优权重以及在梯度下降优化下的训练情况。其次，我们将带局部Metropolis蒙特卡罗步骤的顺序退火过程与不包含这些步骤的过程进行了比较。因此，我们可以对这种情况提供理论预测，指出最佳应用程序是什么。 这项工作为将机器学习技术融入到蒙特卡洛采样和优化中提供了明确的理论基础。
URL
https://arxiv.org/abs/2505.22598
PDF
-
Federated Instrumental Variable Analysis via Federated Generalized Method of Moments
2025-05-27 10:46:43Geetika, Somya Tyagi, Bapi ChatterjeeAbstract
Instrumental variables (IV) analysis is an important applied tool for areas such as healthcare and consumer economics. For IV analysis in high-dimensional settings, the Generalized Method of Moments (GMM) using deep neural networks offers an efficient approach. With non-i.i.d. data sourced from scattered decentralized clients, federated learning is a popular paradigm for training the models while promising data privacy. However, to our knowledge, no federated algorithm for either GMM or IV analysis exists to date. In this work, we introduce federated instrumental variables analysis (FedIV) via federated generalized method of moments (FedGMM). We formulate FedGMM as a federated zero-sum game defined by a federated non-convex non-concave minimax optimization problem, which is solved using federated gradient descent ascent (FedGDA) algorithm. One key challenge arises in theoretically characterizing the federated local optimality. To address this, we present properties and existence results of clients' local equilibria via FedGDA limit points. Thereby, we show that the federated solution consistently estimates the local moment conditions of every participating client. The proposed algorithm is backed by extensive experiments to demonstrate the efficacy of our approach.
Abstract (translated)
工具变量（IV）分析在医疗保健和消费者经济学等领域中是一项重要的应用工具。对于高维环境下的IV分析，采用深度神经网络的广义矩方法（GMM）提供了一种高效的解决方案。面对非独立同分布（non-i.i.d.）的数据源，这些数据通常来自分散且去中心化的客户端，在这种情况下，联邦学习成为训练模型的一种流行范式，并保证了数据隐私的安全性。然而，据我们所知，至今为止还没有专门针对GMM或IV分析的联邦算法。 在本研究中，我们通过引入联邦广义矩方法（FedGMM）来提出联邦工具变量分析（FedIV）。我们将FedGMM定义为一个由联邦非凸非凹极小化极大化优化问题构成的联邦零和博弈，并采用联邦梯度下降上升（FedGDA）算法进行求解。在理论上刻画联邦局部最优性方面，出现了一个关键挑战。为应对这一挑战，我们通过分析FedGDA极限点来阐述客户端局部均衡的特性及其存在性结果，从而表明联邦解决方案能够一致估计每个参与客户的本地矩条件。 我们的方法得到了广泛的实验验证，证明了该算法的有效性和实用性。
URL
https://arxiv.org/abs/2505.21012
PDF
-
BIPNN: Learning to Solve Binary Integer Programming via Hypergraph Neural Networks
2025-05-27 10:31:52Sen Bai, Chunqi Yang, Xin Bai, Xin Zhang, Zhengang JiangAbstract
Binary (0-1) integer programming (BIP) is pivotal in scientific domains requiring discrete decision-making. As the advance of AI computing, recent works explore neural network-based solvers for integer linear programming (ILP) problems. Yet, they lack scalability for tackling nonlinear challenges. To handle nonlinearities, state-of-the-art Branch-and-Cut solvers employ linear relaxations, leading to exponential growth in auxiliary variables and severe computation limitations. To overcome these limitations, we propose BIPNN (Binary Integer Programming Neural Network), an unsupervised learning framework to solve nonlinear BIP problems via hypergraph neural networks (HyperGNN). Specifically, BIPNN reformulates BIPs-constrained, discrete, and nonlinear (sin, log, exp) optimization problems-into unconstrained, differentiable, and polynomial loss functions. The reformulation stems from the observation of a precise one-to-one mapping between polynomial BIP objectives and hypergraph structures, enabling the unsupervised training of HyperGNN to optimize BIP problems in an end-to-end manner. On this basis, we propose a GPU-accelerated and continuous-annealing-enhanced training pipeline for BIPNN. The pipeline enables BIPNN to optimize large-scale nonlinear terms in BIPs fully in parallel via straightforward gradient descent, thus significantly reducing the training cost while ensuring the generation of discrete, high-quality solutions. Extensive experiments on synthetic and real-world datasets highlight the superiority of our approach.
Abstract (translated)
二进制（0-1）整数规划（BIP）在需要离散决策的科学领域中至关重要。随着人工智能计算的发展，最近的研究工作探索了基于神经网络的求解器来解决整数线性规划（ILP）问题。然而，这些方法在应对非线性挑战时缺乏可扩展性。为了处理非线性问题，最先进的分支切割法利用线性松弛技术，导致辅助变量数量呈指数级增长，并且计算限制严重。 为克服这一局限，我们提出了BIPNN（二进制整数规划神经网络），这是一种无监督学习框架，通过超图神经网络（HyperGNN）解决非线性的BIP问题。具体而言，BIPNN将受约束、离散和包含非线性项（如正弦、对数、指数函数）的BIP优化问题转化为无约束、可微分和多项式形式的目标函数。这种转化基于这样一个观察：即多项式BIP目标与超图结构之间存在精确的一一对应关系，这使得能够通过端到端的方式训练HyperGNN来解决BIP问题。 在此基础上，我们提出了一种GPU加速并结合连续退火增强的BIPNN训练流程。该流程使BIPNN能够在大规模非线性项中完全平行地优化整数规划（BIP）中的多项式损失函数，从而通过简单的梯度下降显著降低了训练成本，并且在确保生成离散、高质量解的同时提高了效率。 广泛的合成和现实世界数据集上的实验显示了我们方法的优越性能。
URL
https://arxiv.org/abs/2505.20997
PDF
-
Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
2025-05-27 06:49:02Yang Zheng, Wen Li, Zhaoqiang LiuAbstract
Inverse problems (IPs) involve reconstructing signals from noisy observations. Traditional approaches often rely on handcrafted priors, which can fail to capture the complexity of real-world data. The advent of pre-trained generative models has introduced new paradigms, offering improved reconstructions by learning rich priors from data. Among these, diffusion models (DMs) have emerged as a powerful framework, achieving remarkable reconstruction performance across numerous IPs. However, existing DM-based methods frequently encounter issues such as heavy computational demands and suboptimal convergence. In this work, building upon the idea of the recent work DMPlug~\cite{wang2024dmplug}, we propose two novel methods, DMILO and DMILO-PGD, to address these challenges. Our first method, DMILO, employs intermediate layer optimization (ILO) to alleviate the memory burden inherent in DMPlug. Additionally, by introducing sparse deviations, we expand the range of DMs, enabling the exploration of underlying signals that may lie outside the range of the diffusion model. We further propose DMILO-PGD, which integrates ILO with projected gradient descent (PGD), thereby reducing the risk of suboptimal convergence. We provide an intuitive theoretical analysis of our approach under appropriate conditions and validate its superiority through extensive experiments on diverse image datasets, encompassing both linear and nonlinear IPs. Our results demonstrate significant performance gains over state-of-the-art methods, highlighting the effectiveness of DMILO and DMILO-PGD in addressing common challenges in DM-based IP solvers.
Abstract (translated)
逆向问题（IPs）涉及从含有噪声的观测数据中重构信号。传统的处理方法通常依赖于人工设计的先验知识，这些先验往往难以捕捉到现实世界数据中的复杂性。随着预训练生成模型的出现，新的范例被引入，通过从大量数据中学得更丰富的先验信息来改进重建效果。在这些方法中，扩散模型（DMs）作为一种强大的框架脱颖而出，在众多逆向问题上取得了显著的重构性能。然而，现有的基于扩散模型的方法常常会遇到计算需求重和次优收敛的问题。 在此项工作中，我们借鉴最近的工作 DMPlug~\cite{wang2024dmplug} 的思想，提出了两种新的方法：DMILO 和 DMILO-PGD，以解决这些问题。我们的第一种方法 DMILO 使用中间层优化（ILO）来减轻 DMPlug 中固有的内存负担。通过引入稀疏偏差，我们扩展了扩散模型的应用范围，使其能够探索可能超出扩散模型范围内底层信号的可能性。进一步地，我们提出了DMILO-PGD 方法，它将 ILO 与投影梯度下降（PGD）相结合，从而降低次优收敛的风险。 我们在适当的条件下对我们的方法进行了直观的理论分析，并通过在多种图像数据集上的广泛实验验证了其优越性，包括线性和非线性的逆向问题。我们的结果表明，在最先进的方法中取得了显著的性能提升，突显了 DMILO 和 DMILO-PGD 在基于扩散模型的逆向问题求解器中解决常见挑战的有效性。
URL
https://arxiv.org/abs/2505.20789
PDF
-
Variational Deep Learning via Implicit Regularization
2025-05-26 17:15:57Jonathan Wenger, Beau Coker, Juraj Marusic, John P. CunninghamAbstract
Modern deep learning models generalize remarkably well in-distribution, despite being overparametrized and trained with little to no explicit regularization. Instead, current theory credits implicit regularization imposed by the choice of architecture, hyperparameters and optimization procedure. However, deploying deep learning models out-of-distribution, in sequential decision-making tasks, or in safety-critical domains, necessitates reliable uncertainty quantification, not just a point estimate. The machinery of modern approximate inference -- Bayesian deep learning -- should answer the need for uncertainty quantification, but its effectiveness has been challenged by our inability to define useful explicit inductive biases through priors, as well as the associated computational burden. Instead, in this work we demonstrate, both theoretically and empirically, how to regularize a variational deep network implicitly via the optimization procedure, just as for standard deep learning. We fully characterize the inductive bias of (stochastic) gradient descent in the case of an overparametrized linear model as generalized variational inference and demonstrate the importance of the choice of parametrization. Finally, we show empirically that our approach achieves strong in- and out-of-distribution performance without tuning of additional hyperparameters and with minimal time and memory overhead over standard deep learning.
Abstract (translated)
现代深度学习模型在分布内情况下表现出色，尽管它们参数过多且训练过程中几乎没有显式的正则化。当前理论认为，架构的选择、超参数和优化过程所隐含的规则化是关键因素。然而，在部署这些模型以处理分布外数据、顺序决策任务或安全攸关领域时，可靠的不确定性量化比单一的点估计更为重要。现代近似推理工具——贝叶斯深度学习——应能满足这种对不确定性的需求，但其有效性因我们难以通过先验定义有用的显式归纳偏置以及由此产生的计算负担而受到质疑。 在这项工作中，我们理论和实证地展示了如何通过优化过程隐式正则化变分深层网络，就像标准的深度学习一样。我们在超参数化的线性模型的情况下全面描述了(随机)梯度下降的归纳偏差，并将其归类为广义变分推理。此外，还证明了选择参数化方式的重要性。 最后，我们通过实验展示了我们的方法能够在不调整额外超参数且仅增加少量时间和内存开销的情况下实现强大的分布内和分布外性能。
URL
https://arxiv.org/abs/2505.20235
PDF
-
Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective
2025-05-26 10:52:17Junnan Liu, Hongwei Liu, Linchen Xiao, Shudong Liu, Taolin Zhang, Zihan Ma, Songyang Zhang, Kai ChenAbstract
We propose a novel framework for comprehending the reasoning capabilities of large language models (LLMs) through the perspective of meta-learning. By conceptualizing reasoning trajectories as pseudo-gradient descent updates to the LLM's parameters, we identify parallels between LLM reasoning and various meta-learning paradigms. We formalize the training process for reasoning tasks as a meta-learning setup, with each question treated as an individual task, and reasoning trajectories serving as the inner loop optimization for adapting model parameters. Once trained on a diverse set of questions, the LLM develops fundamental reasoning capabilities that can generalize to previously unseen questions. Extensive empirical evaluations substantiate the strong connection between LLM reasoning and meta-learning, exploring several issues of significant interest from a meta-learning standpoint. Our work not only enhances the understanding of LLM reasoning but also provides practical insights for improving these models through established meta-learning techniques.
Abstract (translated)
我们提出了一种新颖的框架，通过元学习的角度来理解大规模语言模型（LLMs）的推理能力。我们将推理轨迹视为对LLM参数进行伪梯度下降更新，并在此过程中发现了LLM推理与各种元学习范式之间的相似性。我们将用于推理任务的训练过程形式化为一个元学习设置，在这种设置中，每个问题都被视为独立的任务，而推理轨迹则作为内循环优化以适应模型参数。经过一系列多样化的提问训练后，LLM能够发展出可以推广到未见过的问题的基本推理能力。 大量的实证评估证实了LLM推理与元学习之间存在强有力的联系，并探讨了一些从元学习角度来看具有重要意义的问题。我们的工作不仅增强了对LLM推理的理解，还为通过已建立的元学习技术改进这些模型提供了实用见解。
URL
https://arxiv.org/abs/2505.19815
PDF
-
Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models
2025-05-26 07:31:32Rui Cai, Bangzheng Li, Xiaofei Wen, Muhao Chen, Zhe ZhaoAbstract
Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities across tasks, yet they often exhibit difficulty in distinguishing task-relevant from irrelevant signals, particularly in tasks like Visual Question Answering (VQA), which can lead to susceptibility to misleading or spurious inputs. We refer to this broader limitation as the Cross-Modality Competency Problem: the model's inability to fairly evaluate all modalities. This vulnerability becomes more evident in modality-specific tasks such as image classification or pure text question answering, where models are expected to rely solely on one modality. In such tasks, spurious information from irrelevant modalities often leads to significant performance degradation. We refer to this failure as Modality Interference, which serves as a concrete and measurable instance of the cross-modality competency problem. We further design a perturbation-based causal diagnostic experiment to verify and quantify this problem. To mitigate modality interference, we propose a novel framework to fine-tune MLLMs, including perturbation-based data augmentations with both heuristic perturbations and adversarial perturbations via Projected Gradient Descent (PGD), and a consistency regularization strategy applied to model outputs with original and perturbed inputs. Experiments on multiple benchmark datasets (image-heavy, text-heavy, and VQA tasks) and multiple model families with different scales demonstrate significant improvements in robustness and cross-modality competency, indicating our method's effectiveness in boosting unimodal reasoning ability while enhancing performance on multimodal tasks.
Abstract (translated)
多模态大型语言模型（MLLMs）在各种任务中展示了令人印象深刻的性能，但它们经常难以区分与任务相关的信号和不相关的信号，特别是在如视觉问答（VQA）这样的任务中。这会导致这些模型容易受到误导性或错误输入的影响。我们将这种更广泛的问题称为跨模态胜任力问题：即模型无法公平评估所有模式的能力。在特定于某一模式的任务（如图像分类或纯文本问答）中，这一脆弱性变得更为明显，在这类任务中，模型应仅依赖一种模式。在这种情况下，来自不相关模式的错误信息通常会导致性能显著下降。我们将这种失败称为模态干扰，这是跨模态胜任力问题的一个具体而可量化的实例。 为了验证和量化这个问题，我们设计了一个基于扰动的因果诊断实验。为减轻模态干扰，我们提出了一种新颖的微调MLLM框架，包括基于数据增强的启发式和对抗性扰动（通过投影梯度下降实现），以及应用于模型输出的一致性正则化策略，这些输出是基于原始输入和扰动后输入生成的。 在多个基准数据集上进行实验（图像密集型、文本密集型及VQA任务）以及不同规模的多种模型家族显示了显著的鲁棒性和跨模态胜任力改进。这表明我们的方法不仅能够增强单一模式推理能力，还能提高多模式任务的表现。
URL
https://arxiv.org/abs/2505.19616
PDF
-
Minimalist Softmax Attention Provably Learns Constrained Boolean Functions
2025-05-26 05:33:26Jerry Yao-Chieh Hu, Xiwen Zhang, Maojiang Su, Zhao Song, Han LiuAbstract
We study the computational limits of learning $k$-bit Boolean functions (specifically, $\mathrm{AND}$, $\mathrm{OR}$, and their noisy variants), using a minimalist single-head softmax-attention mechanism, where $k=\Theta(d)$ relevant bits are selected from $d$ inputs. We show that these simple $\mathrm{AND}$ and $\mathrm{OR}$ functions are unsolvable with a single-head softmax-attention mechanism alone. However, with teacher forcing, the same minimalist attention is capable of solving them. These findings offer two key insights: Architecturally, solving these Boolean tasks requires only minimalist attention, without deep Transformer blocks or FFNs. Methodologically, one gradient descent update with supervision suffices and replaces the multi-step Chain-of-Thought (CoT) reasoning scheme of [Kim and Suzuki, ICLR 2025] for solving Boolean problems. Together, the bounds expose a fundamental gap between what this minimal architecture achieves under ideal supervision and what is provably impossible under standard training.
Abstract (translated)
我们研究了使用简约的单头softmax注意力机制（其中$ k=\Theta(d) $相关的位从$d$个输入中被选出）来学习$k$-位布尔函数（特别是$\mathrm{AND}$、$\mathrm{OR}$及其噪声变体）的计算极限。结果显示，这些简单的$\mathrm{AND}$和$\mathrm{OR}$函数仅靠单头softmax注意力机制是无法解决的。然而，在使用教师强制训练的情况下，相同的简约注意力机制能够解决这些问题。 这些发现提供了两个关键见解：从架构的角度来看，解决布尔任务只需要简约注意机制，而不需要深层Transformer块或全连接层（FFNs）。从方法论角度来看，一次带有监督的梯度下降更新足以替代[Kim和Suzuki, ICLR 2025]为解决布尔问题提出的多步骤链式思考（CoT）推理方案。总的来说，这些界限揭示了在理想监督下这种最小架构所能实现的效果与在标准训练条件下无法证明可能之间存在着一个根本性的差距。
URL
https://arxiv.org/abs/2505.19531
PDF
-
Navigating loss manifolds via rigid body dynamics: A promising avenue for robustness and generalisation
2025-05-26 05:26:21Mohammed D. Belgoumri, Mohamed Reda Bouadjenek, Hakim Hacid, Imran Razzak, Sunil AryalAbstract
Training large neural networks through gradient-based optimization requires navigating high-dimensional loss landscapes, which often exhibit pathological geometry, leading to undesirable training dynamics. In particular, poor generalization frequently results from convergence to sharp minima that are highly sensitive to input perturbations, causing the model to overfit the training data while failing to generalize to unseen examples. Furthermore, these optimization procedures typically display strong dependence on the fine structure of the loss landscape, leading to unstable training dynamics, due to the fractal-like nature of the loss surface. In this work, we propose an alternative optimizer that simultaneously reduces this dependence, and avoids sharp minima, thereby improving generalization. This is achieved by simulating the motion of the center of a ball rolling on the loss landscape. The degree to which our optimizer departs from the standard gradient descent is controlled by a hyperparameter, representing the radius of the ball. Changing this hyperparameter allows for probing the loss landscape at different scales, making it a valuable tool for understanding its geometry.
Abstract (translated)
通过基于梯度的优化训练大型神经网络需要在高维损失景观中导航，而这种景观通常具有病态几何形状，导致不理想的训练动态。特别是，由于模型收敛于对输入扰动高度敏感的尖锐极小值，从而过拟合训练数据并无法泛化到未见示例，因此经常导致较差的一般性（泛化能力）。此外，这些优化过程通常强烈依赖于损失景观的精细结构，导致不稳定的训练动态，这是由于损失表面具有分形特性。在这项工作中，我们提出了一种替代的优化器，该优化器同时减少了这种依赖关系并避免了尖锐极小值，从而提高了泛化能力。这通过模拟一个球心在损失景观上滚动的运动来实现。我们的优化器与标准梯度下降方法的区别程度由超参数表示（即球体的半径）控制。改变这个超参数可以在不同尺度下探测损失景观，使其成为了解其几何形状的重要工具。
URL
https://arxiv.org/abs/2505.19527
PDF
-
CloneShield: A Framework for Universal Perturbation Against Zero-Shot Voice Cloning
2025-05-25 12:22:00Renyuan Li, Zhibo Liang, Haichuan Zhang, Tianyu Shi, Zhiyuan Cheng, Jia Shi, Carl Yang, Mingjie TangAbstract
Recent breakthroughs in text-to-speech (TTS) voice cloning have raised serious privacy concerns, allowing highly accurate vocal identity replication from just a few seconds of reference audio, while retaining the speaker's vocal authenticity. In this paper, we introduce CloneShield, a universal time-domain adversarial perturbation framework specifically designed to defend against zero-shot voice cloning. Our method provides protection that is robust across speakers and utterances, without requiring any prior knowledge of the synthesized text. We formulate perturbation generation as a multi-objective optimization problem, and propose Multi-Gradient Descent Algorithm (MGDA) to ensure the robust protection across diverse utterances. To preserve natural auditory perception for users, we decompose the adversarial perturbation via Mel-spectrogram representations and fine-tune it for each sample. This design ensures imperceptibility while maintaining strong degradation effects on zero-shot cloned outputs. Experiments on three state-of-the-art zero-shot TTS systems, five benchmark datasets and evaluations from 60 human listeners demonstrate that our method preserves near-original audio quality in protected inputs (PESQ = 3.90, SRS = 0.93) while substantially degrading both speaker similarity and speech quality in cloned samples (PESQ = 1.07, SRS = 0.08).
Abstract (translated)
最近在文本到语音（TTS）声音克隆领域的突破引发了严重的隐私担忧，仅通过几秒钟的参考音频即可实现高度准确的声音身份复制，并保留发言人的声音真实性。本文介绍了CloneShield，这是一个专门设计用于防御零样本声音克隆的通用时域对抗性扰动框架。我们的方法提供了一种无需任何关于合成文本的先验知识就能在不同说话人和话语中保持稳健性的保护方式。我们通过将扰动生成形式化为一个多目标优化问题，并提出多梯度下降算法（MGDA）来确保对各种话语具有鲁棒性的保护措施。 为了保留用户自然的听觉感知，我们将对抗性扰动分解为梅尔频谱图表示，并针对每个样本进行微调。这种设计在保证不可察觉的同时，还能保持对零样本克隆输出的强大破坏效果。我们在三个最先进的零样本TTS系统、五个基准数据集以及来自60名人类听众的评估中进行了实验，结果表明我们的方法可以保留受保护输入接近原始音频的质量（PESQ = 3.90, SRS = 0.93），同时大幅降低克隆样本中的说话人相似性和语音质量（PESQ = 1.07, SRS = 0.08）。
URL
https://arxiv.org/abs/2505.19119
PDF
