Abstract
Reinforcement learning is essential for neural architecture search and hyperparameter optimization, but the conventional approaches impede widespread use due to prohibitive time and computational costs. Inspired by DeepSeek-V3 multi-token prediction architecture, we propose Sequential Policy Gradient modeling (SPG), a novel trajectory generation paradigm for lightweight online hyperparameter optimization. In contrast to conventional policy gradient methods, SPG extends the base model with temporary modules, enabling it to generate state-action (padded) trajectories in a single forward pass. Our experiments demonstrate that models gain performance when retrained with SPG on their original datasets and also outperform standard transfer fine-tuning. We evaluate on five datasets spanning computer vision (ImageNet, COCO), natural language processing (GLUE, SQuAD), and audio (SUPERB) to assess the industrial applicability of SPG. The proposed method demonstrates consistent improvements across widely adopted models, achieving performance gains of $+0.2\sim7\%$, with significantly low computational costs. Fully reproducible code and pre-trained models: this https URL.
Abstract (translated)
强化学习在神经架构搜索和超参数优化中至关重要，但传统方法由于时间成本和计算成本过高而阻碍了其广泛应用。受到DeepSeek-V3多令牌预测架构的启发，我们提出了序列策略梯度模型（SPG），这是一种轻量级在线超参数优化的新颖轨迹生成范式。与传统的策略梯度方法不同，SPG通过添加临时模块来扩展基础模型，使其能够在一次前向传递中生成状态-动作（填充）轨迹。 我们的实验表明，在原始数据集上使用SPG重新训练的模型性能会提高，并且它们也超过了标准迁移微调的效果。我们对涵盖计算机视觉（ImageNet、COCO）、自然语言处理（GLUE、SQuAD）和音频（SUPERB）五个不同领域的数据集进行了评估，以验证SPG在工业应用中的适用性。 所提出的方法在广泛采用的模型上展示了一致性的改进，实现了$+0.2\sim7\%$的性能提升，并且具有显著低的计算成本。完全可重复的代码和预训练模型在此链接中提供：[此URL]（请将“this https URL”替换为实际提供的URL）。
URL
https://arxiv.org/abs/2506.15051
