Abstract
Reinforcement learning fine-tuning has proven effective for steering generative diffusion models toward desired properties in image and molecular domains. Graph diffusion models have similarly been applied to combinatorial structure generation, including neural architecture search (NAS). However, neural architectures are directed acyclic graphs (DAGs) where edge direction encodes functional semantics such as data flow-information that existing graph diffusion methods, designed for undirected structures, discard. We propose Directed Graph Policy Optimization (DGPO), which extends reinforcement learning fine-tuning of discrete graph diffusion models to DAGs via topological node ordering and positional encoding. Validated on NAS-Bench-101 and NAS-Bench-201, DGPO matches the benchmark optimum on all three NAS-Bench-201 tasks (91.61%, 73.49%, 46.77%). The central finding is that the model learns transferable structural priors: pretrained on only 7% of the search space, it generates near-oracle architectures after fine-tuning, within 0.32 percentage points of the full-data model and extrapolating 7.3 percentage points beyond its training ceiling. Bidirectional control experiments confirm genuine reward-driven steering, with inverse optimization reaching near random-chance accuracy (9.5%). These results demonstrate that reinforcement learning-steered discrete diffusion, once extended to handle directionality, provides a controllable generative framework for directed combinatorial structures.
Abstract (translated)
强化学习微调已被证明在引导生成扩散模型朝向图像和分子领域所需的特性方面是有效的。图扩散模型也已经被应用于组合结构生成，包括神经架构搜索（NAS）。然而，神经架构是由有向无环图（DAG）组成的，其中边的方向编码了功能语义，如数据流信息——这种信息现有的针对非定向结构设计的图扩散方法会忽略掉。 我们提出了有向图策略优化（DGPO），这是一种通过拓扑节点排序和位置编码来将离散图扩散模型的强化学习微调扩展到DAG的方法。在NAS-Bench-101和NAS-Bench-201上的验证显示，DGPO在所有三个NAS-Bench-201任务上达到了基准最优值（91.61%，73.49%，46.77%）。主要发现是模型学习到了可迁移的结构先验：仅在搜索空间的7%进行预训练后，在微调阶段就生成了接近Oracle架构，其性能几乎与完全数据模型相当，并且还在训练上限之上额外提升了7.3个百分点。双向控制实验确认了真正的奖励驱动引导，逆向优化达到了接近随机机会准确度（9.5%）。这些结果表明，一旦强化学习引导的离散扩散方法能够处理方向性问题，它就能为有向组合结构提供一个可控的生成框架。
URL
https://arxiv.org/abs/2602.19261
