Abstract
Finding effective prompts for language models (LMs) is critical yet notoriously difficult: the prompt space is combinatorially large, rewards are sparse due to expensive target-LM evaluation. Yet, existing RL-based prompt optimizers often rely on on-policy updates and a meta-prompt sampled from a fixed distribution, leading to poor sample efficiency. We propose GFlowPO, a probabilistic prompt optimization framework that casts prompt search as a posterior inference problem over latent prompts regularized by a meta-prompted reference-LM prior. In the first step, we fine-tune a lightweight prompt-LM with an off-policy Generative Flow Network (GFlowNet) objective, using a replay-based training policy that reuses past prompt evaluations to enable sample-efficient exploration. In the second step, we introduce Dynamic Memory Update (DMU), a training-free mechanism that updates the meta-prompt by injecting both (i) diverse prompts from a replay buffer and (ii) top-performing prompts from a small priority queue, thereby progressively concentrating the search process on high-reward regions. Across few-shot text classification, instruction induction benchmarks, and question answering tasks, GFlowPO consistently outperforms recent discrete prompt optimization baselines.
Abstract (translated)
寻找有效的提示对于语言模型（LM）来说是至关重要的，但同时也非常困难：提示空间在组合上庞大无比，由于目标语言模型的评估成本高昂，因此奖励也极为稀疏。然而，现有的基于强化学习（RL）的提示优化器通常依赖于在线策略更新和从固定分布中抽取的元提示，这导致了较差的样本效率。 我们提出了GFlowPO，这是一种概率性的提示优化框架，它将提示搜索视为一种通过元提示引导的语言模型先验进行后验推理的问题。在第一步中，我们使用基于离线策略的目标生成流网络（Generative Flow Network, GFlowNet）目标训练了一个轻量级的提示语言模型，并采用了基于回放的训练策略来重用过去的提示评估以实现高效的样本探索。第二步中，我们引入了动态内存更新（Dynamic Memory Update, DMU），这是一种无需训练的方法，它通过注入来自回放缓冲区的多样化的提示以及来自优先队列的小型高绩效提示，逐步将搜索过程集中在高奖励区域。 在少量样本文本分类、指令诱导基准测试和问答任务中，GFlowPO始终优于最近提出的离散提示优化基线方法。
URL
https://arxiv.org/abs/2602.03358
