Abstract
Gradient based optimization is fundamental to most modern deep reinforcement learning algorithms, however, it introduces significant sensitivity to hyperparameters, unstable training dynamics, and high computational costs. We propose TabPFN RL, a novel gradient free deep RL framework that repurposes the meta trained transformer TabPFN as a Q function approximator. Originally developed for tabular classification, TabPFN is a transformer pre trained on millions of synthetic datasets to perform inference on new unseen datasets via in context learning. Given an in context dataset of sample label pairs and new unlabeled data, it predicts the most likely labels in a single forward pass, without gradient updates or task specific fine tuning. We use TabPFN to predict Q values using inference only, thereby eliminating the need for back propagation at both training and inference. To cope with the model's fixed context budget, we design a high reward episode gate that retains only the top 5% of trajectories. Empirical evaluations on the Gymnasium classic control suite demonstrate that TabPFN RL matches or surpasses Deep Q Network on CartPole v1, MountainCar v0, and Acrobot v1, without applying gradient descent or any extensive hyperparameter tuning. We discuss the theoretical aspects of how bootstrapped targets and non stationary visitation distributions violate the independence assumptions encoded in TabPFN's prior, yet the model retains a surprising generalization capacity. We further formalize the intrinsic context size limit of in context RL algorithms and propose principled truncation strategies that enable continual learning when the context is full. Our results establish prior fitted networks such as TabPFN as a viable foundation for fast and computationally efficient RL, opening new directions for gradient free RL with large pre trained transformers.
Abstract (translated)
基于梯度的优化是大多数现代深度强化学习算法的基础，然而它引入了对超参数的高度敏感性、不稳定的训练动态以及高昂的计算成本。我们提出了一种新的无梯度深度RL框架——TabPFN RL，该框架重新利用元训练过的变压器模型TabPFN作为Q函数逼近器。最初为表格分类设计的TabPFN是一个在数百万个合成数据集上预训练的Transformer模型，能够通过上下文学习来推断新未见的数据集。给定包含样本标签对的上下文数据集和新的无标签数据，它可以在单次前向传递中预测最可能的标签，无需进行梯度更新或特定任务的微调。 我们在不使用反向传播的情况下，利用TabPFN仅通过推理来预测Q值。为了应对模型固定的上下文预算限制，我们设计了一个高奖励集束门（high reward episode gate），该机制保留了前5%的轨迹。在Gymnasium经典控制套件上的实证评估表明，在不应用梯度下降或进行大量超参数调整的情况下，TabPFN RL在CartPole v1、MountainCar v0和Acrobot v1环境中的表现与深度Q网络相当甚至更优。 我们讨论了如何通过自举目标（bootstrapped targets）以及非静态访问分布破坏TabPFN先验中编码的独立性假设的问题，尽管如此，模型仍然保持了令人惊讶的一般化能力。此外，我们形式化地定义了在上下文RL算法中的内在上下文大小限制，并提出了原理性的截断策略，在上下文满时允许持续学习。 我们的研究结果表明，基于拟合网络（如TabPFN）可以作为快速且计算效率高的强化学习的基础框架，为使用大型预训练变换器进行无梯度的强化学习开辟了新的方向。
URL
https://arxiv.org/abs/2509.11259
