Abstract
Three core tenets of reinforcement learning (RL)--concerning the definition of agency, the objective of learning, and the scope of the reward hypothesis--have been highlighted as key targets for conceptual revision, with major implications for theory and application. We propose a framework, inspired by open-ended evolutionary theory, to reconsider these three "dogmas." We revisit each assumption and address related concerns raised alongside them. To make our arguments relevant to RL as a model of biological learning, we first establish that evolutionary dynamics can plausibly operate within living brains over an individual's lifetime, and are not confined to cross-generational processes. We begin by revisiting the second dogma, drawing on evolutionary insights to enrich the "adaptation-rather-than-search" view of learning. We then address the third dogma regarding the limits of the reward hypothesis, using analogies from evolutionary fitness to illuminate the scalar reward vs. multi-objective debate. After discussing practical implications for exploration in RL, we turn to the first--and arguably most fundamental--issue: the absence of a formal account of agency. We argue that unlike the other two problems, the evolutionary paradigm alone cannot resolve the agency question, though it gestures in a productive direction. We advocate integrating ideas from origins-of-life theory, where the thermodynamics of sustenance and replication offer promising foundations for understanding agency and resource-constrained reinforcement learning in biological systems.
Abstract (translated)
强化学习（RL）的三个核心原则——关于代理定义、学习目标以及奖励假设范围的问题——已被提出需要概念上的重新评估，这对理论和应用都有重大影响。我们借鉴开放进化理论提出了一个框架来重新考虑这三条“信条”。我们将回顾每个假设，并解决与其相关的关切问题。 为了使我们的论点对强化学习作为生物学习模型的相关性得到认可，我们首先建立了一个观点：进化动力学可以在个体一生中在活脑内合理运作，并不仅仅局限于跨代过程。我们将从回溯第二个信条开始，利用进化视角来丰富“适应而非搜索”的学习观念。 接着，我们将讨论第三个信条中的奖励假设的局限性问题，通过类比于进化的适配度概念来阐明标量回报与多目标之间的辩论。在探讨强化学习中探索的实际影响后，我们转向最根本的问题：代理定义的缺乏。我们认为，与其他两个问题不同的是，仅仅依靠进化范式无法解决代理问题，尽管它指明了一个富有成效的方向。 我们提倡将起源理论的思想整合到这一框架内，在这些理论中，维持与复制的热力学基础为理解生物系统中的代理性和资源受限的强化学习提供了有希望的基础。
URL
https://arxiv.org/abs/2507.11482
