Abstract
Transfer learning in deep reinforcement learning is often motivated by improved stability and reduced training cost, but it can also fail under substantial domain shift. This paper presents a controlled empirical study examining how architectural differences between Double Deep Q-Networks (DDQN) and Dueling DQN influence transfer behavior across environments. Using CartPole as a source task and LunarLander as a structurally distinct target task, we evaluate a fixed layer-wise representation transfer protocol under identical hyperparameters and training conditions, with baseline agents trained from scratch used to contextualize transfer effects. Empirical results show that DDQN consistently avoids negative transfer under the examined setup and maintains learning dynamics comparable to baseline performance in the target environment. In contrast, Dueling DQN consistently exhibits negative transfer under identical conditions, characterized by degraded rewards and unstable optimization behavior. Statistical analysis across multiple random seeds confirms a significant performance gap under transfer. These findings suggest that architectural inductive bias is strongly associated with robustness to cross-environment transfer in value-based deep reinforcement learning under the examined transfer protocol.
Abstract (translated)
深度强化学习中的迁移学习通常由改进的稳定性以及降低训练成本所驱动，但在显著的任务域变化下可能会失败。本文通过一项受控实证研究，探讨了双层深度Q网络（DDQN）和斗士DQN之间架构差异如何影响跨环境中的迁移行为。使用CartPole作为源任务，并将LunarLander作为一个结构上明显不同的目标任务，在相同的超参数和训练条件下评估了一个固定的逐层表示迁移协议。我们还利用从头开始训练的基线智能体来定位迁移效果。实证结果表明，DDQN在观察到的设置下始终避免了负向迁移，并且其学习动态与目标环境中的基线性能相当。相比之下，在相同的条件下，斗士DQN则持续表现出负面迁移，表现为奖励下降和优化行为不稳定。多个随机种子上的统计分析确认了转移过程中的显著性能差距。这些发现表明，在所研究的迁移协议下，基于价值的深度强化学习中架构性的归纳偏差与跨环境迁移时的鲁棒性之间存在强相关性。
URL
https://arxiv.org/abs/2602.09810
