Abstract
Evolution Strategies (ES) are effective gradient-free optimization methods that can be competitive with gradient-based approaches for policy search. ES only rely on the total episodic scores of solutions in their population, from which they estimate fitness gradients for their update with no access to true gradient information. However this makes them sensitive to deceptive fitness landscapes, and they tend to only explore one way to solve a problem. Quality-Diversity methods such as MAP-Elites introduced additional information with behavior descriptors (BD) to return a population of diverse solutions, which helps exploration but leads to a large part of the evaluation budget not being focused on finding the best performing solution. Here we show that behavior information can also be leveraged to find the best policy by identifying promising search areas which can then be efficiently explored with ES. We introduce the framework of Quality with Just Enough Diversity (JEDi) which learns the relationship between behavior and fitness to focus evaluations on solutions that matter. When trying to reach higher fitness values, JEDi outperforms both QD and ES methods on hard exploration tasks like mazes and on complex control problems with large policies.
Abstract (translated)
进化策略（ES）是一种有效的无需梯度的优化方法，在策略搜索中可以与基于梯度的方法竞争。ES仅依赖于其种群中解决方案的全面状态，然后根据这些状态估计更新时的适应度梯度，而无需访问真实的梯度信息。然而，这使得它们对欺骗性的 fitness 景观敏感，并且它们倾向于只探索一个问题。诸如 MAP-Elites 这样的质量多样性方法通过引入行为描述符（BD）为种群返回了一个多样性的解决方案，这有助于探索，但导致评估预算的大部分没有集中在找到最佳策略上。我们在这里展示，行为信息也可以用于通过确定有前景的搜索区域来找到最佳策略。我们引入了 Quality with Just Enough Diversity (JEDi) 框架，该框架学会了行为和适应度之间的关系，将评估重点放在解决方案上。在尝试达到更高的 fitness 值时，JEDi 超越了 QD 和 ES 方法在具有挑战性的探索任务（如迷宫）和复杂控制问题（具有大量策略）上的表现。
URL
https://arxiv.org/abs/2405.04308
