Abstract
Undeniably, Large Language Models (LLMs) have stirred an extraordinary wave of innovation in the machine learning research domain, resulting in substantial impact across diverse fields such as reinforcement learning, robotics, and computer vision. Their incorporation has been rapid and transformative, marking a significant paradigm shift in the field of machine learning research. However, the field of experimental design, grounded on black-box optimization, has been much less affected by such a paradigm shift, even though integrating LLMs with optimization presents a unique landscape ripe for exploration. In this position paper, we frame the field of black-box optimization around sequence-based foundation models and organize their relationship with previous literature. We discuss the most promising ways foundational language models can revolutionize optimization, which include harnessing the vast wealth of information encapsulated in free-form text to enrich task comprehension, utilizing highly flexible sequence models such as Transformers to engineer superior optimization strategies, and enhancing performance prediction over previously unseen search spaces.
Abstract (translated)
毫无疑问，大型语言模型（LLMs）在机器学习领域引起了非凡的创新波，对诸如强化学习、机器人学和计算机视觉等各个领域产生了重大影响。它们的出现速度之快、影响之深，标志着机器学习研究领域的范式发生了重大转变。然而，基于黑盒优化的实验设计领域受到的影响要小得多，尽管将LLM与优化相结合，为该领域开拓了独特的探索景观。在本文论文中，我们将围绕基于序列的基础模型来组织黑色盒优化领域，并探讨LLM与之前文献的关系。我们讨论了LLM可以如何通过利用自由的文本中蕴含的丰富信息来改善任务理解，使用具有高度灵活性的序列模型（如Transformer）来设计卓越的优化策略，以及通过增强以前未曾见过的搜索空间的表现来提高性能预测等最具潜力的途径。
URL
https://arxiv.org/abs/2405.03547
