Abstract
GPU design space exploration (DSE) for modern AI workloads, such as Large-Language Model (LLM) inference, is challenging because of GPUs' vast, multi-modal design spaces, high simulation costs, and complex design optimization objectives (e.g. performance, power and area trade-offs). Existing automated DSE methods are often prohibitively expensive, either requiring an excessive number of exploration samples or depending on intricate, manually crafted analyses of interdependent critical paths guided by human heuristics. We present LUMINA, an LLM-driven GPU architecture exploration framework that leverage AI to enhance the DSE efficiency and efficacy for GPUs. LUMINA extracts architectural knowledge from simulator code and performs sensitivity studies to automatically compose DSE rules,which are auto-corrected during exploration. A core component of LUMINA is a DSE Benchmark that comprehensively evaluates and enhances LLMs' capabilities across three fundamental skills required for architecture optimization, which provides a principled and reproducible basis for model selection and ensuring consistent architectural reasoning. In the design space with 4.7 million possible samples, LUMINA identifies 6 designs of better performance and area than an A100 GPU efficiently, using only 20 steps via LLM-assisted bottleneck analysis. In comparison, LUMINA achieves 17.5x higher than design space exploration efficiency, and 32.9% better designs (i.e. Pareto Hypervolume) than Machine-Learning baselines, showcasing its ability to deliver high-quality design guidance with minimal search cost.
Abstract (translated)
GPU 设计空间探索（DSE）对于现代人工智能工作负载，例如大型语言模型（LLM）推理来说是一项挑战。这是因为GPU具有庞大的、多模式的设计空间，高昂的模拟成本以及复杂的优化目标（如性能、功耗和面积之间的权衡）。现有的自动化设计空间探索方法往往过于昂贵，要么需要过多的探索样本数量，要么依赖于复杂的人工构建分析，这些分析基于人类直觉来追踪相互依存的关键路径。 我们提出了LUMINA框架，这是一个由LLM驱动的GPU架构探索框架，利用AI技术提高GPU DSE的效率和效果。LUMINA从模拟器代码中提取架构知识，并执行敏感性研究以自动组合DSE规则，在探索过程中这些规则会自动校正。LUMINA的一个关键组件是一个设计空间探索基准测试，该基准可以全面评估并增强LLM在优化架构所需的三个基本技能方面的功能，为模型选择提供原则性和可复现的基础，并确保一致的架构推理。 在一个包含470万个可能样本的设计空间中，通过使用20步（借助LLM辅助瓶颈分析），LUMINA能够高效地识别出6种性能和面积优于A100 GPU的设计。相比之下，与机器学习基线相比，LUMINA实现了17.5倍更高的设计空间探索效率，并且设计质量提高了32.9%（即帕累托超体积指标）。这展示了其在最小化搜索成本的同时提供高质量设计指导的能力。 总之，LUMINA通过智能利用AI技术显著提升了GPU架构探索的效率和效果，在有限资源内实现了更优的设计。
URL
https://arxiv.org/abs/2603.05904
