Abstract
Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks but come with substantial energy and computational costs, particularly in request-heavy scenarios. In many real-world applications, the full scale and capabilities of LLMs are often unnecessary, as Small Language Models (SLMs) can provide accurate responses for simpler text generation tasks. When enhanced with advanced reasoning strategies, such as Chain-of-Thought (CoT) prompting or Majority Voting, SLMs can approach the performance of larger models while reducing overall computational requirements. However, these strategies can also introduce additional energy costs, creating an energy-accuracy trade-off. Our analysis examines these trade-offs in test-time compute strategies for smaller models compared to larger ones, using the MMLU benchmark. Additionally, we explore the input-output token dynamics of transformer architectures, which result in nonlinear hardware energy operation curves for LLMs. To bridge AI research with its physical impact, we propose \textit{energy efficiency metrics}, including Energy-per-Token, as complements to traditional accuracy benchmarks. Beyond model selection, we propose controlled reasoning in CoT token generation, using operating curves to regulate reasoning depth dynamically. This vision integrates a energy-aware routing mechanism, ensuring that model selection and inference strategies balance accuracy for sustainable AI deployment.
Abstract (translated)
大型语言模型（LLMs）在各类任务中展现出卓越性能，但其高昂的能耗与计算成本在请求密集场景中尤为突出。在实际应用中，LLM的完整规模与能力往往并不必要，因为小型语言模型（SLMs）已能针对简单文本生成任务提供准确响应。当SLMs结合思维链（CoT）提示或多数投票等先进推理策略时，其性能可逼近大型模型，同时降低总体计算需求。然而，这些策略也可能带来额外能耗，形成能耗-精度权衡。本研究通过MMLU基准测试，对比分析了小型与大型模型在测试时计算策略中的此类权衡。此外，我们探讨了Transformer架构的输入输出令牌动态特性，该特性导致LLMs呈现非线性硬件能耗运行曲线。为连接AI研究与物理影响，我们提出将包括每令牌能耗在内的能效指标作为传统精度基准的补充。除模型选择外，我们还提出在CoT令牌生成中实施可控推理，利用运行曲线动态调节推理深度。这一愿景整合了能感知能耗的路由机制，确保模型选择与推理策略在精度与可持续性部署间取得平衡。
URL
https://arxiv.org/abs/2603.20224
