Abstract
The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.
Abstract (translated)
大型语言模型（LLM）系统的性能不仅取决于模型权重，也取决于其工具链：即决定向模型存储、检索和呈现何种信息的代码。然而，工具链的设计目前仍主要依赖人工完成，而现有的文本优化器因过度压缩反馈信息，与这一场景适配不佳。我们提出Meta-Harness，一种用于搜索LLM应用工具链的外循环系统。该系统采用一个可访问源代码、历史候选方案评分及执行轨迹的智能体式提议器，通过文件系统进行迭代优化。在在线文本分类任务中，Meta-Harness相比最先进的上下文管理系统提升了7.7分，同时仅使用四分之一的上下文令牌。在检索增强型数学推理任务中，单次发现的工具链在200道国际数学奥林匹克难度问题上，平均使五个未参与训练的模型准确率提升4.7分。在智能体编程任务中，发现的工具链在TerminalBench-2基准上超越了最佳人工设计的基线方案。这些结果共同表明，更丰富的历史经验访问能力可实现工具链的自动化工程优化。
URL
https://arxiv.org/abs/2603.28052
