Abstract
AI researchers and practitioners increasingly apply large language models (LLMs) to what we call reasoning-intensive regression (RiR), i.e. deducing subtle numerical properties from text. Unlike standard language regression tasks, e.g. for sentiment or similarity, RiR often appears instead in ad-hoc problems like rubric-based scoring or domain-specific retrieval, where much deeper analysis of text is required while only limited task-specific training data and computation are available. We cast three realistic problems as RiR tasks to establish an initial benchmark, and use that to test our hypothesis that prompting frozen LLMs and finetuning Transformer encoders via gradient descent will both often struggle in RiR. We then propose MENTAT, a simple and lightweight method that combines batch-reflective prompt optimization with neural ensemble learning. MENTAT achieves up to 65% improvement over both baselines, though substantial room remains for future advances in RiR.
Abstract (translated)
AI研究人员和实践者越来越多地将大型语言模型（LLMs）应用于我们所说的推理密集型回归（RiR），即从文本中推断出细微的数值属性。与标准的语言回归任务，如情感分析或相似度评估不同，RiR通常出现在一些临时性问题上，例如基于评分细则的评分或特定领域的检索，在这些问题中需要对文本进行更深入的分析，而同时可用的任务特定训练数据和计算资源却非常有限。我们把三个现实中的问题作为RiR任务来建立初步基准，并利用这些任务测试我们的假设：即通过提示冻结的语言模型以及通过梯度下降微调变换器编码器在处理RiR时常常会遇到困难。然后，我们提出了MENTAT方法，这是一种简单而轻量级的方法，结合了批量自省的提示优化与神经集成学习。相比基准线，MENTAT实现了高达65%的改进，尽管未来在RiR领域仍然存在很大的进步空间。
URL
https://arxiv.org/abs/2508.21762
