Abstract
Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks, that is, to leverage many-shot demonstrations purely via in-context learning (ICL) without gradient descent. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge and reasoning for broader chat workflows. Our pretraining procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. We begin with a random-forest teacher, distilling tree-based decision strategies into the LLM to strengthen robustness in numerical modeling. All tasks are serialized with a token-efficient prompt, enabling 3x to 6x more examples per context window and delivering up to 50x amortized throughput via batch inference. Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8), MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an average of about 15% on out-of-distribution tabular classification across finance, physics, biology, and healthcare domains. It exhibits a striking many-shot scaling law: accuracy increases monotonically as in-context demonstrations grow from 8 to 1,024. Without any task-specific training, it attains random-forest-level accuracy across hundreds of shots. General chat capabilities, including knowledge and reasoning, are preserved: it achieves 75.4% on MMLU.
Abstract (translated)
大型语言模型（LLMs）拥有广泛的世界知识和强大的通用推理能力，但它们在标准机器学习（ML）任务上难以从许多上下文示例中进行学习，即无法通过纯上下文学习（ICL）而不使用梯度下降方法来利用多示例演示。我们引入了MachineLearningLM，这是一个便携式的持续预训练框架，它为通用的LLM配备了稳健的上下文机器学习能力，同时保留了其广泛的常识和推理能力，适用于更广泛的任务流程。 我们的预训练过程从数百万个结构因果模型（SCMs）中合成ML任务，并扩展到1,024个示例数量。我们首先使用随机森林教师将基于树的决策策略提炼进LLM，以增强数值建模中的鲁棒性。所有任务都使用一个高效的提示进行序列化，这使得每个上下文窗口内的示例数量最多可增加3至6倍，并通过批量推理实现了最高50倍的平均吞吐量。 尽管采用了适度配置（如Qwen-2.5-7B-Instruct和LoRA秩8），MachineLearningLM在跨金融、物理、生物及医疗领域分布外表格分类任务上，相对于强大的LLM基线模型（例如GPT-5-mini）表现出约15%的平均性能提升。它还展示了显著的多示例扩展法则：随着上下文演示从8个增加到1,024个，准确率单调上升；无需特定任务训练，在数百个示例的情况下也能达到随机森林级别的精度。 此外，其通用聊天能力（包括知识和推理）得到了保持：在MMLU基准测试中，它达到了75.4%的分数。
URL
https://arxiv.org/abs/2509.06806
