Abstract
How effectively can LLM-based AI assistants utilize their memory (context) to perform various tasks? Traditional data benchmarks, which are often manually crafted, suffer from several limitations: they are static, susceptible to overfitting, difficult to interpret, and lack actionable insights--failing to pinpoint the specific capabilities a model lacks when it does not pass a test. In this paper, we present a framework for automatically generating a comprehensive set of tests to evaluate models' abilities to use their memory effectively. Our framework extends the range of capability tests beyond the commonly explored (passkey, key-value, needle in the haystack) search, a dominant focus in the literature. Specifically, we evaluate models on atomic tasks such as searching, recalling, editing, matching, comparing information in context memory, and performing basic operations when inputs are structured into distinct blocks, simulating real-world data. Additionally, we design composite tests to investigate the models' ability to maintain state while operating on memory. Our benchmark enables an interpretable, detailed assessment of memory capabilities of LLMs.
Abstract (translated)
基于大型语言模型（LLM）的AI助手如何有效地利用其记忆（上下文）来执行各种任务？传统的数据基准测试通常由人工创建，存在若干局限性：它们是静态的、容易过拟合、难以解释且缺乏可操作见解——无法在模型未能通过测试时确定具体缺失的能力。在这篇论文中，我们提出了一种框架，用于自动生成一套全面的测试来评估模型利用其记忆的有效能力。我们的框架扩展了常见探索范围之外的能力测试（如密码查找、键值对搜索和大海捞针），这些是文献中的主要关注点。特别地，我们在结构化成不同块的输入上评估模型，执行原子任务，例如搜索、回忆、编辑、匹配和比较上下文记忆中的信息，并在模拟现实世界数据的情况下进行基本操作。此外，我们设计了合成测试来调查模型在处理内存时保持状态的能力。我们的基准测试使LLM的记忆能力能够得到可解释且详细的评估。
URL
https://arxiv.org/abs/2502.03358
