Abstract
Long-running LLM agents require persistent memory to preserve state across interactions, yet most deployed systems manage memory with age-based retention (e.g., TTL). While TTL bounds item lifetime, it does not bound the computational footprint of memory on the request path: as retained items accumulate, retrieval candidate sets and vector similarity scans can grow unpredictably, yielding heavy-tailed latency and unstable throughput. We present AMV-L (Adaptive Memory Value Lifecycle), a memory-management framework that treats agent memory as a managed systems resource. AMV-L assigns each memory item a continuously updated utility score and uses value-driven promotion, demotion, and eviction to maintain lifecycle tiers; retrieval is restricted to a bounded, tier-aware candidate set that decouples the request-path working set from total retained memory. We implement AMV-L in a full-stack LLM serving system and evaluate it under identical long-running workloads against two baselines: TTL and an LRU working-set policy, with fixed prompt-injection caps. Relative to TTL, AMV-L improves throughput by 3.1x and reduces latency by 4.2x (median), 4.7x (p95), and 4.4x (p99), while reducing the fraction of requests exceeding 2s from 13.8% to 0.007%. Compared to LRU, AMV-L trades a small regression in median/p95 latency (+26% / +3%) for improved extreme-tail behavior (-15% p99; -98% >2s) and lower token overhead (approximately 6% fewer tokens/request), while matching retrieval quality (value means within approximately 0-2%). The gains arise primarily from bounding retrieval-set size and vector-search work, not from shortening prompts. Our results show that predictable performance for long-running LLM agents requires explicit control of memory working-set size and value-driven lifecycle management, rather than retention time alone.
Abstract (translated)
长时间运行的LLM代理需要持久性内存来跨交互保持状态，但大多数部署系统使用基于年龄的保留策略（例如TTL）管理内存。虽然TTL可以限制项目的生命期，但它并不能约束请求路径上内存的计算开销：随着被保留项目的积累，检索候选集和向量相似度扫描可能会无序增长，导致尾部延迟加重和吞吐量不稳定。我们提出了AMV-L（自适应记忆价值生命周期），这是一个将代理内存视为管理系统的资源的记忆管理框架。AMV-L为每个内存项分配一个持续更新的效用分数，并使用基于价值驱动的提升、降级和驱逐机制来维持生命周期层级；检索被限制在一个有界且层次感知的候选集中，从而解耦请求路径的工作集与总保留内存的关系。 我们在整个LLM服务系统中实现了AMV-L，并在其下进行了评估，条件是在相同的长时间运行工作负载下对比两个基准：TTL和一个固定提示注入上限的LRU工作集策略。相比TTL，AMV-L在吞吐量上提高了3.1倍，在延迟上减少了4.2倍（中位数）、4.7倍（p95）以及4.4倍（p99），同时将超过2秒请求的比例从13.8%降低至0.007%。 与LRU相比，AMV-L在中位数和p95延迟方面略有退步（+26%，+3%），但在极端尾部行为上表现更好（-15% p99；-98% > 2s）并且具有较低的令牌开销（约减少每请求6%的令牌），同时检索质量相当（值均在大约0至2%范围内）。这些改进主要来源于限制检索集合大小和向量搜索工作的计算负担，而不是缩短提示。 我们的结果显示，对于长时间运行的LLM代理而言，可预测性能需要对内存工作集大小进行显式控制以及基于价值驱动的生命周期管理，而不仅仅是保留时间。
URL
https://arxiv.org/abs/2603.04443
