Abstract
Large Language Models (LLMs) have recently emerged as capable coding assistants that operate over large codebases through either agentic exploration or full-context generation. Existing benchmarks capture a broad range of coding capabilities, such as resolving GitHub issues, but none of them directly isolate and measure how effectively LLMs leverage repository-level context during code generation. To address this, we introduce ReCUBE, a benchmark in which LLMs reconstruct a masked file within a real-world repository, using all remaining source files, dependency specifications, and documentation as their only source of context. ReCUBE evaluates reconstructed code with usage-aware test cases that simulate both internal module logic and external cross-file integration, reflecting real-world software usage patterns. We further propose the Caller-Centric Exploration (CCE) toolkit, a set of dependency graph-based tools that can be integrated into agentic frameworks to guide agents toward the most relevant caller files during repository exploration. Experiments across eight models in four settings show that repository-level context utilization remains highly challenging even for state-of-the-art models, with GPT-5 achieving only 37.57% strict pass rate in the full-context setting. Agents augmented with our CCE toolkit consistently outperform all baselines across all evaluated models, with improvements of up to 7.56% in strict pass rate. We release our benchmark, code, and evaluation framework as open source for the NLP research community.
Abstract (translated)
大语言模型（LLMs）已作为能操作大型代码库的编码助手出现，其通过智能体探索或全上下文生成的方式工作。现有基准测试虽能评估广泛的编码能力（如解决GitHub问题），但均未直接分离和衡量LLMs在代码生成中利用仓库级上下文的效果。为填补此空白，我们提出ReCUBE——一个让LLMs利用剩余源代码文件、依赖规范及文档作为唯一上下文，在真实仓库中重建被掩码文件的基准测试。ReCUBE通过感知代码使用场景的测试用例评估重建结果，这些用例既模拟内部模块逻辑也模拟跨文件集成，反映真实软件使用模式。我们还提出调用者中心探索（CCE）工具包，这是一套基于依赖图的工具，可集成至智能体框架以引导其在仓库探索时聚焦最相关的调用文件。在四种设置下对八个模型的实验表明，即使是最先进的模型，仓库级上下文利用仍极具挑战性——GPT-5在全上下文设置下仅达到37.57%的严格通过率。集成CCE工具包的智能体在所有评估模型中持续超越所有基线，严格通过率最高提升7.56%。我们已将基准测试、代码及评估框架开源，供自然语言处理研究社区使用。
URL
https://arxiv.org/abs/2603.25770
