Abstract
Large Language Models (LLMs) have shown strong capability in diverse software engineering tasks, e.g. code completion, bug fixing, and document generation. However, feature-driven development (FDD), a highly prevalent real-world task that involves developing new functionalities for large, existing codebases, remains underexplored. We therefore introduce SWE-Dev, the first large-scale dataset (with 14,000 training and 500 test samples) designed to evaluate and train autonomous coding systems on real-world feature development tasks. To ensure verifiable and diverse training, SWE-Dev uniquely provides all instances with a runnable environment and its developer-authored executable unit tests. This collection not only provides high-quality data for Supervised Fine-Tuning (SFT), but also enables Reinforcement Learning (RL) by delivering accurate reward signals from executable unit tests. Our extensive evaluations on SWE-Dev, covering 17 chatbot LLMs, 10 reasoning models, and 10 Multi-Agent Systems (MAS), reveal that FDD is a profoundly challenging frontier for current AI (e.g., Claude-3.7-Sonnet achieves only 22.45\% Pass@3 on the hard test split). Crucially, we demonstrate that SWE-Dev serves as an effective platform for model improvement: fine-tuning on training set enabled a 7B model comparable to GPT-4o on \textit{hard} split, underscoring the value of its high-quality training data. Code is available here \href{this https URL}{this https URL}.
Abstract (translated)
大型语言模型（LLMs）在各种软件工程任务中展现了强大的能力，例如代码补全、错误修复和文档生成。然而，特性驱动开发（FDD），这是一种高度流行的真实世界任务，涉及到为庞大的现有代码库添加新功能，这一领域仍然被较少探索。为此，我们引入了SWE-Dev，这是首个大规模数据集（包含14,000个训练样本和500个测试样本），旨在评估和训练自动编码系统在真实世界的特性开发任务上的表现。为了确保可验证且多样化的训练过程，SWE-Dev独特地为所有实例提供了一个运行环境及其由开发者编写的执行单元测试。 这个数据集不仅提供了高质量的数据用于监督微调（SFT），而且还通过提供来自可执行单元测试的准确奖励信号支持强化学习（RL）。我们在SWE-Dev上进行了广泛评估，涵盖了17个聊天机器人LLM、10个推理模型和10个多智能体系统（MAS），发现FDD是当前AI面临的深刻挑战前沿（例如，Claude-3.7-Sonnet在困难测试分割上的Pass@3仅达到22.45%）。至关重要的是，我们展示了SWE-Dev作为一个有效的模型改进平台的作用：在训练集上进行微调使一个70亿参数的模型在“困难”分段的表现可媲美GPT-4o，强调了其高质量训练数据的价值。 代码可以在[\href{this https URL}{此处}]获取。
URL
https://arxiv.org/abs/2505.16975
