Abstract
Scientific knowledge bases accelerate discovery by curating findings from primary literature into structured, queryable formats for both human researchers and emerging AI systems. Maintaining these resources requires expert curators to search relevant papers, reconcile evidence across documents, and produce ontology-grounded annotations - a workflow that existing benchmarks, focused on isolated subtasks like named entity recognition or relation extraction, do not capture. We present FlyBench to evaluate AI agents on end-to-end agentic ontology curation from scientific literature. Given only a gene symbol, agents must search and read from a corpus of 16,898 full-text papers to produce structured annotations: Gene Ontology terms describing function, expression patterns, and historical synonyms linking decades of nomenclature. The benchmark includes 7,397 expert-curated annotations across 100 genes drawn from FlyBase, the Drosophila (fruit fly) knowledge base. We evaluate four baseline agent architectures: memorization, fixed pipeline, single-agent, and multi-agent. We find that architectural choices significantly impact performance, with multi-agent designs outperforming simpler alternatives, yet scaling backbone models yields diminishing returns. All baselines leave substantial room for improvement. Our analysis surfaces several findings to guide future development; for example, agents primarily use retrieval to confirm parametric knowledge rather than discover new information. We hope FlyBench will drive progress on retrieval-augmented scientific reasoning, a capability with broad applications across scientific domains.
Abstract (translated)
科学知识库通过将原始文献中的发现整理成结构化、可查询的格式，加速了科研领域的发现过程。这些资源不仅供人类研究人员使用，还为新兴的人工智能系统提供支持。然而，维护这类资源需要专家编目人员来搜索相关论文，跨文档比对证据，并生成基于本体论的注释——这是一项现有基准测试无法完全捕捉的工作流程，因为现有的基准测试主要集中在孤立的任务上，如命名实体识别或关系抽取等。 我们提出了FlyBench这一评估AI代理在从科学文献中进行端到端代理式本体编目的方法。给定一个基因符号后，代理必须从包含16,898篇全文论文的语料库中搜索并阅读，以生成结构化的注释：描述功能、表达模式以及连接几十年命名惯例的历史同义词的Gene Ontology术语。该基准测试包含了来自FlyBase（果蝇知识库）的100个基因中的7397项专家编目的注释。 我们评估了四种基础代理架构：记忆型、固定管道型、单一代理型和多代理型。研究发现，架构选择对性能有显著影响，其中多代理设计优于简单的替代方案，但扩展模型主体的规模带来的收益却逐渐减少。所有基准测试的表现都有改进的空间。 我们的分析揭示了一些指导未来发展的关键点；例如，代理主要依靠检索来确认参数知识而不是发现新的信息。我们希望FlyBench能够推动基于检索的科学推理的进步，这一能力在各个科研领域内都有着广泛的应用前景。
URL
https://arxiv.org/abs/2602.09163
