Abstract
Large language models (LLMs) and multi-agent orchestration are touted as the next leap in machine translation (MT), but their benefits relative to conventional neural MT (NMT) remain unclear. This paper offers an empirical reality check. We benchmark five paradigms, Google Translate (strong NMT baseline), GPT-4o (general-purpose LLM), o1-preview (reasoning-enhanced LLM), and two GPT-4o-powered agentic workflows (sequential three-stage and iterative refinement), on test data drawn from a legal contract and news prose in three English-source pairs: Spanish, Catalan and Turkish. Automatic evaluation is performed with COMET, BLEU, chrF2 and TER; human evaluation is conducted with expert ratings of adequacy and fluency; efficiency with total input-plus-output token counts mapped to April 2025 pricing. Automatic scores still favour the mature NMT system, which ranks first in seven of twelve metric-language combinations; o1-preview ties or places second in most remaining cases, while both multi-agent workflows trail. Human evaluation reverses part of this narrative: o1-preview produces the most adequate and fluent output in five of six comparisons, and the iterative agent edges ahead once, indicating that reasoning layers capture semantic nuance undervalued by surface metrics. Yet these qualitative gains carry steep costs. The sequential agent consumes roughly five times, and the iterative agent fifteen times, the tokens used by NMT or single-pass LLMs. We advocate multidimensional, cost-aware evaluation protocols and highlight research directions that could tip the balance: leaner coordination strategies, selective agent activation, and hybrid pipelines combining single-pass LLMs with targeted agent intervention.
Abstract (translated)
大型语言模型（LLM）和多代理编排被吹捧为机器翻译（MT）的下一个飞跃，但它们相对于传统的神经网络机器翻译（NMT）的优势仍然不明确。本文提供了一个实证性的现实检验。我们在三种英语源对——西班牙语、加泰罗尼亚语和土耳其语——中选择测试数据来评估五种范式：Google Translate（强大的NMT基准）、GPT-4o（通用LLM）、o1-preview（增强推理的LLM）以及两种由GPT-4o驱动的代理工作流程（顺序三阶段和迭代细化）。自动评价使用COMET、BLEU、chrF2和TER进行；人工评估则通过专家对准确性和流畅性的评分来进行；效率则用输入加输出令牌总数映射到2025年4月的价格来衡量。在十二个指标语言组合中，成熟的NMT系统仍然在七个组合中得分最高；o1-preview在其余大部分情况下的排名紧随其后或并列第二，而两个多代理工作流程则落后。 人工评估部分逆转了这一结果：在六个比较中的五个中，o1-preview产生了最准确和流畅的输出，而在一次比较中迭代代理略胜一筹，这表明推理层捕捉到了表面指标所忽视的意义细微差别。然而，这些定性的提升伴随着高昂的成本：顺序代理消耗的令牌大约是NMT或单次通过LLM五倍左右，而迭代代理则为十五倍。 我们提倡采用多维度且考虑成本的评估协议，并指出可以改变这种平衡的研究方向，包括更精简的协调策略、选择性地激活代理以及结合单次通过LLM和有针对性的代理干预的混合管道。
URL
https://arxiv.org/abs/2505.01560
