Abstract
Comparative evaluation of several systems is a recurrent task in researching. It is a key step before deciding which system to use for our work, or, once our research has been conducted, to demonstrate the potential of the resulting model. Furthermore, it is the main task of competitive, public challenges evaluation. Our proposed software (DEEP) automates both the execution and scoring of machine translation and optical character recognition models. Furthermore, it is easily extensible to other tasks. DEEP is prepared to receive dockerized systems, run them (extracting information at that same time), and assess hypothesis against some references. With this approach, evaluators can achieve a better understanding of the performance of each model. Moreover, the software uses a clustering algorithm based on a statistical analysis of the significance of the results yielded by each model, according to the evaluation metrics. As a result, evaluators are able to identify clusters of performance among the swarm of proposals and have a better understanding of the significance of their differences. Additionally, we offer a visualization web-app to ensure that the results can be adequately understood and interpreted. Finally, we present an exemplary case of use of DEEP.
Abstract (translated)
在研究过程中，多个系统之间的比较评估是一项反复进行的任务。这是决定使用哪种系统来开展我们工作的关键步骤；或者，在完成我们的研究后，展示所获得模型的潜力时也是必不可少的。此外，它还是公开竞赛中评价的主要任务。 我们提出的软件（DEEP）可以自动化执行和评分机器翻译及光学字符识别模型的过程，并且很容易扩展到其他任务上。DEEP准备接收docker化的系统，运行它们的同时提取相关信息，并根据某些参考评估假设。通过这种方法，评审人员能够更好地理解每个模型的表现。此外，该软件使用了一种基于统计分析结果显著性的聚类算法，这些结果来自于每种模型的评价指标。因此，评审者可以识别出提案群中的表现集群，并更清楚地了解它们之间的差异的重要性。 另外，我们提供了一个可视化网络应用程序，确保结果能够被充分理解并解释。最后，我们展示了DEEP的一个典型使用案例。
URL
https://arxiv.org/abs/2602.19583
