Abstract
The iterative character of work in machine learning (ML) and artificial intelligence (AI) and reliance on comparisons against benchmark datasets emphasize the importance of reproducibility in that literature. Yet, resource constraints and inadequate documentation can make running replications particularly challenging. Our work explores the potential of using downstream citation contexts as a signal of reproducibility. We introduce a sentiment analysis framework applied to citation contexts from papers involved in Machine Learning Reproducibility Challenges in order to interpret the positive or negative outcomes of reproduction attempts. Our contributions include training classifiers for reproducibility-related contexts and sentiment analysis, and exploring correlations between citation context sentiment and reproducibility scores. Study data, software, and an artifact appendix are publicly available at this https URL .
Abstract (translated)
机器学习（ML）和人工智能（AI）中工作的迭代性质以及依赖基准数据集的特点强调了在文献中可重复性的重要性。然而，资源限制和不足的文档可以使运行复制尤其具有挑战性。我们的工作探讨了使用下游引用上下文作为可重复性信号的可能性。我们引入了一个应用于参与机器学习可重复性挑战的论文的引用上下文的情绪分析框架，以解释繁殖尝试的积极或消极结果。我们的贡献包括为可重复性相关上下文训练分类器和探索引用上下文情感与可重复性评分之间的相关性。研究数据、软件和一件附件都可以在https://url.com/这个URL公开使用。
URL
https://arxiv.org/abs/2405.03977
