Abstract
Machine learning and artificial intelligence conferences such as NeurIPS and ICML now regularly receive tens of thousands of submissions, posing significant challenges to maintaining the quality and consistency of the peer review process. This challenge is particularly acute for best paper awards, which are an important part of the peer review process, yet whose selection has increasingly become a subject of debate in recent years. In this paper, we introduce an author-assisted mechanism to facilitate the selection of best paper awards. Our method employs the Isotonic Mechanism for eliciting authors' assessments of their own submissions in the form of a ranking, which is subsequently utilized to adjust the raw review scores for optimal estimation of the submissions' ground-truth quality. We demonstrate that authors are incentivized to report truthfully when their utility is a convex additive function of the adjusted scores, and we validate this convexity assumption for best paper awards using publicly accessible review data of ICLR from 2019 to 2023 and NeurIPS from 2021 to 2023. Crucially, in the special case where an author has a single quota -- that is, may nominate only one paper -- we prove that truthfulness holds even when the utility function is merely nondecreasing and additive. This finding represents a substantial relaxation of the assumptions required in prior work. For practical implementation, we extend our mechanism to accommodate the common scenario of overlapping authorship. Finally, simulation results demonstrate that our mechanism significantly improves the quality of papers selected for awards.
Abstract (translated)
机器学习和人工智能会议（如NeurIPS和ICML）现在经常收到数万份投稿，这对保持同行评审过程的质量和一致性提出了重大挑战。特别是在最佳论文奖的选择方面，这是一个同行评审过程中非常重要的一部分，然而近年来其评选过程越来越成为争论的焦点。 在本文中，我们提出了一种作者协助机制来促进最佳论文奖项的选择。我们的方法采用了等单调机制（Isotonic Mechanism），以引导作者对其提交的作品进行排名评估，随后利用这些排名调整原始的评审分数，以便更好地估计作品的真实质量。我们展示了当作者的效用函数是调整后评分的凸加性函数时，他们有动力真实地报告信息，并使用ICLR（2019年至2023年）和NeurIPS（2021年至2023年）期间公开可用的评审数据验证了对于最佳论文奖而言这一凸性的假设。尤为重要的是，在作者只有一个配额的情况下——即只能提名一篇论文时，我们证明即使效用函数只是非减加性函数，也能够保持真实报告行为。这一发现大大放宽了先前工作中所需的假设条件。 为了实际应用，我们将我们的机制扩展到处理常见的重叠作者身份的情况。最后，模拟结果表明，我们的机制显著提高了用于奖励的论文的质量。
URL
https://arxiv.org/abs/2601.15249
