Abstract
AI has revolutionised decision-making across various fields. Yet human judgement remains paramount for high-stakes decision-making. This has fueled explorations of collaborative decision-making between humans and AI systems, aiming to leverage the strengths of both. To explore this dynamic, researchers conduct empirical studies, investigating how humans use AI assistance for decision-making and how this collaboration impacts results. A critical aspect of conducting these studies is the role of participants, often recruited through crowdsourcing platforms. The validity of these studies hinges on the behaviours of the participants, hence effective incentives that can potentially affect these behaviours are a key part of designing and executing these studies. In this work, we aim to address the critical role of incentive design for conducting empirical human-AI decision-making studies, focusing on understanding, designing, and documenting incentive schemes. Through a thematic review of existing research, we explored the current practices, challenges, and opportunities associated with incentive design for human-AI decision-making empirical studies. We identified recurring patterns, or themes, such as what comprises the components of an incentive scheme, how incentive schemes are manipulated by researchers, and the impact they can have on research outcomes. Leveraging the acquired understanding, we curated a set of guidelines to aid researchers in designing effective incentive schemes for their studies, called the Incentive-Tuning Framework, outlining how researchers can undertake, reflect on, and document the incentive design process. By advocating for a standardised yet flexible approach to incentive design and contributing valuable insights along with practical tools, we hope to pave the way for more reliable and generalizable knowledge in the field of human-AI decision-making.
Abstract (translated)
人工智能已经彻底改变了各个领域的决策过程。然而，对于高风险的决策而言，人类判断仍然是至关重要的。这激发了人们探索人与AI系统之间协作决策的可能性，旨在利用双方的优势。为了探讨这种动态关系，研究人员进行了实证研究，调查人类如何使用AI辅助进行决策以及这种合作对结果的影响。这些研究的关键在于参与者的行为，而参与者往往通过众包平台招募。这些研究的有效性在很大程度上取决于参与者的动机行为，因此设计能够影响这些行为的激励措施是开展和执行此类研究的重要组成部分。 这项工作旨在解决进行实证人机决策研究中激励机制设计这一关键问题，重点关注理解、设计和记录激励方案。通过现有研究的主题回顾，我们探讨了与人类-人工智能决策实证研究相关的当前做法、挑战和机遇。我们识别出了反复出现的模式或主题，如构成激励计划的要素、研究人员如何操纵激励计划以及这些计划可能对研究成果产生的影响。 借助获得的理解，我们制定了一套指导原则来帮助研究人员设计有效的激励方案以应用于他们的研究中，并称之为“激励调谐框架”。该框架概述了研究人员应如何进行、反思并记录激励设计方案。通过倡导一种标准化但灵活的激励机制设计方法，并贡献有价值的见解和实用工具，我们希望为人类-人工智能决策领域的可靠性和通用性知识奠定基础。
URL
https://arxiv.org/abs/2601.15064
