Abstract
In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. In a conversational setting such signals are usually unavailable due to the nature of the interactions, and, instead, the evaluation often relies on crowdsourced evaluation labels. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems (TDSs), is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated. We explore and compare two methodologies for assessing TDSs: one includes the user's follow-up utterance and one without. We use both crowdworkers and large language models (LLMs) as annotators to assess system responses across four aspects: relevance, usefulness, interestingness, and explanation quality. Our findings indicate that there is a distinct difference in ratings assigned by both annotator groups in the two setups, indicating user feedback does influence system evaluation. Workers are more susceptible to user feedback on usefulness and interestingness compared to LLMs on interestingness and relevance. User feedback leads to a more personalized assessment of usefulness by workers, aligning closely with the user's explicit feedback. Additionally, in cases of ambiguous or complex user requests, user feedback improves agreement among crowdworkers. These findings emphasize the significance of user feedback in refining system evaluations and suggest the potential for automated feedback integration in future research. We publicly release the annotated data to foster research in this area.
Abstract (translated)
在便携式检索中，评估很大程度上依赖于用户行为，包括隐含反馈。在会话环境中，这些信号通常由于交互的性质而无法获得，相反，评估通常依赖于由 crowdworkers 提供的手动评估标签。用户反馈在会话感知中 turn 的评估者中的作用已经被研究得很少。我们关注的是，考虑用户反馈（无论是显性还是隐性反馈）如何影响会话导向对话系统（TDSs）的评估。我们探讨并比较了两种评估 TDSs 的方法：一种包括用户的后续陈述，另一种不包括。我们使用 both crowdworkers 和 large language models (LLMs) 作为标注者来评估系统的响应跨越四个方面：相关性、有用性、有趣性和解释质量。我们的研究结果表明，在两种设置中，评估者小组分配给用户的评分存在显著差异，这表明用户反馈确实会影响系统评估。与 LLMs 相比，工人在有用性和有趣性上更容易受到用户反馈的影响。用户反馈使工作者能够进行更个性化的有用性评估，与用户的明确反馈非常贴近。此外，在模糊或复杂用户请求的情况下，用户反馈会改善 crowdworkers 之间的共识。这些发现强调了用户反馈在优化系统评估中的重要性，并表明在未来的研究中，自动反馈集成具有很大的潜力。我们公开发布评估数据，以促进该领域的研究。
URL
https://arxiv.org/abs/2404.12994
