Abstract
Conversational information seeking has evolved rapidly in the last few years with the development of Large Language Models (LLMs), providing the basis for interpreting and responding in a naturalistic manner to user requests. The extended TREC Interactive Knowledge Assistance Track (iKAT) collection aims to enable researchers to test and evaluate their Conversational Search Agents (CSA). The collection contains a set of 36 personalized dialogues over 20 different topics each coupled with a Personal Text Knowledge Base (PTKB) that defines the bespoke user personas. A total of 344 turns with approximately 26,000 passages are provided as assessments on relevance, as well as additional assessments on generated responses over four key dimensions: relevance, completeness, groundedness, and naturalness. The collection challenges CSA to efficiently navigate diverse personal contexts, elicit pertinent persona information, and employ context for relevant conversations. The integration of a PTKB and the emphasis on decisional search tasks contribute to the uniqueness of this test collection, making it an essential benchmark for advancing research in conversational and interactive knowledge assistants.
Abstract (translated)
近年来，随着大型语言模型（LLMs）的发展，会话信息寻求已经迅速发展，为用户提供了以自然方式理解和回应请求的基础。TREC Interactive Knowledge Assistance Track (iKAT)扩展收藏旨在使研究人员能够测试和评估他们的会话搜索代理（CSA）。该收藏包含20个不同主题的个性化对话，每个主题都附带一个个人文本知识库（PTKB），定义了特定的用户人格。该收藏提供了关于相关性的评估以及关于生成响应的四个关键方面的额外评估：相关性、完整性、 groundedness 和自然性。该收藏挑战CSA有效地浏览多样的人格背景，唤起相关人物信息，并利用相关对话的上下文。集成PTKB和强调决策搜索任务使该测试收藏独特，成为推动研究在会话和交互式知识助手领域的重要基准。
URL
https://arxiv.org/abs/2405.02637
