Abstract
This paper proposes a framework for modeling multimodal conversational turn organization via the proposition of correlations between language and interactive gestures, based on analysis as to how pragmatic frames are conceptualized and evoked by communicators. As a means to provide evidence for the analysis, we developed an annotation methodology to enrich a multimodal dataset (annotated for semantic frames) with pragmatic frames modeling conversational turn organization. Although conversational turn organization has been studied by researchers from diverse fields, the specific strategies, especially gestures used by communicators, had not yet been encoded in a dataset that can be used for machine learning. To fill this gap, we enriched the Frame2 dataset with annotations of gestures used for turn organization. The Frame2 dataset features 10 episodes from the Brazilian TV series Pedro Pelo Mundo annotated for semantic frames evoked in both video and text. This dataset allowed us to closely observe how communicators use interactive gestures outside a laboratory, in settings, to our knowledge, not previously recorded in related literature. Our results have confirmed that communicators involved in face-to-face conversation make use of gestures as a tool for passing, taking and keeping conversational turns, and also revealed variations of some gestures that had not been documented before. We propose that the use of these gestures arises from the conceptualization of pragmatic frames, involving mental spaces, blending and conceptual metaphors. In addition, our data demonstrate that the annotation of pragmatic frames contributes to a deeper understanding of human cognition and language.
Abstract (translated)
本文提出了一种通过探讨语言与互动手势之间的相关性来建模多模式对话轮次组织的框架，该框架基于对实用语境如何由交际者构想和唤起的分析。为了为这一分析提供证据，我们开发了一种标注方法，以增强一个已用语义框架标注过的多模态数据集，并对其进行对话轮次组织的实用性框架建模。尽管来自不同领域的研究人员已经研究了对话轮次组织，但具体策略（尤其是交际者使用的手势）尚未在可用于机器学习的数据集中编码。为填补这一空白，我们通过添加用于转话语组织的手势标注来丰富Frame2数据集。Frame2数据集包含了10个取自巴西电视剧《Pedro Pelo Mundo》的片段，并对视频和文本中唤起的语义框架进行了标注。该数据集使我们可以仔细观察交际者在实验室外的情境下如何使用互动手势，据我们所知，这些情境之前并未被相关文献记录。 我们的研究结果证实了面对面对话中的交际者利用手势作为传递、获取和保持对话轮次的工具，并揭示了一些以前未被记录的手势的变化形式。我们认为这些手势的使用源于实用性框架的概念化，包括心理空间、融合以及概念隐喻。此外，我们的数据还表明，对实用语境进行标注有助于更深入地理解人类认知与语言的关系。
URL
https://arxiv.org/abs/2509.09804
