Abstract
When face-to-face communication becomes effortful due to background noise or interfering talkers, the role of visual cues becomes increasingly important for communication success. While previous research has selectively examined head or hand movements, here we explore movements of the whole body in acoustically adverse conditions. We hypothesized that increasing background noise in conversations would lead to increased gesture frequency in hand, head, trunk, and leg movements typical of conversation. Increased use of hand movements should support the speaker's role, while increased head and trunk movements may help the listener. We conducted a free dyadic conversation experiment with normal-hearing participants (n=8) in a virtual acoustic environment. Conversational movements were described with a newly developed labeling system for typical conversational actions, and the frequency of individual types was analyzed. In addition, we analyzed gesture quality by assessing hand-speech synchrony, with the hypothesis that higher levels of background noise would lead to a loss of synchrony according to an interactive coupling model. Higher noise levels led to increased hand-gesture complexity during speaking and listening, more pronounced up-down head movements, and contrary to expectations, head movements during listening generally decreased relative to speaking. Synchrony and peak velocity were unaffected by noise, while gesture quality scaled only modestly. The results support previous findings regarding gesturing frequency, but we found only limited evidence for changes in speech-gesture synchrony. This work reveals communication patterns of the whole body and illustrates multimodal adaptation to communication demands.
Abstract (translated)
当面对面交流因背景噪音或干扰说话者而变得困难时，视觉线索在沟通成功中的作用变得更加重要。虽然之前的研究所选择地研究头部或手部动作，但在这里我们探讨了在声学条件不利的情况下全身运动的作用。我们的假设是，在对话中增加背景噪声会导致手势频率的增加，包括手、头、躯干和腿部的动作。手部动作的增加应该支持说话者的角色，而头部和躯干动作的增加可能有助于听者。 我们进行了一项自由双人对话实验，参与者为正常听力人士（n=8），在虚拟声学环境中完成。对话运动用一个新的标签系统来描述典型的对话行为，并分析了个别类型的频率。此外，通过评估手部动作与言语的同步性，我们还分析了手势质量，假设根据交互耦合模型，在较高水平背景噪声的情况下会失去这种同步性。 较高的噪音水平导致在说话和倾听时的手势复杂度增加，头部上下运动更加明显，但出乎意料的是，听者头部位移通常相对于说话而言减少了。同步性和峰值速度不受噪声的影响，而手势质量仅适度变化。结果支持了关于手势频率的先前发现，但我们仅发现了有限的证据表明语音与动作之间的同步性有所改变。 这项工作揭示了整个身体在交流中的模式，并展示了多模态适应于沟通需求的方式。
URL
https://arxiv.org/abs/2512.03636
