Abstract
Nonverbal communication is integral to human interaction, with gestures, facial expressions, and body language conveying critical aspects of intent and emotion. However, existing large language models (LLMs) fail to effectively incorporate these nonverbal elements, limiting their capacity to create fully immersive conversational experiences. We introduce MARS, a multimodal language model designed to understand and generate nonverbal cues alongside text, bridging this gap in conversational AI. Our key innovation is VENUS, a large-scale dataset comprising annotated videos with time-aligned text, facial expressions, and body language. Leveraging VENUS, we train MARS with a next-token prediction objective, combining text with vector-quantized nonverbal representations to achieve multimodal understanding and generation within a unified framework. Based on various analyses of the VENUS datasets, we validate its substantial scale and high effectiveness. Our quantitative and qualitative results demonstrate that MARS successfully generates text and nonverbal languages, corresponding to conversational input.
Abstract (translated)
非言语沟通在人类互动中至关重要，手势、面部表情和身体语言传达了意图和情感的重要方面。然而，现有的大型语言模型（LLM）未能有效地融合这些非言语元素，这限制了它们创造全面沉浸式对话体验的能力。为此，我们推出了MARS，这是一个多模态语言模型，旨在理解并生成与文本相关的非言语线索，在会话AI中填补这一空白。我们的关键创新是VENUS，一个大规模的数据集，包括带有时间对齐的文字、面部表情和身体语言的注释视频。利用VENUS数据集，我们通过结合文字与向量量化后的非言语表示进行训练，以实现多模态理解和生成的一体化框架。基于对VENUS数据集的各种分析，我们验证了其规模庞大且有效性高。我们的定量和定性结果显示，MARS成功地根据对话输入产生了相应的文本和非言语表达。
URL
https://arxiv.org/abs/2506.00958
