Abstract
Conversational systems relying on text-based large language models (LLMs) often overlook paralinguistic cues, essential for understanding emotions and intentions. Speech-language models (SLMs), which use speech as input, are emerging as a promising solution. However, SLMs built by extending frozen LLMs struggle to capture paralinguistic information and exhibit reduced context understanding. We identify entangled information and improper training strategies as key issues. To address these issues, we propose two heterogeneous adapters and suggest a weakly supervised training strategy. Our approach disentangles paralinguistic and linguistic information, enabling SLMs to interpret speech through structured representations. It also preserves contextual understanding by avoiding the generation of task-specific vectors through controlled randomness. This approach trains only the adapters on common datasets, ensuring parameter and data efficiency. Experiments demonstrate competitive performance in emotional conversation tasks, showcasing the model's ability to effectively integrate both paralinguistic and linguistic information within contextual settings.
Abstract (translated)
依赖文本的大规模语言模型（LLM）构建的对话系统往往忽视了对情感和意图理解至关重要的副语言线索。以语音为输入的语音-语言模型（SLMs）正逐渐成为解决这一问题的有前景方案。然而，基于冻结状态下的LLM扩展而来的SLMs难以捕捉到副语言信息，并且在上下文理解方面表现较差。我们发现纠缠的信息和不当的训练策略是主要问题所在。 为了应对这些问题，我们提出两种异构适配器并建议采用弱监督式训练策略。我们的方法能够分离出副语言与语言信息，使得SLMs可以通过结构化表示来解读语音，并通过控制随机性避免生成特定任务向量的方式保持上下文理解能力。这一方法仅在通用数据集上对适配器进行训练，从而确保参数和数据的高效使用。 实验结果表明，在情感对话等任务中我们的模型能表现出色，这证明了该模型能够有效地将副语言与语言信息融入到特定的上下文中去。
URL
https://arxiv.org/abs/2508.08095
