Abstract
Recent advancements in speech captioning models have enabled the generation of rich, fine-grained captions for emotional speech. However, the evaluation of such captions remains a critical bottleneck: traditional N-gram metrics fail to capture semantic nuances, while LLM judges often suffer from reasoning inconsistency and context-collapse when processing long-form descriptions. In this work, we propose EmoSURA, a novel evaluation framework that shifts the paradigm from holistic scoring to atomic verification. EmoSURA decomposes complex captions into Atomic Perceptual Units, which are self-contained statements regarding vocal or emotional attributes, and employs an audio-grounded verification mechanism to validate each unit against the raw speech signal. Furthermore, we address the scarcity of standardized evaluation resources by introducing SURABench, a carefully balanced and stratified benchmark. Our experiments show that EmoSURA achieves a positive correlation with human judgments, offering a more reliable assessment for long-form captions compared to traditional metrics, which demonstrated negative correlations due to their sensitivity to caption length.
Abstract (translated)
最近在语音字幕生成模型方面的进展已经使情感化语音的丰富且精细的字幕生成成为可能。然而，评估这类字幕仍然是一个关键瓶颈：传统的N-gram度量无法捕捉语义细微差别，而大型语言模型（LLM）评委在处理长篇描述时常常会遭遇推理不一致和上下文坍缩的问题。 为了解决这个问题，我们提出了EmoSURA这一新颖的评估框架。该框架从整体评分转向原子验证，将复杂的字幕分解成原子感知单元，这些单元是关于语音或情感属性的独立陈述，并采用基于音频的验证机制来验证每个单位与原始语音信号的一致性。 此外，为了应对标准化评估资源匮乏的问题，我们引入了SURABench，这是一个精心平衡和分层的标准。我们的实验表明，EmoSURA能够与人类判断产生正相关关系，在对长篇字幕进行评价时相比传统度量标准提供更为可靠的评估结果（传统度量标准由于其对字幕长度的敏感性而产生了负相关）。
URL
https://arxiv.org/abs/2603.09820
