Abstract
Acquiring large-scale emotional speech data with strong consistency remains a challenge for speech synthesis. This paper presents MIKU-PAL, a fully automated multimodal pipeline for extracting high-consistency emotional speech from unlabeled video data. Leveraging face detection and tracking algorithms, we developed an automatic emotion analysis system using a multimodal large language model (MLLM). Our results demonstrate that MIKU-PAL can achieve human-level accuracy (68.5% on MELD) and superior consistency (0.93 Fleiss kappa score) while being much cheaper and faster than human annotation. With the high-quality, flexible, and consistent annotation from MIKU-PAL, we can annotate fine-grained speech emotion categories of up to 26 types, validated by human annotators with 83% rationality ratings. Based on our proposed system, we further released a fine-grained emotional speech dataset MIKU-EmoBench(131.2 hours) as a new benchmark for emotional text-to-speech and visual voice cloning.
Abstract (translated)
获取大规模且一致性较强的情感语音数据仍然是语音合成领域的一个挑战。本文介绍了一种名为MIKU-PAL的全自动多模态管道，用于从无标签视频数据中提取高一致性的感情语音。通过利用面部检测和跟踪算法，我们开发了一个基于多模态大型语言模型（MLLM）的自动情感分析系统。我们的结果表明，MIKU-PAL能够在MELD数据集上实现与人类标注相当的准确率（68.5%），并且具有更好的一致性（Fleiss Kappa评分为0.93），同时成本更低、速度更快。基于MIKU-PAL提供的高质量、灵活且一致的标注，我们能够为精细的情感语音类别进行标签标注，涵盖了多达26种类型，并得到了人类注释者的83%合理性评分的认可。 在此基础上，我们进一步发布了一个细粒度情感语音数据集MIKU-EmoBench（131.2小时），旨在成为情感文本到语音和视觉声音克隆的新基准。
URL
https://arxiv.org/abs/2505.15772
