Abstract
Current speech language models exceed the size and latency constraints of many deployment environments. We build compact, expressive speech generation models through layer-aligned distillation, matching hidden states, attention maps, and softened logits to compress large multimodal transformers by 3x with minimal loss in performance. We introduce TinyWave, a family of 2B-parameter models for speech-to-speech and interleaved speech-text generation, trained on 50,000 hours of public audio. TinyWave supports (i) speech-only generation using phonetic or expressive tokens and (ii) mixed speech-text continuations. Evaluation on Libri-Light shows TinyWave within 1.4 normalized perplexity points of its teacher. Accuracy on spoken StoryCloze and SALMon reaches 93-97% of the teacher's performance, outperforming size-matched baselines. These models are optimized for deployment on commodity hardware, enabling applications in real-time conversational agents, assistive technologies, and low-resource environments. We release models, training code, and evaluation scripts to support reproducible research on compact, expressive speech generation.
Abstract (translated)
当前的语音语言模型超出了许多部署环境中的大小和延迟限制。我们通过层对齐蒸馏、匹配隐藏状态、注意力图以及软化后的logits，将大型多模态变压器压缩了3倍，并且性能几乎没有损失。我们介绍了TinyWave，这是一个参数量为20亿的模型家族，用于语音到语音及交错语音文本生成，其训练数据集包含了5万小时的公开音频。TinyWave支持(i)仅使用音素或表达性标记进行语音生成；(ii)混合语音-文本延续。在Libri-Light上的评估显示，TinyWave的困惑度比它的教师模型只低1.4个标准化点。对于口语StoryCloze和SALMon任务，准确率达到了老师模型性能的93%-97%，超过了与之大小相匹配的基础模型。这些模型针对商品硬件进行了优化，使其实时对话代理、辅助技术和资源匮乏环境中的应用成为可能。我们发布了模型、训练代码以及评估脚本以支持关于紧凑且表达性语音生成的研究复现。
URL
https://arxiv.org/abs/2506.23670
