arXiv_SD
-
Training-Free Multi-Step Inference for Target Speaker Extraction
2026-03-11 16:05:14Zhenghai You, Ying Shi, Lantian Li, Dong WangAbstract
Target speaker extraction (TSE) aims to recover a target speaker's speech from a mixture using a reference utterance as a cue. Most TSE systems adopt conditional auto-encoder architectures with one-step inference. Inspired by test-time scaling, we propose a training-free multi-step inference method that enables iterative refinement with a frozen pretrained model. At each step, new candidates are generated by interpolating the original mixture and the previous estimate, and the best candidate is selected for further refinement until convergence. Experiments show that, when ground-truth target speech is available, optimizing an intrusive metric (SI-SDRi) yields consistent gains across multiple evaluation metrics. Without ground truth, optimizing non-intrusive metrics (UTMOS or SpkSim) improves the corresponding metric but may hurt others. We therefore introduce joint metric optimization to balance these objectives, enabling controllable extraction preferences for practical deployment.
Abstract (translated)
目标说话人提取（TSE）旨在通过使用参考语音作为线索，从混合信号中恢复出目标说话人的语音。大多数TSE系统采用条件自动编码器架构，并进行一步推理。受测试时间缩放的启发，我们提出了一种无需训练的多步推理方法，该方法能够利用冻结的预训练模型进行迭代细化。在每一步中，通过插值原始混合信号和前一次估计结果生成新的候选语音，然后选择最佳候选者继续进行进一步优化直至收敛。 实验表明，在有真实目标语音的情况下，优化侵入性指标（SI-SDRi）可以跨多个评估标准取得一致的改进效果。而在没有真实数据时，虽然通过非侵入性指标（UTMOS或SpkSim）进行优化能提高对应指标的表现，但可能会损害其他指标的效果。因此，我们引入了联合度量优化方法以平衡这些目标，使实用部署中可以实现可控提取偏好。
URL
https://arxiv.org/abs/2603.10921
PDF
-
VoxCare: Studying Natural Communication Behaviors of Hospital Caregivers through Wearable Sensing of Egocentric Audio
2026-03-11 15:32:13Tiantian Feng, Kleanthis Avramidis, Anfeng Xu, Deqi Wang, Brandon M Booth, Shrikanth NarayananAbstract
Healthcare professionals work in complex, high-stakes environments where effective communication is critical for care delivery, team coordination, and individual well-being. However, communication activity in everyday clinical settings remains challenging to measure and largely unexplored in human behavioral research. We present VoxCare, a scalable egocentric wearable audio sensing and computing system that captures natural communication behaviors of hospital professionals in real-world settings without storing raw audio. VoxCare performs real-time, on-device acoustic feature extraction and applies a speech foundation model-guided teacher-student framework to identify foreground speech activity. From these features, VoxCare derives interpretable behavioral measures of communication frequency, duration, and vocal arousal. Our analyses reveal how, when, and how often clinicians communicate across different shifts and working units, and suggest that communication activity reflects underlying workload and stress. By enabling continuous assessment of communication patterns in everyday contexts, this study provides data-driven approaches to understand the behaviors of healthcare providers and ultimately improve healthcare delivery.
Abstract (translated)
医疗专业人员在复杂且高风险的环境中工作，有效沟通对于护理提供、团队协作和个人福祉至关重要。然而，在日常临床环境中的沟通活动仍然难以量化和研究，特别是在人类行为学领域。本文介绍了VoxCare系统，这是一个可扩展的自为中心式穿戴音频感知与计算系统，能够在不存储原始音频的情况下捕捉医院专业人员在现实世界环境中自然发生的沟通行为。VoxCare能够实时、在设备上进行声学特征提取，并采用以语音基础模型指导的教师-学生框架来识别前景中的言语活动。从这些特征中，VoxCare推导出可解释的行为指标，包括沟通频率、持续时间和语调的紧张程度。 我们的分析揭示了临床医生在不同班次和工作单位中何时以及如何频繁地进行沟通，并表明沟通行为反映了潜在的工作量和压力水平。通过连续评估日常环境中医疗保健提供者的沟通模式，这项研究为理解其行为提供了基于数据的方法，并最终旨在改善医疗服务的质量。
URL
https://arxiv.org/abs/2603.10888
PDF
-
OSUM-Pangu: An Open-Source Multidimension Speech Understanding Foundation Model Built upon OpenPangu on Ascend NPUs
2026-03-11 15:15:04Yujie Liao, Xuelong Geng, Hongfei Xue, Shuiyuan Wang, Lei XieAbstract
Recent advancements in Speech Large Language Models have significantly enhanced multi-dimensional speech understanding. However, the majority of high-performance frameworks are predominantly optimized for GPU centric ecosystems and proprietary backbones, creating a significant gap for deployment on non-CUDA computing infrastructures. In this paper, we present OSUM-Pangu, a fully open-source speech understanding foundation model developed on a completely non-CUDA software and hardware stack. By integrating an audio encoder with the openPangu-7B LLM backbone, we successfully implement the entire training and inference pipeline on the Ascend NPU platform. To facilitate efficient task alignment under non-CUDA resource constraints, we adopt a practical training process that sequentially bridges speech perception and user intent recognition. Experimental results demonstrate that OSUM-Pangu achieves task accuracy comparable to mainstream GPU-based models while maintaining robust natural language interaction capabilities. Our work provides a reproducible, non-CUDA baseline for the open-source speech community, promoting the independent evolution of multimodal intelligence.
Abstract (translated)
最近在语音大型语言模型方面的进展显著提高了多维度的语音理解能力。然而，大多数高性能框架主要针对以GPU为中心的生态系统和专有骨干网络进行了优化，这为部署到非CUDA计算基础设施上造成了明显的差距。本文介绍了OSUM-Pangu，这是一个完全开源的语音理解基础模型，在一个完全不依赖于CUDA软件和硬件堆栈上开发而成。通过将音频编码器与开放源代码的openPangu-7B LLM骨干网络集成，我们成功地在Ascend NPU平台上实现了整个训练和推理流程。为了在非CUDA资源限制条件下促进高效的任务对齐，我们采用了一种实际的培训过程，该过程依次连接语音感知和用户意图识别。实验结果显示，OSUM-Pangu能够达到与主流GPU基础模型相当的任务准确度，并保持强大的自然语言交互能力。我们的工作为开源语音社区提供了可重复、非CUDA的标准基线，推动了多模态智能的独立发展。
URL
https://arxiv.org/abs/2603.10862
PDF
-
Geo-ATBench: A Benchmark for Geospatial Audio Tagging with Geospatial Semantic Context
2026-03-11 10:34:49Yuanbo Hou, Yanru Wu, Qiaoqiao Ren, Shengchen Li, Stephen Roberts, Dick BotteldoorenAbstract
Environmental sound understanding in computational auditory scene analysis (CASA) is often formulated as an audio-only recognition problem. This formulation leaves a persistent drawback in multi-label audio tagging (AT): acoustic similarity can make certain events difficult to separate from waveforms alone. In such cases, disambiguating cues often lie outside the waveform. Geospatial semantic context (GSC), derived from geographic information system data, e.g., points of interest (POI), provides location-tied environmental priors that can help reduce this ambiguity. A systematic study of this direction is enabled through the proposed geospatial audio tagging (Geo-AT) task, which conditions multi-label sound event tagging on GSC alongside audio. To benchmark Geo-AT, Geo-ATBench is introduced as a polyphonic audio benchmark with geographical annotations, containing 10.71 hours of audio across 28 event categories; each clip is paired with a GSC representation from 11 semantic context categories. GeoFusion-AT is proposed as a unified geo-audio fusion framework that evaluates feature-, representation-, and decision-level fusion on representative audio backbones, with audio- and GSC-only baselines. Results show that incorporating GSC improves AT performance, especially on acoustically confounded labels, indicating geospatial semantics provide effective priors beyond audio alone. A crowdsourced listening study with 10 participants on 579 samples shows that there is no significant difference in performance between models on Geo-ATBench labels and aggregated human labels, supporting Geo-ATBench as a human-aligned benchmark. The Geo-AT task, benchmark Geo-ATBench, and reproducible geo-audio fusion framework GeoFusion-AT provide a foundation for studying AT with geospatial semantic context within the CASA community. Dataset, code, models are on homepage (this https URL).
Abstract (translated)
在计算听觉场景分析（CASA）中的环境声音理解通常被表述为一个仅基于音频的识别问题。这种表述方式导致了多标签音频标记（AT）的一个持久缺陷：声学相似性使得某些事件难以仅通过波形来区分。在这种情况下，明确这些事件所需的信息往往存在于波形之外。从地理信息系统数据中提取的地空间语义上下文（GSC），例如兴趣点（POI），提供了与地理位置相关的环境先验信息，有助于减少这种不确定性。 为了系统地研究这一方向，提出了一个基于地空间音频标记（Geo-AT）的任务，该任务在多标签声音事件标注时结合了地理空间语境和音频。为评估Geo-AT，提出了一种名为Geo-ATBench的多声部音频基准测试集，并带有地理位置注释，其中包括跨越28个类别、总计10.71小时的音频；每个音频片段都配有一个从11个语义上下文类别中选取的地空间表示。提出了GeoFusion-AT作为统一的地空间音频融合框架，在代表性音频骨干网络上评估特征级、表征级和决策级融合，并带有仅基于音频和GSC的基线模型。 结果表明，结合地空间信息可以提升音频标记性能，特别是在声学混淆标签上的表现尤为显著。这说明了地空间语义提供了有效的先验信息，超越了单独的音频数据所能提供的范畴。一项由10名参与者参与、涉及579个样本的人群外包聆听研究表明，Geo-ATBench模型与汇总人类标注之间的性能没有显著差异，证明了Geo-ATBench是一个人机一致性的基准测试。 该地空间音频标记任务（Geo-AT）、基准Geo-ATBench以及可重复的地空间音频融合框架GeoFusion-AT为CASA社区研究带有地空间语义上下文的音频标签提供了一个基础。数据集、代码和模型可在其主页上找到（此链接：this https URL）。
URL
https://arxiv.org/abs/2603.10623
PDF
-
Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing
2026-03-11 09:43:49Hao Shi, Yusuke Fujita, Roman Koshkin, Mengjie Zhao, Yuan Gao, Lianbo Liu, Yui SudoAbstract
Large language models (LLMs) provide strong semantic priors that can improve multi-talker automatic speech recognition (MT-ASR), but using an LLM as an autoregressive decoder is computationally expensive and remains fragile under heavy overlap. In this paper, we propose an encoder-only MT-ASR framework that adapts an LLM to multi-talker conditioning and distills its semantic guidance into the encoder during training, while retaining fast CTC-style decoding at inference. Our model employs a post-encoder separator with serialized CTC to produce talker-ordered transcripts, and leverages an adapted LLM-based SOT objective as a multi-talker-aware teacher signal to explicitly regularize mixed-speech representations. To further support variable numbers of talkers, we introduce a Talker-Count Head that predicts the talker count and dynamically selects the appropriate decoding branch. Experiments on LibriMix show that the proposed encoder-only model achieves comparable performance to LLM-based systems in the two-talker condition, while delivering significant improvements in the three-talker condition with significant small RTF.
Abstract (translated)
大型语言模型（LLM）提供了强大的语义先验，可以改善多说话人自动语音识别（MT-ASR），但将LLM用作自回归解码器在计算上是昂贵的，并且在重叠严重时仍然不稳定。本文提出了一种编码器专用的多说话人语音识别框架，该框架使大型语言模型适应于多说话人的条件，并在训练过程中将其语义指导蒸馏到编码器中，同时保持快速的CTC风格解码进行推理。我们的模型采用了后编码分割器和序列化的CTC来生成有序说话人记录，并利用了基于改编LLM的SOT目标作为具有多说话人意识的教师信号，以明确地规范混合语音表示。为了进一步支持可变数量的说话人，我们引入了一个说话人数预测头（Talker-Count Head），该头部可以预测说话人的数量并动态选择适当的解码分支。在LibriMix数据集上的实验表明，所提出的编码器专用模型在双说话人条件下实现了与基于LLM系统的相当性能，在三说话人条件下的表现显著提升，并且实时因子（RTF）较小。
URL
https://arxiv.org/abs/2603.10587
PDF
-
Multimodal Self-Attention Network with Temporal Alignment for Audio-Visual Emotion Recognition
2026-03-11 07:46:25Inyong Koo, yeeun Seong, Minseok Son, Jaehyuk Jang, Changick KimAbstract
Audio-visual emotion recognition (AVER) methods typically fuse utterance-level features, and even frame-level attention models seldom address the frame-rate mismatch across modalities. In this paper, we propose a Transformer-based framework focusing on the temporal alignment of multimodal features. Our design employs a multimodal self-attention encoder that simultaneously captures intra- and inter-modal dependencies within a shared feature space. To address heterogeneous sampling rates, we incorporate Temporally-aligned Rotary Position Embeddings (TaRoPE), which implicitly synchronize audio and video tokens. Furthermore, we introduce a Cross-Temporal Matching (CTM) loss that enforces consistency among temporally proximate pairs, guiding the encoder toward better alignment. Experiments on CREMA-D and RAVDESS datasets demonstrate consistent improvements over recent baselines, suggesting that explicitly addressing frame-rate mismatch helps preserve temporal cues and enhances cross-modal fusion.
Abstract (translated)
音频-视觉情感识别（AVER）方法通常融合话语级特征，即使是一些帧级别的注意力模型也较少解决跨模态的帧率不匹配问题。在本文中，我们提出了一种基于Transformer框架的方法，专注于多模态特征的时间对齐。我们的设计采用了一个多模态自注意编码器，在共享的特征空间内同时捕捉了内部和跨模态之间的依赖关系。为了解决异质采样率的问题，我们引入了时间对齐旋转位置嵌入（TaRoPE），它隐式地同步音频和视频标记。此外，我们提出了一种交叉时间匹配（CTM）损失函数，该函数强制相邻时间点的一致性，引导编码器向更好的对齐方向发展。在CREMA-D和RAVDESS数据集上的实验表明，与最近的基准方法相比，我们的方法取得了持续改进的效果，这表明明确解决帧率不匹配问题有助于保留时间线索并增强跨模态融合效果。
URL
https://arxiv.org/abs/2603.11095
PDF
-
Synthetic Data Domain Adaptation for ASR via LLM-based Text and Phonetic Respelling Augmentation
2026-03-11 06:56:56Natsuo Yamashita, Koichi Nagatsuka, Hiroaki Kokubo, Kota Dohi, Tuan Vu HoAbstract
End-to-end automatic speech recognition often degrades on domain-specific data due to scarce in-domain resources. We propose a synthetic-data-based domain adaptation framework with two contributions: (1) a large language model (LLM)-based text augmentation pipeline with a filtering strategy that balances lexical diversity, perplexity, and domain-term coverage, and (2) phonetic respelling augmentation (PRA), a novel method that introduces pronunciation variability through LLM-generated orthographic pseudo-spellings. Unlike conventional acoustic-level methods such as SpecAugment, PRA provides phonetic diversity before speech synthesis, enabling synthetic speech to better approximate real-world variability. Experimental results across four domain-specific datasets demonstrate consistent reductions in word error rate, confirming that combining domain-specific lexical coverage with realistic pronunciation variation significantly improves ASR robustness.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16920
PDF
-
V2A-DPO: Omni-Preference Optimization for Video-to-Audio Generation
2026-03-11 05:56:14Nolan Chan, Timmy Gang, Yongqian Wang, Yuzhe Liang, Dingdong WangAbstract
This paper introduces V2A-DPO, a novel Direct Preference Optimization (DPO) framework tailored for flow-based video-to-audio generation (V2A) models, incorporating key adaptations to effectively align generated audio with human preferences. Our approach incorporates three core innovations: (1) AudioScore-a comprehensive human preference-aligned scoring system for assessing semantic consistency, temporal alignment, and perceptual quality of synthesized audio; (2) an automated AudioScore-driven pipeline for generating large-scale preference pair data for DPO optimization; (3) a curriculum learning-empowered DPO optimization strategy specifically tailored for flow-based generative models. Experiments on benchmark VGGSound dataset demonstrate that human-preference aligned Frieren and MMAudio using V2A-DPO outperform their counterparts optimized using Denoising Diffusion Policy Optimization (DDPO) as well as pre-trained baselines. Furthermore, our DPO-optimized MMAudio achieves state-of-the-art performance across multiple metrics, surpassing published V2A models.
Abstract (translated)
本文介绍了V2A-DPO，这是一种新颖的直接偏好优化（DPO）框架，专门针对基于流的视频到音频生成（V2A）模型进行设计，并作出关键调整以有效使生成的音频与人类偏好相一致。我们的方法包含三个核心创新： 1. AudioScore：一种综合的人类偏好对齐评分系统，用于评估合成音频在语义一致性、时间同步性和感知质量方面的表现。 2. 自动化的AudioScore驱动管道，用于为DPO优化生成大规模的偏好配对数据。 3. 一种基于课程学习增强的DPO优化策略，特别针对流式生成模型进行了定制。 在基准VGGSound数据集上的实验表明，使用V2A-DPO的人类偏好对齐Frieren和MMAudio优于使用去噪扩散策略优化（DDPO）以及预训练基线进行优化的同类模型。此外，我们通过DPO优化的MMAudio在多个指标上达到了最先进的性能水平，超越了已发布的视频到音频生成模型。
URL
https://arxiv.org/abs/2603.11089
PDF
-
FireRedASR2S: A State-of-the-Art Industrial-Grade All-in-One Automatic Speech Recognition System
2026-03-11 05:11:06Kaituo Xu, Yan Jia, Kai Huang, Junjie Chen, Wenpeng Li, Kun Liu, Feng-Long Xie, Xu Tang, Yao HuAbstract
We present FireRedASR2S, a state-of-the-art industrial-grade all-in-one automatic speech recognition (ASR) system. It integrates four modules in a unified pipeline: ASR, Voice Activity Detection (VAD), Spoken Language Identification (LID), and Punctuation Prediction (Punc). All modules achieve SOTA performance on the evaluated benchmarks: FireRedASR2: An ASR module with two variants, FireRedASR2-LLM (8B+ parameters) and FireRedASR2-AED (1B+ parameters), supporting speech and singing transcription for Mandarin, Chinese dialects and accents, English, and code-switching. Compared to FireRedASR, FireRedASR2 delivers improved recognition accuracy and broader dialect and accent coverage. FireRedASR2-LLM achieves 2.89% average CER on 4 public Mandarin benchmarks and 11.55% on 19 public Chinese dialects and accents benchmarks, outperforming competitive baselines including Doubao-ASR, Qwen3-ASR, and Fun-ASR. FireRedVAD: An ultra-lightweight module (0.6M parameters) based on the Deep Feedforward Sequential Memory Network (DFSMN), supporting streaming VAD, non-streaming VAD, and multi-label VAD (mVAD). On the FLEURS-VAD-102 benchmark, it achieves 97.57% frame-level F1 and 99.60% AUC-ROC, outperforming Silero-VAD, TEN-VAD, FunASR-VAD, and WebRTC-VAD. FireRedLID: An Encoder-Decoder LID module supporting 100+ languages and 20+ Chinese dialects and accents. On FLEURS (82 languages), it achieves 97.18% utterance-level accuracy, outperforming Whisper and SpeechBrain. FireRedPunc: A BERT-style punctuation prediction module for Chinese and English. On multi-domain benchmarks, it achieves 78.90% average F1, outperforming FunASR-Punc (62.77%). To advance research in speech processing, we release model weights and code at this https URL.
Abstract (translated)
我们介绍FireRedASR2S，这是一个最先进的工业级一体化自动语音识别（ASR）系统。它在一个统一的流水线中集成了四个模块：ASR、语音活动检测（VAD）、口语语言识别（LID）和标点预测（Punc）。所有模块在评估基准上均达到了最先进水平（SOTA）： - FireRedASR2: 包含两个变体，FireRedASR2-LLM（8B+参数）和支持语音和演唱转写的FireRedASR2-AED（1B+参数），支持普通话、中国方言及口音、英语以及代码切换。与前代相比，FireRedASR2在识别准确性和更广泛的方言及口音覆盖范围方面都有所改进。FireRedASR2-LLM在中国语料库的四个公共基准上平均CER为2.89%，在19个公共中国方言和口音基准上的平均CER为11.55%，优于包括Doubao-ASR、Qwen3-ASR和Fun-ASR在内的竞争基线。 - FireRedVAD：一个基于深度前馈序列记忆网络（DFSMN）的超轻量级模块，支持流式语音活动检测（streaming VAD）、非流式语音活动检测（non-streaming VAD）以及多标签语音活动检测（mVAD）。在FLEURS-VAD-102基准上，它实现了97.57%的帧级F1和99.60%AUC-ROC，优于Silero-VAD、TEN-VAD、FunASR-VAD及WebRTC-VAD。 - FireRedLID：一个支持超过100种语言和20多种中国方言及口音的编码器解码器模块。在FLEURS（82种语言）上，它实现了97.18%的句子级准确率，优于Whisper和SpeechBrain。 - FireRedPunc：针对中文和英文的BERT风格标点预测模块，在多领域基准上平均F1为78.90%，优于FunASR-Punc（62.77%）。 为了推进语音处理领域的研究，我们将在[此链接](https://this-url.com)发布模型权重及代码。
URL
https://arxiv.org/abs/2603.10420
PDF
-
PRoADS: Provably Secure and Robust Audio Diffusion Steganography with latent optimization and backward Euler Inversion
2026-03-11 01:29:30YongPeng Yan, Yanan Li, Qiyang Xiao, Yanzhen RenAbstract
This paper proposes PRoADS, a provably secure and robust audio steganographic framework based on audio diffusion models. As a generative steganography scheme, PRoADS embeds secret messages into the initial noise of diffusion models via orthogonal matrix projection. To address the reconstruction errors in diffusion inversion that cause high bit error rates (BER), we introduce Latent Optimization and Backward Euler Inversion to minimize the latent reconstruction and diffusion inversion errors. Comprehensive experiments demonstrate that our scheme sustains a remarkably low BER of 0.15\% under 64 kbps MP3 compression, significantly outperforming existing methods and exhibiting strong robustness.
Abstract (translated)
本文提出了PRoADS，这是一种基于音频扩散模型的可证明安全且鲁棒的音频隐写框架。作为一种生成式隐写方案，PRoADS通过正交矩阵投影将秘密消息嵌入到扩散模型的初始噪声中。为了应对在扩散逆向过程中导致高比特误码率（BER）的重建误差问题，我们引入了潜在优化和反向欧拉逆向技术来最小化潜伏态重建和扩散逆过程中的错误。全面的实验表明，在64kbps MP3压缩条件下，我们的方案能够保持极低的0.15%比特误码率，这显著优于现有的方法，并表现出强大的鲁棒性。
URL
https://arxiv.org/abs/2603.10314
PDF
-
nlm: Real-Time Non-linear Modal Synthesis in Max
Abstract
We present \texttt{nlm}, a set of Max externals that enable efficient real-time non-linear modal synthesis for strings, membranes, and plates. The externals, implemented in C++, offer interactive control of physical parameters, allow the loading of custom modal data, and provide multichannel output. By integrating interactive physical-modelling capabilities into a familiar environment, \texttt{nlm} lowers the barrier for composers, performers, and sound designers to explore the expressive potential of non-linear modal synthesis. The externals are available as open-source software at this https URL.
Abstract (translated)
我们介绍\texttt{nlm}，这是一个实现于C++的Max外部对象集，用于高效实时地进行弦、膜和板等物体的非线性模态合成。这些外部对象提供了物理参数的交互控制功能，支持加载自定义模态数据，并提供多声道输出。通过将互动物理建模能力集成到一个熟悉的环境中，\texttt{nlm} 降低了作曲家、表演者和声音设计师探索非线性模态合成功能表达潜力的门槛。这些外部对象作为开源软件在[此处](https://这个URL)提供。 （请注意，上述文本中的“this https URL”应该是一个实际链接地址，在引用时请替换为正确的网址。）
URL
https://arxiv.org/abs/2603.10240
PDF
-
A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AI
2026-03-10 21:17:17Karim Helwani, Hoang Do, James Luan, Sriram SrinivasanAbstract
We present a real-time front-end for voice-based conversational AI to enable natural turn-taking in two-speaker scenarios by combining primary speaker segmentation with hierarchical End-of-Turn (EOT) detection. To operate robustly in multi-speaker environments, the system continuously identifies and tracks the primary user, ensuring that downstream EOT decisions are not confounded by background conversations. The tracked activity segments are fed to a hierarchical, causal EOT model that predicts the immediate conversational state by independently analyzing per-speaker speech features from both the primary speaker and the bot. Simultaneously, the model anticipates near-future states ($t{+}10/20/30$\,ms) through probabilistic predictions that are aware of the conversation partner's speech. Task-specific knowledge distillation compresses wav2vec~2.0 representations (768\,D) into a compact MFCC-based student (32\,D) for efficient deployment. The system achieves 82\% multi-class frame-level F1 and 70.6\% F1 on Backchannel detection, with 69.3\% F1 on a binary Final vs.\ Others task. On an end-to-end turn-detection benchmark, our model reaches 87.7\% recall vs.\ 58.9\% for Smart Turn~v3 while keeping a median detection latency of 36\,ms versus 800--1300\,ms. Despite using only 1.14\,M parameters, the proposed model matches or exceeds transformer-based baselines while substantially reducing latency and memory footprint, making it suitable for edge deployment.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13379
PDF
-
EmoSURA: Towards Accurate Evaluation of Detailed and Long-Context Emotional Speech Captions
2026-03-10 15:44:48Xin Jing, Andreas Triantafyllopoulos, Jiadong Wang, Shahin Amiriparian, Jun Luo, Bj\"orn SchullerAbstract
Recent advancements in speech captioning models have enabled the generation of rich, fine-grained captions for emotional speech. However, the evaluation of such captions remains a critical bottleneck: traditional N-gram metrics fail to capture semantic nuances, while LLM judges often suffer from reasoning inconsistency and context-collapse when processing long-form descriptions. In this work, we propose EmoSURA, a novel evaluation framework that shifts the paradigm from holistic scoring to atomic verification. EmoSURA decomposes complex captions into Atomic Perceptual Units, which are self-contained statements regarding vocal or emotional attributes, and employs an audio-grounded verification mechanism to validate each unit against the raw speech signal. Furthermore, we address the scarcity of standardized evaluation resources by introducing SURABench, a carefully balanced and stratified benchmark. Our experiments show that EmoSURA achieves a positive correlation with human judgments, offering a more reliable assessment for long-form captions compared to traditional metrics, which demonstrated negative correlations due to their sensitivity to caption length.
Abstract (translated)
最近在语音字幕生成模型方面的进展已经使情感化语音的丰富且精细的字幕生成成为可能。然而，评估这类字幕仍然是一个关键瓶颈：传统的N-gram度量无法捕捉语义细微差别，而大型语言模型（LLM）评委在处理长篇描述时常常会遭遇推理不一致和上下文坍缩的问题。 为了解决这个问题，我们提出了EmoSURA这一新颖的评估框架。该框架从整体评分转向原子验证，将复杂的字幕分解成原子感知单元，这些单元是关于语音或情感属性的独立陈述，并采用基于音频的验证机制来验证每个单位与原始语音信号的一致性。 此外，为了应对标准化评估资源匮乏的问题，我们引入了SURABench，这是一个精心平衡和分层的标准。我们的实验表明，EmoSURA能够与人类判断产生正相关关系，在对长篇字幕进行评价时相比传统度量标准提供更为可靠的评估结果（传统度量标准由于其对字幕长度的敏感性而产生了负相关）。
URL
https://arxiv.org/abs/2603.09820
PDF
-
Paralinguistic Emotion-Aware Validation Timing Detection in Japanese Empathetic Spoken Dialogue
2026-03-10 07:41:02Zi Haur Pang, Yahui Fu, Yuan Gao, Tatsuya KawaharaAbstract
Emotional Validation is a psychotherapy communication technique that involves recognizing, understanding, and explicitly acknowledging another person's feelings and actions, which strengthens alliance and reduces negative affect. To maximize the emotional support provided by validation, it is crucial to deliver it with appropriate timing and frequency. This study investigates validation timing detection from the speech perspective. Leveraging both paralinguistic and emotional information, we propose a paralinguistic- and emotion-aware model for validation timing detection without relying on textual context. Specifically, we first conduct continued self-supervised training and fine-tuning on different HuBERT backbones to obtain (i) a paralinguistics-aware Self-Supervised Learning (SSL) encoder and (ii) a multi-task speech emotion classification encoder. We then fuse these encoders and further fine-tune the combined model on the downstream validation timing detection task. Experimental evaluations on the TUT Emotional Storytelling Corpus (TESC) compare multiple models, fusion mechanisms, and training strategies, and demonstrate that the proposed approach achieves significant improvements over conventional speech baselines. Our results indicate that non-linguistic speech cues, when integrated with affect-related representations, carry sufficient signal to decide when validation should be expressed, offering a speech-first pathway toward more empathetic human-robot interaction.
Abstract (translated)
情感确认是心理疗法中的一个沟通技巧，涉及识别、理解和明确承认他人的情感和行为，这有助于加强关系并减少负面情绪。为了最大限度地提高情感支持的效果，适时且频繁地提供情感确认至关重要。本研究从语音角度探讨了情感确认时机的检测问题。通过利用副语言和情感信息，我们提出了一种不需要文本上下文、结合副语言感知与情感意识的情感确认时机检测模型。 具体来说，首先在不同的HuBERT骨干网络上进行持续自监督训练和微调，以获得（i）一种具有副语言感知能力的自监督学习(SSL)编码器以及(ii)一个多任务语音情绪分类编码器。然后将这些编码器融合，并进一步对组合模型进行下游情感确认时机检测任务的微调。 在TUT情感叙事语料库(TESC)上进行了多项实验评估，比较了多种模型、融合机制和训练策略的效果，表明所提出的方法在传统语音基线模型上有显著改进。我们的结果表明，当非语言性语音线索与情绪相关的表示结合时，足以决定何时表达确认信息，为实现更富有同理心的人机互动提供了一种以语音为主导的路径。
URL
https://arxiv.org/abs/2603.09307
PDF
-
The Costs of Reproducibility in Music Separation Research: a Replication of Band-Split RNN
Abstract
Music source separation is the task of isolating the instrumental tracks from a music song. Despite its spectacular recent progress, the trend towards more complex architectures and training protocols exacerbates reproducibility issues. The band-split recurrent neural networks (BSRNN) model is promising in this regard, since it yields close to state-of-the-art results on public datasets, and requires reasonable resources for training. Unfortunately, it is not straightforward to reproduce since its full code is not available. In this paper, we attempt to replicate BSRNN as closely as possible to the original paper through extensive experiments, which allows us to conduct a critical reflection on this reproducibility issue. Our contributions are three-fold. First, this study yields several insights on the model design and training pipeline, which sheds light on potential future improvements. In particular, since we were unsuccessful in reproducing the original results, we explore additional variants that ultimately yield an optimized BSRNN model, whose performance largely improves that of the original. Second, we discuss reproducibility issues from both methodological and practical perspectives. We notably underline how substantial time and energy costs could have been saved upon availability of the full pipeline. Third, our code and pre-trained models are released publicly to foster reproducible research. We hope that this study will contribute to spread awareness on the importance of reproducible research in the music separation community, and help promoting more transparent and sustainable practices.
Abstract (translated)
音乐源分离的任务是从一首歌曲中单独提取出乐器轨道。尽管这一领域近期取得了显著进展，但复杂架构和训练协议的发展加剧了再现性问题。带分频循环神经网络（BSRNN）模型在这一方面很有前景，因为它能在公开数据集上取得接近当前最佳的成果，并且需要合理的资源进行训练。然而，由于其完整代码不可用，它并不容易被复制。在这篇文章中，我们通过大量实验试图尽可能忠实地重现原始论文中的BSRNN，这使我们能够对这一再现性问题进行批判性的反思。 我们的贡献有三个方面： 1. 本研究提供了关于模型设计和训练流程的多个见解，为潜在未来改进提供了思路。特别是由于我们在复制原始结果方面不成功，我们探索了额外变体，并最终得到了一个优化后的BSRNN模型，其性能远超原版。 2. 我们从方法论和实践两个角度讨论了再现性问题。特别地，我们指出了如果整个流程完整可得，将能大幅节省时间和精力成本。 3. 我们的代码及预训练模型已公开发布，以促进可重复研究。希望这项研究有助于在音乐分离社区传播关于可重复研究重要性的认识，并帮助推广更加透明和可持续的做法。 通过上述努力，我们希望能对推进音乐源分离领域的科学进步做出贡献，同时也强调了开放性和透明性对于科学研究的重要性。
URL
https://arxiv.org/abs/2603.09187
PDF
-
LL-SDR: Low-Latency Speech enhancement through Discrete Representations
2026-03-10 01:27:31Jingyi Li, Luca Della Libera, Mirco Ravanelli, Cem SubakanAbstract
Many speech enhancement (SE) methods rely on continuous representations. Recently, discrete audio tokens have been explored to enable autoregressive generation for SE. However, it remains unclear whether discretization itself consistently improves SE performance. In this paper, we introduce LL-SDR, a token-based speech enhancement framework that explicitly leverages discretization to better separate speech and noise. Our first contribution is a Variance-Ordered Residual Vector Quantizer (VO-RVQ), designed to disentangle speech and noise distributions during tokenization. Second, we propose a latent-space discriminator to better align enhanced embeddings with semantic embeddings. Experiments show that LL-SDR outperforms continuous baselines and matches the performance of autoregressive token-based approaches, while enabling lightweight, low-latency speech enhancement in both reverberant and non-reverberant noisy environments. Demos and source code are available at our project websites.
Abstract (translated)
许多语音增强（SE）方法依赖于连续表示。近期，离散音频令牌已被探索用于实现语音增强的自回归生成。然而，离散化本身是否能持续提升语音增强性能仍不明确。本文中，我们提出了LL-SDR，一个基于令牌的语音增强框架，它明确利用离散化来更好地区分语音与噪声。我们的首个贡献是方差有序残差向量量化器（VO-RVQ），旨在令牌化过程中解耦语音与噪声分布。其次，我们提出了潜在空间判别器，以更好对齐增强后的嵌入与语义嵌入。实验表明，LL-SDR优于连续基线方法，并与自回归令牌式方法性能相当，同时能在混响与非混响噪声环境中实现轻量、低延迟的语音增强。演示与源代码已发布至项目网站。
URL
https://arxiv.org/abs/2603.20242
PDF
-
Trade-offs Between Capacity and Robustness in Neural Audio Codecs for Adversarially Robust Speech Recognition
2026-03-10 00:10:23Jordan Prescott, Thanathai Lertpetchpun, Shrikanth NarayananAbstract
Adversarial perturbations exploit vulnerabilities in automatic speech recognition (ASR) systems while preserving human perceived linguistic content. Neural audio codecs impose a discrete bottleneck that can suppress fine-grained signal variations associated with adversarial noise. We examine how the granularity of this bottleneck, controlled by residual vector quantization (RVQ) depth, shapes adversarial robustness. We observe a non-monotonic trade-off under gradient-based attacks: shallow quantization suppresses adversarial perturbations but degrades speech content, while deeper quantization preserves both content and perturbations. Intermediate depths balance these effects and minimize transcription error. We further show that adversarially induced changes in discrete codebook tokens strongly correlate with transcription error. These gains persist under adaptive attacks, where neural codec configurations outperform traditional compression defenses.
Abstract (translated)
对抗性扰动在利用自动语音识别（ASR）系统的漏洞的同时，保持人类感知的语言内容不变。神经音频编解码器施加了一个离散瓶颈，可以抑制与对抗噪声相关的细微信号变化。我们研究了这个瓶颈的粒度如何通过残差向量量化（RVQ）深度来控制，并探讨这对对抗鲁棒性的影响。我们在基于梯度的攻击下观察到一种非单调权衡：浅层量化可以抑制对抗性扰动，但会降低语音内容的质量；而深层量化则既能保留内容又能保留扰动。中间深度则能平衡这些效果并最小化转录错误。此外，我们还发现由对抗攻击引起的离散代码本令牌的变化与转录误差有很强的相关性。这种优势在自适应攻击下依然存在，其中神经编解码器配置优于传统的压缩防御措施。
URL
https://arxiv.org/abs/2603.09034
PDF
-
Universal Speech Content Factorization
2026-03-09 22:11:40Henry Li Xinyuan, Zexin Cai, Lin Zhang, Leibny Paola Garc\'ia-Perera, Berrak Sisman, Sanjeev Khudanpur, Nicholas Andrews, Matthew WiesnerAbstract
We propose Universal Speech Content Factorization (USCF), a simple and invertible linear method for extracting a low-rank speech representation in which speaker timbre is suppressed while phonetic content is preserved. USCF extends Speech Content Factorization, a closed-set voice conversion (VC) method, to an open-set setting by learning a universal speech-to-content mapping via least-squares optimization and deriving speaker-specific transformations from only a few seconds of target speech. We show through embedding analysis that USCF effectively removes speaker-dependent variation. As a zero-shot VC system, USCF achieves competitive intelligibility, naturalness, and speaker similarity compared to methods that require substantially more target-speaker data or additional neural training. Finally, we demonstrate that as a training-efficient timbre-disentangled speech feature, USCF features can serve as the acoustic representation for training timbre-prompted text-to-speech models. Speech samples and code are publicly available.
Abstract (translated)
我们提出了通用语音内容因子化（USCF），这是一种简单且可逆的线性方法，用于提取一种低秩语音表示，在这种表示中，说话人的音色被抑制而语音内容得以保留。USCF通过最小二乘优化学习一个通用的语音到内容映射，并仅使用几秒钟的目标说话人音频来推导出特定说话者的转换，从而将封闭集内的声音转换方法——语音内容因子化扩展到了开放集中。我们通过嵌入分析展示了USCF能够有效地移除与说话人相关的变异性。作为零样本声音转换系统，USCF在可懂度、自然度和说话人相似性方面取得了优于需要大量目标说话人数据或额外神经网络训练的方法的竞争力结果。最后，我们演示了USCF作为一种高效的音色解耦语音特征，在训练由音色调用的文本到语音模型中可以作为声学表示使用。语音样本和代码已公开提供。
URL
https://arxiv.org/abs/2603.08977
PDF
-
Scalable Neural Vocoder from Range-Null Space Decomposition
2026-03-09 16:29:42Andong Li, Tong Lei, Zhihang Sun, Rilin Chen, Xiaodong Li, Dong Yu, Chengshi ZhengAbstract
Although deep neural networks have facilitated significant progress of neural vocoders in recent years, they usually suffer from intrinsic challenges like opaque modeling, inflexible retraining under different input configurations, and parameter-performance trade-off. These inherent hurdles can heavily impede the development of this field. To resolve these problems, in this paper, we propose a novel neural vocoder in the time-frequency (T-F) domain. Specifically, we bridge the connection between the classical range-null decomposition (RND) theory and the vocoder task, where the reconstruction of the target spectrogram is formulated into the superimposition between range-space and null-space. The former aims to project the representation in the original mel-domain into the target linear-scale domain, and the latter can be instantiated via neural networks to further infill the spectral details. To fully leverage the spectrum prior, an elaborate dual-path framework is devised, where the spectrum is hierarchically encoded and decoded, and the cross- and narrow-band modules are leveraged for effectively modeling along sub-band and time dimensions. To enable inference under various configurations, we propose a simple yet effective strategy, which transforms the multi-condition adaption in the inference stage into the data augmentation in the training stage. Comprehensive experiments are conducted on various benchmarks. Quantitative and qualitative results show that while enjoying lightweight network structure and scalable inference paradigm, the proposed framework achieves state-ofthe-art performance among existing advanced methods. Code is available at this https URL.
Abstract (translated)
尽管深度神经网络在过去几年中促进了神经声码器的显著进步，但它们通常会遇到诸如模型不透明、在不同输入配置下灵活重训练困难以及参数性能权衡等内在挑战。这些固有的障碍可能会严重阻碍这一领域的发展。为了解决这些问题，在本文中我们提出了一种基于时间-频率（T-F）域的新颖神经声码器。具体来说，我们将经典的范围空-零空间分解（RND）理论与声码器任务之间的连接建立了起来，并将目标频谱图的重建公式化为范围空间和零空间的叠加。前者旨在将原始梅尔域中的表示投影到目标线性尺度域中，而后者可以通过神经网络进一步填充频谱细节。为了充分利用频谱先验知识，我们设计了一个精致的双路径框架，在该框架下频谱被分层编码和解码，并利用跨频段和窄带模块沿子频段和时间维度进行有效建模。为实现在各种配置下的推断，我们提出了一种简单而有效的策略，将推理阶段中的多条件适应转换为训练阶段中的数据增强。我们在多个基准上进行了全面的实验。定量和定性结果表明，在享受轻量级网络结构和可扩展推断范式的同时，所提出的框架在现有的先进方法中实现了最佳性能。代码可在提供的链接获取。
URL
https://arxiv.org/abs/2603.08574
PDF
-
LoopLens: Supporting Search as Creation in Loop-Based Music Composition
Abstract
Creativity support tools (CSTs) typically frame search as information retrieval, yet in practices like electronic dance music production, search serves as a creative medium for collage-style composition. To address this gap, we present LoopLens, a research probe for loop-based music composition that visualizes audio search results to support creative foraging and assembling. We evaluated LoopLens in a within-subject user study with 16 participants of diverse musical domain expertise, performing both open-ended (divergent) and goal-directed (convergent) tasks. Our results reveal a clear behavioral split: participants with domain expertise leveraged multimodal cues to quickly exploit a narrow set of loops, while those without domain knowledge relied primarily on audio impressions, engaging in broad exploration often constrained by limited musical vocabulary for query formulation. This behavioral dichotomy provides a new lens for understanding the balance between exploration and exploitation in creative search and offers clear design implications for supporting vocabulary-independent discovery in future CSTs.
Abstract (translated)
创意支持工具（CST）通常将搜索视为信息检索，但在如电子舞曲制作等实践中，搜索被视为一种用于拼贴式创作的创造性媒介。为解决这一差距，我们介绍了LoopLens，这是一个基于循环的音乐作曲研究探针，它通过可视化音频搜索结果来支持创造性的搜集和组装工作。我们在一项包含16名具有不同音乐领域专业知识的参与者的研究中评估了LoopLens，这些参与者在开放性（发散）任务和目标导向（收敛）任务中表现出了不同的行为模式。 我们的结果显示了一个明显的两极分化：那些拥有特定领域知识的参与者利用多模态线索快速地从有限的一组循环中提取信息；而没有相关专业知识的参与者则主要依赖于音频印象，在探索时往往因缺乏音乐词汇来构建查询而受到限制。这种行为上的二元性为理解创造性搜索中的探索与开发之间的平衡提供了一个新的视角，并对未来的CST在支持词汇无关发现的设计方面提供了明确的方向。
URL
https://arxiv.org/abs/2603.08571
PDF
