arXiv_SD
-
Soundscapes in Spectrograms: Pioneering Multilabel Classification for South Asian Sounds
2026-03-09 09:36:12Sudip Chakrabarty, Pappu Bishwas, Rajdeep Chatterjee, Tathagata Bandyopadhyay, Digonto Biswas, Bibek HowladerAbstract
Environmental sound classification is a field of growing importance for urban monitoring and cultural soundscape analysis, especially within the acoustically rich environments of South Asia. These regions present a unique challenge as multiple natural, human, and cultural sounds often overlap, straining traditional methods that frequently rely on Mel Frequency Cepstral Coefficients (MFCC). This study introduces a novel spectrogram-based methodology with a superior ability to capture these complex auditory patterns. A Convolutional Neural Network (CNN) architecture is implemented to solve a demanding multilabel, multiclass classification problem on the SAS-KIIT dataset. To demonstrate robustness and comparability, the approach is also validated using the renowned UrbanSound8K dataset. The results confirm that the proposed spectrogram-based method significantly outperforms existing MFCC-based techniques, achieving higher classification accuracy across both datasets. This improvement lays the groundwork for more robust and accurate audio classification systems in real-world applications.
Abstract (translated)
环境声音分类在城市监测和文化声景分析领域中日益重要，尤其是在南亚这些声学丰富的环境中。这些地区面临的独特挑战是多种自然、人类和文化声音往往相互重叠，这对传统方法构成了考验，因为它们通常依赖于梅尔频率倒谱系数（MFCC）。本研究引入了一种基于频谱图的新颖方法，该方法具有捕捉复杂听觉模式的优越能力。通过在SAS-KIIT数据集上解决一个严格的多标签、多类分类问题，实施了卷积神经网络（CNN）架构。为了证明其稳健性和可比性，该方法还使用著名的UrbanSound8K数据集进行了验证。结果确认提出的基于频谱图的方法显著优于现有的MFCC基技术，在两个数据集中均实现了更高的分类精度。这一改进为实际应用中的更加强大和准确的声音分类系统奠定了基础。
URL
https://arxiv.org/abs/2603.08154
PDF
-
PathBench: Speech Intelligibility Benchmark for Automatic Pathological Speech Assessment
2026-03-09 08:37:28Bence Mark Halpern, Thomas Tienkamp, Defne Abur, Tomoki TodaAbstract
Automatic speech intelligibility assessment is crucial for monitoring speech disorders and therapy efficacy. However, existing methods are difficult to compare: research is fragmented across private datasets with inconsistent protocols. We introduce PathBench, a unified benchmark for pathological speech assessment using public datasets. We compare reference-free, reference-text, and reference-audio methods across three protocols (Matched Content, Extended, and Full) representing how a linguist (controlled stimuli) versus machine learning specialist (maximum data) would approach the same data. We establish benchmark baselines across six datasets, enabling systematic evaluation of future methodological advances, and introduce Dual-ASR Articulatory Precision (DArtP), achieving the highest average correlation among reference-free methods.
Abstract (translated)
自动语音清晰度评估对于监测言语障碍和治疗效果至关重要。然而，现有的方法难以进行比较：研究碎片化地分布在使用不一致协议的私人数据集上。我们引入了PathBench，这是一个统一基准，用于使用公共数据集进行病理语音评估。我们在三种不同的协议（匹配内容、扩展和完整）下对比了无参考、文本参考和音频参考的方法，这三种协议分别代表语言学家（控制刺激）与机器学习专家（最大数据量）在处理同一数据时的不同方法。我们建立了六个数据集上的基准线，从而能够对未来的研究进展进行系统的评估，并引入了一种新的指标——双ASR发音精确度（Dual-ASR Articulatory Precision, DArtP），该指标在无参考的方法中达到了最高的平均相关性。
URL
https://arxiv.org/abs/2603.08097
PDF
-
WhispEar: A Bi-directional Framework for Scaling Whispered Speech Conversion via Pseudo-Parallel Whisper Generation
2026-03-09 07:36:43Zihao Fang, Yingda Shen, Zifan Guan, Tongtong Song, Zhenyi Liu, Zhizheng WuAbstract
Whispered speech lacks vocal fold vibration and fundamental frequency, resulting in degraded acoustic cues and making whisper-to-normal (W2N) conversion challenging, especially with limited parallel data. We propose WhispEar, a bidirectional framework based on unified semantic representations that capture speaking-mode-invariant information shared by whispered and normal speech. The framework contains both W2N and normal-to-whisper (N2W) models. Notably, the N2W model enables zero-shot pseudo-parallel whisper generation from abundant normal speech, allowing scalable data augmentation for W2N training. Increasing generated data consistently improves performance. We also release the largest bilingual (Chinese-English) whispered-normal parallel corpus to date. Experiments demonstrate that WhispEar outperforms strong baselines and benefits significantly from scalable pseudo-parallel data.
Abstract (translated)
耳语缺乏声带振动和基频，导致声学线索降级，使得从耳语转换为正常语音（W2N）变得具有挑战性，尤其是在平行数据有限的情况下。我们提出了 WhispEar，这是一个基于统一语义表示的双向框架，该框架捕捉了耳语和正常语音之间共有的说话模式不变的信息。该框架包含 W2N 和正常到耳语（N2W）模型。特别地，N2W 模型能够从大量的正常语言中零样本生成伪平行耳语数据，从而为 W2N 训练提供可扩展的数据增强支持。随着生成数据的增加，性能会持续提升。此外，我们还发布了迄今为止最大的双语（中文-英语）耳语与正常语音平行语料库。实验表明 WhispEar 超过了强大的基线模型，并且从可扩展的伪平行数据中受益匪浅。
URL
https://arxiv.org/abs/2603.08046
PDF
-
Unsupervised Domain Adaptation for Audio Deepfake Detection with Modular Statistical Transformations
2026-03-09 04:00:38Urawee Thani, Gagandeep Singh, Priyanka SinghAbstract
Audio deepfake detection systems trained on one dataset often fail when deployed on data from different sources due to distributional shifts in recording conditions, synthesis methods, and acoustic environments. We present a modular pipeline for unsupervised domain adaptation that combines pre-trained Wav2Vec 2.0 embeddings with statistical transformations to improve cross-domain generalization without requiring labeled target data. Our approach applies power transformation for feature normalization, ANOVA-based feature selection, joint PCA for domain-agnostic dimensionality reduction, and CORAL alignment to match source and target covariance structures before classification via logistic regression. We evaluate on two cross-domain transfer scenarios: ASVspoof 2019 LA to Fake-or-Real (FoR) and FoR to ASVspoof, achieving 62.7--63.6\% accuracy with balanced performance across real and fake classes. Systematic ablation experiments reveal that feature selection (+3.5%) and CORAL alignment (+3.2%) provide the largest individual contributions, with the complete pipeline improving accuracy by 10.7% over baseline. While performance is modest compared to within-domain detection (94-96%), our pipeline offers transparency and modularity, making it suitable for deployment scenarios requiring interpretable decisions.
Abstract (translated)
音频深度伪造检测系统在不同数据集上的表现通常会下降，这是由于录音条件、合成方法和声学环境的分布变化造成的。我们提出了一种模块化的无监督领域适应管道，该管道结合了预训练的Wav2Vec 2.0嵌入与统计变换，以改善跨域泛化能力，并且不需要标记的目标数据。我们的方法包括用于特征归一化的幂转换、基于ANOVA的特征选择、用于领域无关维度减少的联合主成分分析（PCA）以及CORAL对齐来匹配源和目标协方差结构，在这些步骤之后通过逻辑回归进行分类。 我们在两个跨域传输场景中进行了评估：ASVspoof 2019 LA到Fake-or-Real (FoR) 和 FoR 到 ASVspoof，实现了62.7%至63.6%的准确率，并且真实和伪造类别的性能平衡。系统消融实验显示，特征选择（+3.5%）和CORAL对齐（+3.2%）提供了最大的单个贡献，而完整的管道相比于基准模型提升了10.7%的准确性。 尽管与领域内检测相比表现较弱（94-96%），我们的方法在可解释性和模块化方面具有优势，因此非常适合需要可解释决策的应用场景。
URL
https://arxiv.org/abs/2603.07935
PDF
-
Analysis-Driven Procedural Generation of an Engine Sound Dataset with Embedded Control Annotations
Abstract
Computational engine sound modeling is central to the automotive audio industry, particularly for active sound design, virtual prototyping, and emerging data-driven engine sound synthesis methods. These applications require large volumes of standardized, clean audio recordings with precisely time-aligned operating-state annotations: data that is difficult to obtain due to high costs, specialized measurement equipment requirements, and inevitable noise contamination. We present an analysis-driven framework for generating engine audio with sample-accurate control annotations. The method extracts harmonic structures from real recordings through pitch-adaptive spectral analysis, which then drive an extended parametric harmonic-plus-noise synthesizer. With this framework, we generate the Procedural Engine Sounds Dataset (19 hours, 5,935 files), a set of engine audio signals with sample-accurate RPM and torque annotations, spanning a wide range of operating conditions, signal complexities, and harmonic profiles. Comparison against real recordings validates that the synthesized data preserves characteristic harmonic structures, and baseline experiments confirm its suitability for learning-based parameter estimation and synthesis tasks. The dataset is released publicly to support research on engine timbre analysis, control parameter estimation, acoustic modeling and neural generative networks.
Abstract (translated)
计算引擎声音建模在汽车音频行业中至关重要，特别是在主动声设计、虚拟原型制作和新兴的数据驱动式发动机声音合成方法方面。这些应用需要大量的标准化且干净的音频记录，并带有精确的时间对齐的操作状态注释：由于成本高昂、测量设备专门化要求以及不可避免的噪声污染等原因，获取此类数据非常困难。 我们提出了一种基于分析的方法框架，用于生成具有样本级控制注解的引擎音频。该方法通过音高适应性频谱分析从真实记录中提取谐波结构，然后驱动扩展参数化的谐波加噪音合成器。使用此框架，我们生成了程序化发动机声音数据集（19小时，5,935个文件），包含带有样本准确转速和扭矩注释的引擎音频信号，涵盖了广泛的运行条件、信号复杂性和谐波配置。 与真实记录进行对比验证表明，所合成的数据保留了特征性的谐波结构。基准实验确认其适用于基于学习的参数估计和合成任务。该数据集公开发布以支持关于发动机音色分析、控制参数估计、声学建模及神经生成网络的研究工作。
URL
https://arxiv.org/abs/2603.07584
PDF
-
Evaluating Parkinson's Disease Detection in Anonymized Speech: A Performance and Acoustic Analysis
2026-03-08 09:07:26Carlos Franzreb, Francisco Teixeira, Ben Luks, Sebastian M\"oller, Alberto AbadAbstract
Automatic detection of Parkinson's disease (PD) from speech is a promising non-invasive diagnostic tool, but it raises significant privacy concerns. Speaker anonymization mitigates these risks, but it may suppress the pathological information necessary for PD detection. We assess the trade-off between privacy and PD detection for two anonymizers (STT-TTS and kNN-VC) using two Spanish datasets. STT-TTS provides better privacy but severely degrades PD detection by eradicating prosodic information. kNN-VC preserves macro-prosodic features such as duration and F0 contours, achieving F1 scores only 3-7\% lower than original baselines, demonstrating that privacy-preserving PD detection is viable when using appropriate anonymization. Finally, an acoustic distortion analysis characterizes specific weaknesses in kNN-VC, offering insights for designing anonymizers that better preserve PD information.
Abstract (translated)
从语音中自动检测帕金森病（PD）是一种有前景的非侵入性诊断工具，但这种方法会引发重大的隐私问题。说话人匿名化技术可以减轻这些风险，但它可能会抑制用于PD检测所需的关键病理信息。我们使用两个西班牙语数据集评估了两种匿名化方法（STT-TTS和kNN-VC）在保护隐私与PD检测准确性之间的权衡。 STT-TTS提供了更好的隐私保护，但这种方法会严重降低PD的检测能力，因为它消除了语音中的韵律特征。另一方面，kNN-VC能够保持音长、基频轮廓等宏观韵律特性，在两个数据集中实现的F1得分仅比原始基准低3%-7%，这表明在使用适当的匿名化技术时，可以实现既保护隐私又具有较高检测准确性的PD诊断。 最后，声学失真分析揭示了kNN-VC的一些特定弱点，并为设计能够在保持PD相关信息的同时更好地保护隐私的技术提供了见解。
URL
https://arxiv.org/abs/2603.07544
PDF
-
Fast and Flexible Audio Bandwidth Extension via Vocos
Abstract
We propose a Vocos-based bandwidth extension model that enhances audio at 8-48 kHz by generating missing high-frequency content. Inputs are resampled to 48 kHz and processed by a neural vocoder backbone, enabling a single network to support arbitrary upsampling ratios. A lightweight Linkwitz-Riley-inspired refiner merges the original low band with the generated high frequencies via a smooth crossover. On validation, the model achieves competitive log-spectral distance while running at a real-time factor of 0.0001 on an NVIDIA A100 GPU and 0.0053 on an 8-core CPU, demonstrating practical, high-quality BWE at extreme throughput.
Abstract (translated)
我们提出了一种基于Vocos的带宽扩展模型，该模型通过生成缺失的高频内容来增强8-48 kHz范围内的音频。输入信号被重采样为48 kHz，并由神经声码器骨干网络进行处理，使得单一网络能够支持任意上采样的比率。轻量级的Linkwitz-Riley启发式精炼器通过平滑过渡将原始低频段与生成的高频部分合并起来。在验证过程中，该模型达到了具有竞争力的对数谱距，并且在NVIDIA A100 GPU上的实时因子为0.0001，在8核CPU上为0.0053，展示了其在极高吞吐量下的实用性和高质量带宽扩展能力。
URL
https://arxiv.org/abs/2603.07285
PDF
-
Seeing the Context: Rich Visual Context-Aware Speech Recognition via Multimodal Reasoning
2026-03-07 15:45:54Wenjie Tian, Mingchen Shao, Bingshen Mu, Xuelong Geng, Chengyou Wang, Yujie Liao, Zhixian Zhao, Ziyu Zhang, Jingbin Hu, Mengqi Wei, Lei XieAbstract
Audio-visual speech recognition (AVSR) is an extension of ASR that incorporates visual signals. Current AVSR approaches primarily focus on lip motion, largely overlooking rich context present in the video such as speaking scene and on-screen text. To tackle such CAVSR (AVSR including rich visual Context), we propose VASR designed to "see" and reason the visual context to improve speech recognition. Specifically, we construct an Audio-Visual Chain-of-Thought (AV-CoT) that explicitly enforces intermediate cross-modal grounding between acoustic signals and visual evidence. This evidence-driven reasoning mitigates the "single-modality dominance" problem, where models either over-rely on visual context or fail to utilize it. Besides, to address the data scarcity, we construct and release a corresponding data pipeline and test set. Experiments show that AV-CoT effectively mitigates the single-modality dominance, achieving state-of-the-art performance in CAVSR. The project is open-sourced.
Abstract (translated)
音频-视觉语音识别（AVSR）是自动语音识别（ASR）的扩展，它结合了视觉信号。目前的AVSR方法主要关注唇部运动，而忽略了视频中存在的丰富背景信息，如说话场景和屏幕上的文本。为了应对包含丰富视觉上下文（CAVSR）的音频-视觉语音识别挑战，我们提出了一种名为VASR的方法，该方法旨在“看到”并推理视觉上下文以提高语音识别性能。具体来说，我们构建了一个音频-视觉链式思考（AV-CoT），它明确地强制执行声学信号与视觉证据之间的中间跨模态基础工作。这种基于证据的推理减轻了“单一模态主导”的问题，即模型要么过度依赖视觉上下文，要么未能有效利用它。 此外，为了解决数据稀缺的问题，我们构建并发布了相应的数据管道和测试集。实验表明，AV-CoT有效地缓解了单一模态主导的问题，在CAVSR方面取得了最先进的性能表现。该项目已经开源。
URL
https://arxiv.org/abs/2603.07263
PDF
-
Towards Objective Gastrointestinal Auscultation: Automated Segmentation and Annotation of Bowel Sound Patterns
2026-03-07 13:45:47Zahra Mansour, Verena Uslar, Dirk Weyhe, Danilo Hollosi, Nils StrodthoffAbstract
Bowel sounds (BS) are typically momentary and have low amplitude, making them difficult to detect accurately through manual auscultation. This leads to significant variability in clinical assessment. Digital acoustic sensors allow the acquisition of high-quality BS and enable automated signal analysis, offering the potential to provide clinicians with both objective and quantitative feedback on bowel activity. This study presents an automated pipeline for bowel sound segmentation and classification using a wearable acoustic SonicGuard sensor. BS signals from 83 subjects were recorded using a SonicGuard sensor. Data from 40 subjects were manually annotated by clinical experts and used to train an automatic annotation algorithm, while the remaining subjects were used for further model evaluation. An energy-based event detection algorithm was developed to detect BS events. Detected sound segments were then classified into BS patterns using a pretrained Audio Spectrogram Transformer (AST) model. Model performance was evaluated separately for healthy individuals and patients. The best configuration used two specialized models, one trained on healthy subjects and one on patients, achieving (accuracy: 0.97, AUROC: 0.98) for healthy group and (accuracy: 0.96, AUROC: 0.98) for patient group. The auto-annotation method reduced manual labeling time by approximately 70%, and expert review showed that less than 12% of automatically detected segments required correction. The proposed automated segmentation and classification system enables quantitative assessment of bowel activity, providing clinicians with an objective diagnostic tool that may improve the diagnostic of gastrointestinal function and support the annotation of large-scale datasets.
Abstract (translated)
肠鸣音（BS）通常是短暂的，振幅较低，通过手动听诊难以准确检测，这会导致临床评估中的显著变异性。数字声学传感器能够获取高质量的肠鸣音信号，并允许进行自动信号分析，为临床医生提供有关肠道活动的客观和量化反馈的可能性。这项研究提出了一种使用可穿戴声学SonicGuard传感器进行肠鸣音分割和分类的自动化流水线方法。 研究人员记录了83名受试者的BS信号，其中40名受试者的数据由临床专家手动标注并用于训练自动注释算法，其余受试者则用于进一步评估模型性能。开发了一种基于能量的事件检测算法来识别肠鸣音事件。检测到的声音片段随后使用预训练的音频光谱转换器（AST）模型进行分类。 对于健康个体和患者群体分别评估了该模型的性能。最佳配置采用两种专门针对不同人群训练的模型：一种在健康受试者上训练，另一种在患者群体上训练，在健康组中实现了0.97的准确率和0.98的AUROC值；在患者组中，则为0.96的准确率和0.98的AUROC值。 自动注释方法将手动标注时间减少了大约70%，并且专家审查发现，仅约12%的自动检测片段需要纠正。提出的自动化分割和分类系统能够对肠道活动进行量化评估，向临床医生提供客观诊断工具，有助于改善胃肠道功能的诊断，并支持大规模数据集的注释工作。
URL
https://arxiv.org/abs/2603.07215
PDF
-
Toward Multimodal Industrial Fault Analysis: A Single-Speed Chain Conveyor Dataset with Audio and Vibration Signals
2026-03-07 09:38:52Zhang Chen, Yucong Zhang, Xiaoxiao Miao, Ming LiAbstract
We introduce a multimodal industrial fault analysis dataset collected from a single-speed chain conveyor (SSCC) system, targeting system-level fault detection in production lines. The dataset consists of multimodal signals, including three audio and four vibration channels. It covers normal operation and four representative fault types under multiple speeds, loads, and both clean and realistic factory-noise conditions reproduced on-site. It is explicitly designed to support channel-wise analysis and multimodal fusion research. We establish standardized evaluation protocols for unsupervised fault detection with normal-only training and supervised fault classification with balanced dataset splits across different operating conditions and fault types. A unified channel-wise kNN baseline is provided to enable fair comparison of representation quality without task-specific training. The dataset offers a practical and extensible benchmark for robust multimodal industrial fault analysis.
Abstract (translated)
我们介绍了一个从单速链式输送机（SSCC）系统收集的多模态工业故障分析数据集，旨在用于生产线中的系统级故障检测。该数据集包含多模态信号，包括三个音频通道和四个振动通道。它涵盖了正常操作以及在多种速度、负载条件下，并且在干净环境及真实的工厂噪音环境下再现的四种典型的故障类型。此数据集明确设计用于支持逐通道分析及多模态融合研究。 我们为无监督故障检测（仅使用正常情况作为训练数据）和有监督故障分类（跨不同操作条件和故障类型的平衡数据集分割），建立了标准化的评估协议。此外，还提供了一个统一的逐通道kNN基线方法，以便在不进行特定任务训练的情况下公平比较表示质量。 该数据集为鲁棒多模态工业故障分析提供了实用且可扩展的基准测试。
URL
https://arxiv.org/abs/2603.07130
PDF
-
Adaptive Discovery of Interpretable Audio Attributes with Multimodal LLMs for Low-Resource Classification
2026-03-07 02:27:21Kosuke Yoshimura, Hisashi KashimaAbstract
In predictive modeling for low-resource audio classification, extracting high-accuracy and interpretable attributes is critical. Particularly in high-reliability applications, interpretable audio attributes are indispensable. While human-driven attribute discovery is effective, its low throughput becomes a bottleneck. We propose a method for adaptively discovering interpretable audio attributes using Multimodal Large Language Models (MLLMs). By replacing humans in the AdaFlock framework with MLLMs, our method achieves significantly faster attribute discovery. Our method dynamically identifies salient acoustic characteristics via prompting and constructs an attribute-based ensemble classifier. Experimental results across various audio tasks demonstrate that our method outperforms direct MLLM prediction in the majority of evaluated cases. The entire training completes within 11 minutes, proving it a practical, adaptive solution that surpasses conventional human-reliant approaches.
Abstract (translated)
在低资源音频分类的预测建模中，提取高准确性和可解释性的属性至关重要。特别是在需要高度可靠的应用场景下，可解释性音频属性不可或缺。虽然人工驱动的属性发现方法非常有效，但其低吞吐量成为了瓶颈问题。我们提出了一种使用多模态大型语言模型（MLLM）自适应地发现可解释音频属性的方法。通过在AdaFlock框架中用MLLM替代人类，我们的方法实现了显著更快的属性发现速度。我们的方法能够动态识别关键声学特征并通过提示构建基于属性的集成分类器。实验结果表明，在各种音频任务上，我们的方法在大多数评估案例中的表现都优于直接使用MLLM进行预测的方法。整个训练过程仅需11分钟即可完成，证明了这种方法是一种实用且自适应的解决方案，超越了传统依赖人工的方法。
URL
https://arxiv.org/abs/2603.06991
PDF
-
How Well Do Current Speech Deepfake Detection Methods Generalize to the Real World?
2026-03-06 03:18:16Daixian Li, Jun Xue, Yanzhen Ren, Zhuolin Yi, Yihuan Huang, Guanxiang Feng, Yi ChaiAbstract
Recent advances in speech synthesis and voice conversion have greatly improved the naturalness and authenticity of generated audio. Meanwhile, evolving encoding, compression, and transmission mechanisms on social media platforms further obscure deepfake artifacts. These factors complicate reliable detection in real-world environments, underscoring the need for representative evaluation benchmarks. To this end, we introduce ML-ITW (Multilingual In-The-Wild), a multilingual dataset covering 14 languages, seven major platforms, and 180 public figures, totaling 28.39 hours of audio. We evaluate three detection paradigms: end-to-end neural models, self-supervised feature-based (SSL) methods, and audio large language models (Audio LLMs). Experimental results reveal significant performance degradation across diverse languages and real-world acoustic conditions, highlighting the limited generalization ability of existing detectors in practical scenarios. The ML-ITW dataset is publicly available.
Abstract (translated)
近期在语音合成和声音转换领域的进展大大提升了生成音频的自然度和真实性。与此同时，社交媒体平台上的编码、压缩和传输机制也在不断发展，这进一步模糊了深度伪造（deepfake）的痕迹。这些因素使得在现实环境中可靠地检测到深度伪造变得更加复杂，强调了需要具有代表性的评估基准的重要性。 为此，我们引入了ML-ITW（Multilingual In-The-Wild），这是一个涵盖14种语言、七个主要平台和180位公众人物的多语种数据集，总共有28.39小时的音频。我们在三个检测范式上进行了评估：端到端神经模型、自监督特征方法（SSL）以及音频大型语言模型（Audio LLMs）。实验结果表明，在多种语言及现实世界声学条件下，现有检测器的表现显著下降，这突显了它们在实际场景中的泛化能力有限。ML-ITW数据集现已公开提供。
URL
https://arxiv.org/abs/2603.05852
PDF
-
Which Data Matter? Embedding-Based Data Selection for Speech Recognition
2026-03-06 02:07:08Zakaria Aldeneh, Skyler Seto, Maureen de Seyssel, Jie Chi, Zijin Gu, Takuya Higuchi, Jee-weon Jung, Shinji Watanabe, David Grangier, Barry-John Theobald, Tatiana LikhomanenkoAbstract
Modern ASR systems are typically trained on large-scale pseudo-labeled, in-the-wild data spanning multiple domains. While such heterogeneous data benefit generalist models designed for broad deployment, they pose challenges for specialist models targeting specific domains: specialist models lack the capacity to learn from all available data, and one must pay closer attention to addressing the mismatch between training and test conditions. In this work, we study targeted data selection as a strategy to address these challenges, selecting relevant subsets from 100k hours of in-the-wild training data to optimize performance on target domains. We represent speech samples using embeddings that capture complementary characteristic--speaker attributes, phonetic content, and semantic meaning--and analyze how relevance and diversity along these axes when performing data selection affect downstream ASR performance. Our experiments with CTC-based Conformer models show that training on a strategically selected 5% subset can exceed the performance of models trained on the full dataset by up to 36.8% relative WER reduction on target domains.
Abstract (translated)
现代的自动语音识别（ASR）系统通常是在大规模伪标注的真实场景数据上进行训练，这些数据涵盖了多个领域。虽然这种异质性数据对旨在广泛部署的一般模型有利，但对于针对特定领域的专业模型来说却带来了挑战：专业模型缺乏从所有可用数据中学习的能力，并且必须更加注意解决训练和测试条件之间的不匹配问题。在这项工作中，我们研究了有目标的数据选择策略来应对这些挑战，即从10万小时的真实场景训练数据中挑选出相关的子集以优化特定领域的性能。我们将语音样本表示为捕获互补特征的嵌入向量——包括说话人属性、音素内容和语义意义，并分析在进行数据选择时沿这些轴的相关性和多样性如何影响下游ASR性能。我们的实验表明，基于CTC的Conformer模型在经过精心挑选的5%子集上训练时，在目标领域中的相对错误词率（WER）可以比在完整数据集上训练的模型降低高达36.8%，这证明了有目标的数据选择策略的有效性。
URL
https://arxiv.org/abs/2603.05819
PDF
-
Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial
2026-03-05 17:35:59Jielin Qiu, Zixiang Chen, Liangwei Yang, Ming Zhu, Zhiwei Liu, Juntao Tan, Wenting Zhao, Rithesh Murthy, Roshan Ram, Akshara Prabhakar, Shelby Heinecke, Caiming Xiong, Silvio Savarese, Huan WangAbstract
We present a technical tutorial for building enterprise-grade realtime voice agents from first principles. While over 25 open-source speech-to-speech models and numerous voice agent frameworks exist, no single resource explains the complete pipeline from individual components to a working streaming voice agent with function calling capabilities. Through systematic investigation, we find that (1) native speech-to-speech models like Qwen2.5-Omni, while capable of high-quality audio generation, are too slow for realtime interaction ($\sim$13s time-to-first-audio); (2) the industry-standard approach uses a cascaded streaming pipeline: STT $\rightarrow$ LLM $\rightarrow$ TTS, where each component streams its output to the next; and (3) the key to ``realtime'' is not any single fast model but rather \textit{streaming and pipelining} across components. We build a complete voice agent using Deepgram (streaming STT), vLLM-served LLMs with function calling (streaming text generation), and ElevenLabs (streaming TTS), achieving a measured P50 time-to-first-audio of 947ms (best case 729ms) with cloud LLM APIs, and comparable latency with self-hosted vLLM on NVIDIA A10G GPU. We release the full codebase as a tutorial with working, tested code for every component.
Abstract (translated)
我们提供了一个从基础原理构建企业级实时语音代理的技术教程。尽管存在超过25个开源的语音到语音模型和许多语音代理框架，但没有单一资源能够解释从组件到具有功能调用能力的工作流式语音代理的完整流程。通过系统性研究，我们发现： 1. 本机语音转语音模型（如Qwen2.5-Omni）虽然能生成高质量音频，但由于处理速度过慢而不适合实时交互（约需13秒才能产生首个音频）； 2. 行业标准的方法采用级联流式管道：STT → LLM → TTS，其中每个组件将其输出流传输给下一个组件；以及 3. 实现“实时”的关键不在于单一的快速模型，而在于各组件之间的\textit{流处理和流水线}技术。 我们使用Deepgram（流式语音转文本），由vLLM服务的语言模型（具有功能调用的流式文本生成）和ElevenLabs（流式文本到语音转换），构建了一个完整的语音代理。在使用云端语言模型API时，我们的方案实现了中位数为947毫秒、最佳情况为729毫秒的时间至首次音频输出，并且与自托管vLLM在NVIDIA A10G GPU上的延迟相近。 我们将整个代码库作为教程发布，包括每个组件经过测试的工作代码。
URL
https://arxiv.org/abs/2603.05413
PDF
-
Koopman Regularized Deep Speech Disentanglement for Speaker Verification
2026-03-05 17:30:18Nikos Chazaridis, Mohammad Belal, Rafael Mestre, Timothy J. Norman, Christine EversAbstract
Human speech contains both linguistic content and speaker dependent characteristics making speaker verification a key technology in identity critical applications. Modern deep learning speaker verification systems aim to learn speaker representations that are invariant to semantic content and nuisance factors such as ambient noise. However, many existing approaches depend on labelled data, textual supervision or large pretrained models as feature extractors, limiting scalability and practical deployment, raising sustainability concerns. We propose Deep Koopman Speech Disentanglement Autoencoder (DKSD-AE), a structured autoencoder that combines a novel multi-step Koopman operator learning module with instance normalization to disentangle speaker and content dynamics. Quantitative experiments across multiple datasets demonstrate that DKSD-AE achieves improved or competitive speaker verification performance compared to state-of-the-art baselines while maintaining high content EER, confirming effective disentanglement. These results are obtained with substantially fewer parameters and without textual supervision. Moreover, performance remains stable under increased evaluation scale, highlighting representation robustness and generalization. Our findings suggest that Koopman-based temporal modelling, when combined with instance normalization, provides an efficient and principled solution for speaker-focused representation learning.
Abstract (translated)
人类的言语包含了语言内容和说话人特有的特征，因此语音验证在身份认证的关键应用中扮演着重要角色。现代深度学习语音识别系统的目标是学习出不受语义内容及环境噪声等干扰因素影响的说话人表示方法。然而，许多现有的方法依赖于标注数据、文本监督或大规模预训练模型作为特征提取器，这限制了其可扩展性和实际部署，并引发了可持续性的问题。 我们提出了一种基于深度Koopman运算符学习模块和实例归一化相结合的结构化自动编码器——Deep Koopman Speech Disentanglement Autoencoder (DKSD-AE)。这种设计旨在分离说话人特性和内容动态变化，从而实现有效的解耦。跨多个数据集进行的数量化实验表明，相较于最先进的基准模型，DKSD-AE在保持高语言错误接受率（Content EER）的同时，在语音验证性能方面取得了显著或相当的改进，这证明了其高效地实现了特征分离。值得注意的是，这些成果是在使用远少于现有方法参数量且无需文本监督的情况下实现的。 此外，随着评估规模的增加，DKSD-AE的表现依然保持稳定，彰显出其在表示能力和泛化能力上的优越性。我们的研究结果表明，在实例归一化的配合下，基于Koopman的时间建模为以说话人特征学习为中心的目标提供了一种高效且原理明确的方法。
URL
https://arxiv.org/abs/2603.05577
PDF
-
Hierarchical Decoding for Discrete Speech Synthesis with Multi-Resolution Spoof Detection
2026-03-05 16:59:26Junchuan Zhao, Minh Duc Vu, Ye WangAbstract
Neural codec language models enable high-quality discrete speech synthesis, yet their inference remains vulnerable to token-level artifacts and distributional drift that degrade perceptual realism. Rather than relying on preference optimization or retraining, we propose MSpoof-TTS, a training-free inference framework that improves zero-shot synthesis through multi-resolution spoof guidance. We introduce a Multi-Resolution Token-based Spoof Detection framework that evaluates codec sequences at different temporal granularities to detect locally inconsistent or unnatural patterns. We then integrate the spoof detectors into a hierarchical decoding strategy, progressively pruning low-quality candidates and re-ranking hypotheses. This discriminator-guided generation enhances robustness without modifying model parameters. Experiments validate the effectiveness of our framework for robust and high-quality codec-based speech generation.
Abstract (translated)
神经编码语言模型能够生成高质量的离散语音合成，但其推理过程仍然容易受到破坏感知真实性的令牌级伪迹和分布漂移的影响。不同于依赖偏好优化或重新训练的方法，我们提出了MSpoof-TTS，这是一种无训练推断框架，通过多分辨率伪造引导来改进零样本合成效果。 我们引入了一种基于多分辨率令牌的伪造检测框架，该框架能够在不同的时间粒度上评估编码序列以识别局部不一致或不自然的模式。然后，我们将这些伪造检测器整合到一个分层解码策略中，逐步剔除低质量候选者并重新排列假设。这种由鉴别器引导的生成方法可以在无需修改模型参数的情况下增强鲁棒性。 实验验证了我们框架在基于编码的语音生成中的有效性和其提升合成质量和鲁棒性的能力。
URL
https://arxiv.org/abs/2603.05373
PDF
-
SLICE: Speech Enhancement via Layer-wise Injection of Conditioning Embeddings
2026-03-05 15:42:31Seokhoon Moon, Kyudan Jung, Jaegul ChooAbstract
Real-world speech is often corrupted by multiple degradations simultaneously, including additive noise, reverberation, and nonlinear distortion. Diffusion-based enhancement methods perform well on single degradations but struggle with compound corruptions. Prior noise-aware approaches inject conditioning at the input layer only, which can degrade performance below that of an unconditioned model. To address this, we propose injecting degradation conditioning, derived from a pretrained encoder with multi-task heads for noise type, reverberation, and distortion, into the timestep embedding so that it propagates through all residual blocks without architectural changes. In controlled experiments where only the injection method varies, input-level conditioning performs worse than no encoder at all on compound degradations, while layer-wise injection achieves the best results. The method also generalizes to diverse real-world recordings.
Abstract (translated)
现实中的语音经常同时受到多种退化的干扰，包括加性噪声、混响和非线性失真。基于扩散的增强方法在处理单一退化时效果良好，但在面对复合退化时则表现不佳。先前的噪音感知方法仅在输入层注入条件信息，这可能导致性能低于未经条件训练的模型。为了解决这个问题，我们提出了一种方法，即从一个预训练的、具有多任务头部（分别用于噪声类型、混响和失真）的编码器中提取退化条件，并将其注入时间步嵌入中，从而使这种条件信息能够在不改变架构的情况下传播到所有残差块。在只有注入方法不同的受控实验中，输入级别的条件处理对于复合降质的表现不如完全没有编码器的情况，而逐层注入则取得了最佳结果。该方法还能推广应用于多样化的现实世界录音。
URL
https://arxiv.org/abs/2603.05302
PDF
-
PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio
2026-03-05 12:54:52Yuanjian Chen, Yang Xiao, Han Yin, Xubo Liu, Jinjie Huang, Ting DangAbstract
Large Audio Language Models (LALMs) are increasingly capable of reasoning over audio. However, existing benchmarks provide limited coverage of reasoning in polyphonic audio, where multiple sound events co-occur and induce compositional structure. In this work, we introduce PolyBench, a benchmark designed to evaluate compositional reasoning in polyphonic audio. PolyBench comprises five evaluation subsets covering counting, classification, detection, concurrency, and duration estimation, requiring reasoning over multiple concurrent events and their relations. Evaluation of state-of-the-art LALMs reveals consistent performance degradation in polyphonic audio, indicating a fundamental bottleneck in current LALMs.
Abstract (translated)
大型音频语言模型（LALM）在处理音频推理方面的能力越来越强。然而，现有的评估基准对于多声部音频中的推理覆盖不足，其中多个声音事件同时发生并产生组合结构。在这项工作中，我们引入了PolyBench这一评估基准，旨在评测多声部音频中组合推理的性能。PolyBench 包含五个评估子集，涵盖了计数、分类、检测、并发性和持续时间估计等方面，要求模型能够处理多个同时发生的事件及其之间的关系。 对最新技术的LALM进行评估后发现，在多声部音频上其表现有所下降，这表明当前LALM在这一领域存在基本的技术瓶颈。
URL
https://arxiv.org/abs/2603.05128
PDF
-
TW-Sound580K: A Regional Audio-Text Dataset with Verification-Guided Curation for Localized Audio-Language Modeling
2026-03-05 12:07:19Hao-Hui Xie, Ho-Lam Chung, Yi-Cheng Lin, Ke-Han Lu, Wenze Ren, Xie Chen, Hung-yi LeeAbstract
Large Audio-Language Models (LALMs) typically struggle with localized dialectal prosody due to the scarcity of specialized corpora. We present TW-Sound580K, a Taiwanese audio-text instruction dataset developed through a Verify-Generate-Critique (VGC) protocol. This pipeline leverages Dual-ASR validation to filter 522K raw clips, subsequently expanding them into 580,000 high-fidelity instruction pairs using a teacher model. The dataset's utility is demonstrated through Tai-LALM, which fine-tunes a DeSTA 2.5-Audio-initialized backbone and incorporates a dynamic Dual-ASR Arbitration strategy to optimize transcription selection during inference. On the TAU Benchmark, Tai-LALM reaches 49.1% accuracy, marking a 6.5% absolute improvement over the zero-shot baseline (42.6% with ASR text conditioning). This confirms that integrating regional corpora with rigorous curation and dynamic arbitration significantly enhances LALM performance on localized speech.
Abstract (translated)
大型音频-语言模型（LALMs）通常在处理地方方言韵律方面遇到困难，这是因为缺乏专门的语料库。我们提出了TW-Sound580K，这是一个通过验证-生成-批评(VGC)协议开发的台湾音频-文本指令数据集。这一流程利用双ASR验证来筛选出52.2万个原始片段，并随后使用教师模型将其扩展为58万个高质量的指令对。 该数据集的应用价值通过Tai-LALM得到了展示，即通过对DeSTA 2.5-Audio初始化的骨干网络进行微调并采用动态双ASR仲裁策略，在推理过程中优化转录选择。在TAU基准测试中，Tai-LALM达到了49.1%的准确率，比零样本基线（使用ASR文本条件为42.6%）提高了6.5个百分点的绝对值。这证实了整合具有严格策划和动态仲裁的地方语料库能够显著提高LALM在地方口语处理上的表现。
URL
https://arxiv.org/abs/2603.05094
PDF
-
Training Dynamics-Aware Multi-Factor Curriculum Learning for Target Speaker Extraction
2026-03-05 08:41:17Yun Liu, Xuechen Liu, Xiaoxiao Miao, Junichi YamagishiAbstract
Target speaker extraction (TSE) aims to isolate a specific speaker's voice from multi-speaker mixtures. Despite strong benchmark results, real-world performance often degrades due to different interacting factors. Previous curriculum learning approaches for TSE typically address these factors separately, failing to capture their complex interactions and relying on predefined difficulty factors that may not align with actual model learning behavior. To address this challenge, we first propose a multi-factor curriculum learning strategy that jointly schedules SNR thresholds, speaker counts, overlap ratios, and synthetic/real proportions, enabling progressive learning from simple to complex scenarios. However, determining optimal scheduling without predefined assumptions remains challenging. We therefore introduce TSE-Datamap, a visualization framework that grounds curriculum design in observed training dynamics by tracking confidence and variability across training epochs. Our analysis reveals three characteristic data regions: (i) easy-to-learn examples where models consistently perform well, (ii) ambiguous examples where models oscillate between alternative predictions, and (iii) hard-to-learn examples where models persistently struggle. Guided by these data-driven insights, our methods improve extraction results over random sampling, with particularly strong gains in challenging multi-speaker scenarios.
Abstract (translated)
目标说话人提取（TSE）旨在从多说话人的混合信号中分离出特定说话人的声音。尽管在基准测试中取得了强劲的结果，但实际应用中的性能往往因各种相互作用的因素而下降。先前的课程学习方法通常单独解决这些因素，未能捕捉它们之间的复杂交互，并且依赖于预定义的难度因子，这些因子可能与模型的实际学习行为不一致。为了解决这一挑战，我们首先提出了一种多因素课程学习策略，该策略同时调度信噪比阈值、说话人数目、重叠比率以及合成/真实比例，从而实现从简单到复杂场景的逐步学习。然而，在没有预设假设的情况下确定最佳时间表仍然具有挑战性。因此，我们引入了TSE-Datamap，这是一种可视化框架，通过在整个训练周期中跟踪信心和变化来基于观察到的训练动态设计课程。 我们的分析揭示了三个特征数据区域： (i) 模型始终表现良好的易于学习的例子； (ii) 模型在多个预测之间摇摆不定的模棱两可的例子； (iii) 模型持续挣扎难以学习的例子。 根据这些数据驱动的见解，与随机采样相比，我们的方法提高了提取结果，在多说话人场景等具有挑战性的任务中尤其显著。
URL
https://arxiv.org/abs/2603.04943
PDF
