arXiv_SD
-
Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations
2026-04-02 14:28:42Haitong Sun, Stephen McIntosh, Kwanghee Choi, Eunjung Yeo, Daisuke Saito, Nobuaki MinematsuAbstract
Speech representations from self-supervised speech models (S3Ms) are known to be sensitive to phonemic contrasts, but their sensitivity to prosodic contrasts has not been directly measured. The ABX discrimination task has been used to measure phonemic contrast in S3M representations via minimal pairs. We introduce prosodic ABX, an extension of this framework to evaluate prosodic contrast with only a handful of examples and no explicit labels. Also, we build and release a dataset of English and Japanese minimal pairs and use it along with a Mandarin dataset to evaluate contrast in English stress, Japanese pitch accent, and Mandarin tone. Finally, we show that model and layer rankings are often preserved across several experimental conditions, making it practical for low-resource settings.
Abstract (translated)
自监督语音模型（S3Ms）的语音表征已知对音位对比敏感，但其对韵律对比的敏感性尚未被直接测量。ABX辨别任务已通过最小对立对用于测量S3M表征中的音位对比。我们引入了韵律ABX，这是该框架的扩展，仅用少量示例且无需显式标签即可评估韵律对比。此外，我们构建并发布了一个英语和日语最小对立对数据集，并将其与一个普通话数据集结合使用，以评估英语重音、日语音高重音和普通话声调的对比。最后，我们表明模型和层级的排名通常在多种实验条件下保持一致，使其在低资源场景中具有实用价值。
URL
https://arxiv.org/abs/2604.02102
PDF
-
Woosh: A Sound Effects Foundation Model
2026-04-02 11:49:00Ga\"etan Hadjeres, Marc Ferras, Khaled Koutini, Benno Weck, Alexandre Bittar, Thomas Hummel, Zineb Lahrici, Hakim Missoum, Joan Serr\`a, Yuki MitsufujiAbstract
The audio research community depends on open generative models as foundational tools for building novel approaches and establishing baselines. In this report, we present Woosh, Sony AI's publicly released sound effect foundation model, detailing its architecture, training process, and an evaluation against other popular open models. Being optimized for sound effects, we provide (1) a high-quality audio encoder/decoder model and (2) a text-audio alignment model for conditioning, together with (3) text-to-audio and (4) video-to-audio generative models. Distilled text-to-audio and video-to-audio models are also included in the release, allowing for low-resource operation and fast inference. Our evaluation on both public and private data shows competitive or better performance for each module when compared to existing open alternatives like StableAudio-Open and TangoFlux. Inference code and model weights are available at this https URL. Demo samples can be found at this https URL.
Abstract (translated)
音频研究社区依赖开源生成模型作为基础工具，以构建新颖方法并建立基准。在本报告中，我们介绍索尼AI公开发布的音效基础模型Woosh，详细说明其架构、训练过程，并与其他流行开源模型进行对比评估。针对音效优化，我们提供了（1）高质量的音频编码器/解码器模型和（2）用于条件控制的文本-音频对齐模型，以及（3）文本到音频和（4）视频到音频的生成模型。本次发布还包含蒸馏后的文本到音频与视频到音频模型，支持低资源运行与快速推理。我们在公开及私有数据上的评估显示，与StableAudio-Open和TangoFlux等现有开源方案相比，各模块均具有竞争力或更优表现。推理代码与模型权重可通过此https链接获取，演示样本可见于此https链接。
URL
https://arxiv.org/abs/2604.01929
PDF
-
FastTurn: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection
2026-04-02 11:00:37Chengyou Wang, Hongfei Xue, Chunjiang He, Jingbin Hu, Shuiyuan Wang, Bo Wu, Yuyu Ji, Jimeng Zheng, Ruofei Chen, Zhou Zhu, Lei XieAbstract
Recent advances in AudioLLMs have enabled spoken dialogue systems to move beyond turn-based interaction toward real-time full-duplex communication, where the agent must decide when to speak, yield, or interrupt while the user is still talking. Existing full-duplex approaches either rely on voice activity cues, which lack semantic understanding, or on ASR-based modules, which introduce latency and degrade under overlapping speech and noise. Moreover, available datasets rarely capture realistic interaction dynamics, limiting evaluation and deployment. To mitigate the problem, we propose \textbf{FastTurn}, a unified framework for low-latency and robust turn detection. To advance latency while maintaining performance, FastTurn combines streaming CTC decoding with acoustic features, enabling early decisions from partial observations while preserving semantic cues. We also release a test set based on real human dialogue, capturing authentic turn transitions, overlapping speech, backchannels, pauses, pitch variation, and environmental noise. Experiments show FastTurn achieves higher decision accuracy with lower interruption latency than representative baselines and remains robust under challenging acoustic conditions, demonstrating its effectiveness for practical full-duplex dialogue systems.
Abstract (translated)
URL
https://arxiv.org/abs/2604.01897
PDF
-
GAP-URGENet: A Generative-Predictive Fusion Framework for Universal Speech Enhancement
2026-04-02 09:45:58Xiaobin Rong, Yushi Wang, Zheng Wang, Jing LuAbstract
We introduce GAP-URGENet, a generative-predictive fusion framework developed for Track 1 of the ICASSP 2026 URGENT Challenge. The system integrates a generative branch, which performs full-stack speech restoration in a self-supervised representation domain and reconstructs the waveform via a neural vocoder, along with a predictive branch that performs spectrogram-domain enhancement, providing complementary cues. Outputs from both branches are fused by a post-processing module, which also performs bandwidth extension to generate the enhanced waveform at 48 kHz, later downsampled to the original sampling rate. This generative-predictive fusion improves robustness and perceptual quality, achieving top performance in the blind-test phase and ranking 1st in the objective evaluation. Audio examples are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.01832
PDF
-
PhiNet: Speaker Verification with Phonetic Interpretability
2026-04-02 04:01:50Yi Ma, Shuai Wang, Tianchi Liu, Haizhou LiAbstract
Despite remarkable progress, automatic speaker verification (ASV) systems typically lack the transparency required for high-accountability applications. Motivated by how human experts perform forensic speaker comparison (FSC), we propose a speaker verification network with phonetic interpretability, PhiNet, designed to enhance both local and global interpretability by leveraging phonetic evidence in decision-making. For users, PhiNet provides detailed phonetic-level comparisons that enable manual inspection of speaker-specific features and facilitate a more critical evaluation of verification outcomes. For developers, it offers explicit reasoning behind verification decisions, simplifying error tracing and informing hyperparameter selection. In our experiments, we demonstrate PhiNet's interpretability with practical examples, including its application in analyzing the impact of different hyperparameters. We conduct both qualitative and quantitative evaluations of the proposed interpretability methods and assess speaker verification performance across multiple benchmark datasets, including VoxCeleb, SITW, and LibriSpeech. Results show that PhiNet achieves performance comparable to traditional black-box ASV models while offering meaningful, interpretable explanations for its decisions, bridging the gap between ASV and forensic analysis.
Abstract (translated)
URL
https://arxiv.org/abs/2604.01590
PDF
-
Acoustic and perceptual differences between standard and accented Chinese speech and their voice clones
2026-04-02 03:17:41Tianle Yang, Chengzhe Sun, Phil Rose, Siwei LyuAbstract
Voice cloning is often evaluated in terms of overall quality, but less is known about accent preservation and its perceptual consequences. We compare standard and heavily accented Mandarin speech and their voice clones using a combined computational and perceptual design. Embedding-based analyses show no reliable accented-standard difference in original-clone distances across systems. In the perception study, clones are rated as more similar to their originals for standard than for accented speakers, and intelligibility increases from original to clone, with a larger gain for accented speech. These results show that accent variation can shape perceived identity match and intelligibility in voice cloning even when it is not reflected in an off-the-shelf speaker-embedding distance, and they motivate evaluating speaker identity preservation and accent preservation as separable dimensions.
Abstract (translated)
URL
https://arxiv.org/abs/2604.01562
PDF
-
Evolutionary Multi-Objective Fusion of Deepfake Speech Detectors
2026-04-01 19:17:59Vojt\v{e}ch Stan\v{e}k, Martin Pere\v{s}\'ini, Luk\'a\v{s} Sekanina, Anton Firc, Kamil MalinkaAbstract
While deepfake speech detectors built on large self-supervised learning (SSL) models achieve high accuracy, employing standard ensemble fusion to further enhance robustness often results in oversized systems with diminishing returns. To address this, we propose an evolutionary multi-objective score fusion framework that jointly minimizes detection error and system complexity. We explore two encodings optimized by NSGA-II: binary-coded detector selection for score averaging and a real-valued scheme that optimizes detector weights for a weighted sum. Experiments on the ASVspoof 5 dataset with 36 SSL-based detectors show that the obtained Pareto fronts outperform simple averaging and logistic regression baselines. The real-valued variant achieves 2.37% EER (0.0684 minDCF) and identifies configurations that match state-of-the-art performance while significantly reducing system complexity, requiring only half the parameters. Our method also provides a diverse set of trade-off solutions, enabling deployment choices that balance accuracy and computational cost.
Abstract (translated)
URL
https://arxiv.org/abs/2604.01330
PDF
-
FineLAP: Taming Heterogeneous Supervision for Fine-grained Language-Audio Pretraining
2026-04-01 17:09:25Xiquan Li, Xuenan Xu, Ziyang Ma, Wenxi Chen, Haolin He, Qiuqiang Kong, Xie ChenAbstract
Contrastively pretrained audio-language models (e.g., CLAP) excel at clip-level understanding but struggle with frame-level tasks. Existing extensions fail to exploit the varying granularity of real-world audio-text data, where massive clip-level textual descriptions coexist with limited frame-level annotations. This paper proposes Fine-grained Language-Audio Pretraining (FineLAP), a novel training paradigm that advances both clip- and frame-level alignment in CLAP with heterogeneous data. FineLAP introduces a dual-stream sigmoid loss with a cluster-based sampling strategy to jointly learn from clip- and frame-level supervision. To capture both global semantics and local details, FineLAP uses a decoupled audio projector on top of a self-supervised encoder. To alleviate the scarcity of temporally annotated data, we present FineLAP-100k, a large-scale synthetic SED dataset constructed through a scalable curation pipeline. Extensive experiments demonstrate that FineLAP achieves SOTA performance across multiple audio understanding tasks, including retrieval, classification, sound event detection, and text-to-audio grounding. Ablation studies further show that coarse- and fine-grained alignment are mutually beneficial, providing insights for building better audio-language models (ALMs).
Abstract (translated)
URL
https://arxiv.org/abs/2604.01155
PDF
-
Sona: Real-Time Multi-Target Sound Attenuation for Noise Sensitivity
2026-04-01 03:52:18Jeremy Zhengqi Huang, Emani Hicks, Sidharth, Gillian R. Hayes, Dhruv JainAbstract
For people with noise sensitivity, everyday soundscapes can be overwhelming. Existing tools such as active noise cancellation reduce discomfort by suppressing the entire acoustic environment, often at the cost of awareness of surrounding people and events. We present Sona, an interactive mobile system for real-time soundscape mediation that selectively attenuates bothersome sounds while preserving desired audio. Sona is built on a target-conditioned neural pipeline that supports simultaneous attenuation of multiple overlapping sound sources, overcoming the single-target limitation of prior systems. It runs in real time on-device and supports user-extensible sound classes through in-situ audio examples, without retraining. Sona is informed by a formative study with 68 noise-sensitive individuals. Through technical benchmarking and an in-situ study with 10 participants, we show that Sona achieves low-latency, multi-target attenuation suitable for live listening, and enables meaningful reductions in bothersome sounds while maintaining awareness of surroundings. These results point toward a new class of personal AI systems that support comfort and social participation by mediating real-world acoustic environments.
Abstract (translated)
URL
https://arxiv.org/abs/2604.00447
PDF
-
Vocal Prognostic Digital Biomarkers in Monitoring Chronic Heart Failure: A Longitudinal Observational Study
2026-03-31 23:08:43Fan Wu, Matthias P. N\"agele, Daryush D. Mehta, Elgar Fleisch, Frank Ruschitzka, Andreas J. Flammer, Filipe BarataAbstract
Objective: This study aimed to evaluate which voice features can predict health deterioration in patients with chronic HF. Background: Heart failure (HF) is a chronic condition with progressive deterioration and acute decompensations, often requiring hospitalization and imposing substantial healthcare and economic burdens. Current standard-of-care (SoC) home monitoring, such as weight tracking, lacks predictive accuracy and requires high patient engagement. Voice is a promising non-invasive biomarker, though prior studies have mainly focused on acute HF stages. Methods: In a 2-month longitudinal study, 32 patients with HF collected daily voice recordings and SoC measures of weight and blood pressure at home, with biweekly questionnaires for health status. Acoustic analysis generated detailed vowel and speech features. Time-series features were extracted from aggregated lookback windows (e.g., 7 days) to predict next-day health status. Explainable machine learning with nested cross-validation identified top vocal biomarkers, and a case study illustrated model application. Results: A total of 21,863 recordings were analyzed. Acoustic vowel features showed strong correlations with health status. Time-series voice features within the lookback window outperformed corresponding standard care measures, achieving peak sensitivity and specificity of 0.826 and 0.782 versus 0.783 and 0.567 for SoC metrics. Key prognostic voice features identifying deterioration included delayed energy shift, low energy variability, and higher shimmer variability in vowels, along with reduced speaking and articulation rate, lower phonation ratio, decreased voice quality, and increased formant variability in speech. Conclusion: Voice-based monitoring offers a non-invasive approach to detect early health changes in chronic HF, supporting proactive and personalized care.
Abstract (translated)
目的：本研究旨在评估哪些语音特征能够预测慢性心力衰竭（HF）患者的健康恶化情况。背景：心力衰竭（HF）是一种慢性进行性疾病，常出现急性失代偿，需频繁住院，造成巨大的医疗和经济负担。当前标准护理（SoC）的家庭监测（如体重追踪）预测准确性有限且需患者高度配合。语音作为一种非侵入性生物标志物具有潜力，但既往研究多聚焦于急性HF阶段。方法：在一项为期2个月的纵向研究中，32名HF患者每日居家录制语音并记录标准护理指标（体重、血压），每两周填写健康状况问卷。通过声学分析生成详细的元音及语音特征，并从聚合回溯窗口（如7天）中提取时序特征以预测次日健康状况。采用可解释机器学习结合嵌套交叉验证筛选关键语音生物标志物，并通过案例研究展示模型应用。结果：共分析21,863条录音。声学元音特征与健康状况呈强相关性。回溯窗口内的时序语音特征表现优于标准护理指标，其峰值敏感度和特异度分别达0.826和0.782，而标准护理指标仅为0.783和0.567。识别健康恶化的关键预后语音特征包括：元音能量延迟偏移、低能量变异度、更高抖动变异度，以及言语中说话/ articulation速率降低、发声时间比减少、音质下降和共振峰变异度增加。结论：基于语音的监测为慢性HF早期健康变化检测提供了非侵入性方案，有助于实现主动式个性化医疗。
URL
https://arxiv.org/abs/2604.00308
PDF
-
MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control
2026-03-31 22:35:58Sahil Kumar, Namrataben Patel, Honggang Wang, Youshan ZhangAbstract
MambaVoiceCloning (MVC) asks whether the conditioning path of diffusion-based TTS can be made fully SSM-only at inference, removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody, while preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time O(T) conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba-TTS systems that remain hybrid at inference, MVC removes attention-based duration and style modules under a fixed StyleTTS2 mel-diffusion-vocoder backbone. Trained on LJSpeech/LibriTTS and evaluated on VCTK, CSS10 (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but statistically reliable gains over StyleTTS2, VITS, and Mamba-attention hybrids in MOS/CMOS, F0 RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by 1.6x. Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and deployability.
Abstract (translated)
MambaVoiceCloning (MVC) 探讨能否在推理阶段将基于扩散的TTS的条件路径完全改造为仅使用SSM，从而在文本、节奏和韵律层面彻底移除所有注意力机制以及显式的RNN式循环层，同时在受控条件下保持或提升音质。MVC结合了门控双向Mamba文本编码器、由轻量级对齐教师网络（训练后丢弃）监督的时序双向Mamba，以及采用自适应层归一化调制的表现力Mamba，实现了线性时间复杂度O(T)的条件建模，并具备有界激活内存与实用的有限前瞻流式处理。与推理阶段仍为混合架构的先前Mamba-TTS系统不同，MVC在固定的StyleTTS2梅尔扩散声码器骨架上移除了基于注意力的时长与风格模块。该模型在LJSpeech/LibriTTS上训练，并在VCTK、CSS10（西班牙语/德语/法语）以及长篇古腾堡语料上进行评估。结果显示，MVC在MOS/CMOS听感评分、F0均方根误差、梅尔倒谱系数距离和词错误率上，相较于StyleTTS2、VITS及Mamba-注意力混合模型取得了适度但统计显著的提升，同时将编码器参数量压缩至2100万，并将吞吐量提高了1.6倍。扩散过程仍是主要延迟来源，但纯SSM条件路径在内存占用、稳定性与部署便利性方面具有改进优势。
URL
https://arxiv.org/abs/2604.00292
PDF
-
Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS
2026-03-31 21:47:52Kirill Borodin, Vasiliy Kudryavtsev, Maxim Maslov, Nikita Vasiliev, Mikhail Gorodnichev, Grach MkrtchianAbstract
We investigate multi-stage pretraining for prosody modeling in diffusion-based TTS. A speaker-conditioned dual-stream encoder is trained with masked language modeling followed by SigLIP-style cross-modal contrastive learning using mixed-phoneme batches, with an additional same-phoneme refinement stage studied separately. We evaluate intrinsic text-audio retrieval and downstream synthesis in Grad-TTS and a latent diffusion TTS system. The two-stage curriculum (MLM + mixed-phoneme contrastive learning) achieves the best overall synthesis quality in terms of intelligibility, speaker similarity, and perceptual measures. Although same-phoneme refinement improves prosodic retrieval, it reduces phoneme discrimination and degrades synthesis. These findings indicate that improvements in embedding-space metrics do not necessarily translate to better generative performance and highlight the need to balance phoneme discrimination and prosodic sensitivity in TTS pretraining.
Abstract (translated)
URL
https://arxiv.org/abs/2604.01247
PDF
-
Evaluating Generalization and Robustness in Russian Anti-Spoofing: The RuASD Initiative
2026-03-31 20:35:26Ksenia Lysikova, Kirill Borodin, Kirill BorodinAbstract
RuASD (Russian AntiSpoofing Dataset) is a dedicated, reproducible benchmark for Russian-language speech anti-spoofing designed to evaluate both in-domain discrimination and robustness to deployment-style distribution shifts. It combines a large spoof subset synthesized using 37 modern Russian-capable TTS and voice-cloning systems with a bona fide subset curated from multiple heterogeneous open Russian speech corpora, enabling systematic evaluation across diverse data sources. To emulate typical dissemination and channel effects in a controlled and reproducible manner, RuASD includes configurable simulations of platform and transmission distortions, including room reverberation, additive noise/music, and a range of speech-codec transcodings implemented via a unified processing chain. We benchmark a diverse set of publicly available anti-spoofing countermeasures spanning lightweight supervised architectures, graph-attention models, SSL-based detectors, and large-scale pretrained systems, and report reference results on both clean and simulated conditions to characterize robustness under realistic perturbation pipelines. The dataset is publickly available at \href{this https URL}{\underline{Hugging Face}} and \href{this https URL}{\underline{ModelScope}}.
Abstract (translated)
RuASD（俄语反欺骗数据集）是一个专门设计、可复现的俄语语音反欺骗基准，用于评估模型在域内判别能力以及对实际部署场景分布偏移的鲁棒性。该数据集结合了通过37种现代俄语TTS及声音克隆系统合成的大规模欺骗样本，以及从多个异构公开俄语语音语料库中精心筛选的真实语音样本，从而实现对不同数据来源的系统化评估。为在可控且可复现的条件下模拟典型传播与信道效应，RuASD包含可配置的平台与传输失真模拟，涵盖房间混响、加性噪声/音乐干扰，以及通过统一处理链实现的一系列语音编解码转码操作。我们针对一系列公开可用的反欺骗对策进行了基准测试，涵盖轻量级监督架构、图注意力模型、基于自监督学习的检测器及大规模预训练系统，并在纯净条件与模拟条件下报告了参考结果，以刻画其在真实扰动流程下的鲁棒性。该数据集已在\href{this https URL}{\underline{Hugging Face}}和\href{this https URL}{\underline{ModelScope}}公开。
URL
https://arxiv.org/abs/2604.02374
PDF
-
SIREN: Spatially-Informed Reconstruction of Binaural Audio with Vision
2026-03-31 14:44:09Mingyeong Song, Seoyeon Ko, Junhyug NohAbstract
Binaural audio delivers spatial cues essential for immersion, yet most consumer videos are monaural due to capture constraints. We introduce SIREN, a visually guided mono to binaural framework that explicitly predicts left and right channels. A ViT-based encoder learns dual-head self-attention to produce a shared scene map and end-to-end L/R attention, replacing hand-crafted masks. A soft, annealed spatial prior gently biases early L/R grounding, and a two-stage, confidence-weighted waveform-domain fusion (guided by mono reconstruction and interaural phase consistency) suppresses crosstalk when aggregating multi-crop and overlapping windows. Evaluated on FAIR-Play and MUSIC-Stereo, SIREN yields consistent gains on time-frequency and phase-sensitive metrics with competitive SNR. The design is modular and generic, requires no task-specific annotations, and integrates with standard audio-visual pipelines.
Abstract (translated)
URL
https://arxiv.org/abs/2603.29820
PDF
-
A Comprehensive Corpus of Biomechanically Constrained Piano Chords: Generation, Analysis, and Implications for Voicing and Psychoacoustics
Abstract
I present the generation and analysis of the largest known open-source corpus of playable piano chords (approximately 19.3 million entries). This dataset enumerates the two-handed search space subject to biomechanical constraints (two hands, each with 1.5 octave reach) to an unprecedented extent. To demonstrate the corpus's utility, the relationship between voicing shape and psychoacoustic targets was modeled. Harmonicity proved intrinsic to pitch-class identity: voicing statistics added negligible variance ($\Delta R^2 \approx 0.014\%$, $p \approx 0.13$). Conversely, voicing significantly predicted dissonance ($\Delta R^2 \approx 6.75\%$, $p \approx 0.0008$). Crucially, skewness ($\beta \approx +0.145$) was approximately 5.8$\times$ more effective than spread ($\beta \approx -0.025$) at predicting roughness. The analysis challenges the pedagogical emphasis on ``spread'': skewness is a stronger predictor of dissonance than spread. This suggests that clarity in ``open voicings'' is driven less by width than by negative skewness; achieving lower-register clearance by placing wide gaps at the bottom and allowing tighter clustering in the treble. The results demonstrate the corpus's ability to enable future research, especially in areas such as generative modeling, voice-leading topology, and psychoacoustic analysis.
Abstract (translated)
URL
https://arxiv.org/abs/2603.29710
PDF
-
LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space
2026-03-31 07:07:52Detai Xin, Shujie Hu, Chengzuo Yang, Chen Huang, Guoqiao Yu, Guanglu Wan, Xunliang CaiAbstract
We present LongCat-AudioDiT, a novel, non-autoregressive diffusion-based text-to-speech (TTS) model that achieves state-of-the-art (SOTA) performance. Unlike previous methods that rely on intermediate acoustic representations such as mel-spectrograms, the core innovation of LongCat-AudioDiT lies in operating directly within the waveform latent space. This approach effectively mitigates compounding errors and drastically simplifies the TTS pipeline, requiring only a waveform variational autoencoder (Wav-VAE) and a diffusion backbone. Furthermore, we introduce two critical improvements to the inference process: first, we identify and rectify a long-standing training-inference mismatch; second, we replace traditional classifier-free guidance with adaptive projection guidance to elevate generation quality. Experimental results demonstrate that, despite the absence of complex multi-stage training pipelines or high-quality human-annotated datasets, LongCat-AudioDiT achieves SOTA zero-shot voice cloning performance on the Seed benchmark while maintaining competitive intelligibility. Specifically, our largest variant, LongCat-AudioDiT-3.5B, outperforms the previous SOTA model (Seed-TTS), improving the speaker similarity (SIM) scores from 0.809 to 0.818 on Seed-ZH, and from 0.776 to 0.797 on Seed-Hard. Finally, through comprehensive ablation studies and systematic analysis, we validate the effectiveness of our proposed modules. Notably, we investigate the interplay between the Wav-VAE and the TTS backbone, revealing the counterintuitive finding that superior reconstruction fidelity in the Wav-VAE does not necessarily lead to better overall TTS performance. Code and model weights are released to foster further research within the speech community.
Abstract (translated)
我们提出了LongCat-AudioDiT，一种新颖的、基于非自回归扩散的文本转语音（TTS）模型，实现了最先进（SOTA）的性能。与以往依赖梅尔频谱图等中间声学表示的方法不同，LongCat-AudioDiT的核心创新在于直接在波形隐空间中进行操作。该方法有效缓解了误差累积问题，并大幅简化了TTS流程，仅需一个波形变分自编码器（Wav-VAE）和一个扩散主干网络。此外，我们为推理过程引入了两项关键改进：首先，我们识别并修正了一个长期存在的训练-推理不匹配问题；其次，我们用自适应投影引导替代了传统的无分类器引导，以提升生成质量。实验结果表明，尽管没有复杂的多阶段训练流程或高质量的人工标注数据集，LongCat-AudioDiT在Seed基准上实现了SOTA的零样本声音克隆性能，同时保持了具有竞争力的可懂度。具体而言，我们最大的变体LongCat-AudioDiT-3.5B超越了之前的SOTA模型（Seed-TTS），将说话人相似度（SIM）分数从Seed-ZH上的0.809提升至0.818，从Seed-Hard上的0.776提升至0.797。最后，通过全面的消融实验和系统性分析，我们验证了所提出模块的有效性。值得注意的是，我们研究了Wav-VAE与TTS主干网络之间的相互作用，揭示了一个反直觉的发现：Wav-VAE中更高的重建保真度并不一定带来更好的整体TTS性能。我们已公开代码和模型权重，以促进语音社区内的进一步研究。
URL
https://arxiv.org/abs/2603.29339
PDF
-
Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models
2026-03-31 04:57:56Ashish Seth, Sonal Kumar, Ramaneswaran Selvakumar, Nishit Anand, Utkarsh Tyagi, Prem Seetharaman, Ramani Duraiswami, Dinesh ManochaAbstract
Large Audio Language Models (LALMs) achieve strong performance on audio-language tasks; however, their reliability in real-world settings remains underexplored. We introduce Audio Hallucination Attacks (AHA), an attack suite called AHA-Eval, comprising 6.5K QA pairs designed to test whether LALMs genuinely ground their responses in the audio input. AHA targets two attack surfaces: (i) query-based attacks, which exploit question structure to induce hallucinations about absent sounds, and (ii) audio-based attacks, which inject synthetic speech describing non-existent events into the audio stream. Evaluating state-of-the-art LALMs, including Audio Flamingo 3 and Gemini 3 Pro, we observe high attack success rates of 95.35% and 79.65%, respectively, revealing a reliability gap that is hidden by standard benchmark performance. To mitigate this, we propose a 120K QA post-alignment dataset, AHA-Guard, which successfully reduces attack success rates by up to 49%.
Abstract (translated)
大型音频语言模型（LALMs）在音频-语言任务上表现出色；然而，其在真实场景中的可靠性尚未得到充分探索。我们提出了音频幻觉攻击（AHA）及其评估套件AHA-Eval，该套件包含6500组问答对，用于测试LALMs是否真正基于音频输入生成响应。AHA针对两种攻击面：（i）基于查询的攻击，利用问题结构诱使模型对不存在的音频产生幻觉；（ii）基于音频的攻击，向音频流中注入描述虚构事件的合成语音。在对包括Audio Flamingo 3和Gemini 3 Pro在内的先进LALMs进行评估时，我们观察到其攻击成功率分别高达95.35%和79.65%，揭示了标准基准测试所掩盖的可靠性差距。为缓解此问题，我们提出了一个包含12万组问答的后对齐数据集AHA-Guard，该数据集成功将攻击成功率最高降低了49%。
URL
https://arxiv.org/abs/2603.29263
PDF
-
Asymmetric Encoder-Decoder Based on Time-Frequency Correlation for Speech Separation
2026-03-31 00:37:15Ui-Hyeop Shin, Hyung-Min ParkAbstract
Speech separation in realistic acoustic environments remains challenging because overlapping speakers, background noise, and reverberation must be resolved simultaneously. Although recent time-frequency (TF) domain models have shown strong performance, most still rely on late-split architectures, where speaker disentanglement is deferred to the final stage, creating an information bottleneck and weakening discriminability under adverse conditions. To address this issue, we propose SR-CorrNet, an asymmetric encoder-decoder framework that introduces the separation-reconstruction (SepRe) strategy into a TF dual-path backbone. The encoder performs coarse separation from mixture observations, while the weight-shared decoder progressively reconstructs speaker-discriminative features with cross-speaker interaction, enabling stage-wise refinement. To complement this architecture, we formulate speech separation as a structured correlation-to-filter problem: spatio-spectro-temporal correlations computed from the observations are used as input features, and the corresponding deep filters are estimated to recover target signals. We further incorporate an attractor-based dynamic split module to adapt the number of output streams to the actual speaker configuration. Experimental results on WSJ0-2/3/4/5Mix, WHAMR!, and LibriCSS demonstrate consistent improvements across anechoic, noisy-reverberant, and real-recorded conditions in both single- and multi-channel settings, highlighting the effectiveness of TF-domain SepRe with correlation-based filter estimation for speech separation.
Abstract (translated)
在真实声学环境中的语音分离仍是一项挑战，因为重叠的说话人、背景噪声和混响必须同时得到解决。尽管近期时频域模型展现了强劲性能，但大多数仍依赖晚期分离架构，即将说话人解耦推迟至最后阶段，这造成了信息瓶颈并在恶劣条件下削弱了判别能力。为应对这一问题，我们提出了SR-CorrNet，一个将分离-重建策略引入时频双路径骨干网络的非对称编解码框架。编码器从混合观测中执行粗分离，而权重共享的解码器通过跨说话人交互逐步重建具有说话人判别性的特征，实现逐阶段细化。为补充此架构，我们将语音分离表述为一个结构化相关性-滤波器问题：利用从观测中计算出的时空频相关性作为输入特征，并估计相应的深度滤波器以恢复目标信号。我们进一步引入基于吸引子的动态分离模块，使输出流数量能自适应实际说话人配置。在WSJ0-2/3/4/5Mix、WHAMR!和LibriCSS数据集上的实验结果表明，在单通道和多通道设置下，该方法在消声、噪声混响及真实录音条件中均实现持续改进，凸显了基于相关性的滤波器估计与时频域分离-重建策略在语音分离中的有效性。
URL
https://arxiv.org/abs/2603.29097
PDF
-
IQRA 2026: Interspeech Challenge on Automatic Assessment Pronunciation for Modern Standard Arabic
2026-03-31 00:05:07Yassine El Kheir, Amit Meghanani, Mostafa Shahin, Omnia Ibrahim, Shammur Absar Chowdhury, Nada AlMarwani, Youssef Elshahawy, Ahmed AliAbstract
We present the findings of the second edition of the IQRA Interspeech Challenge, a challenge on automatic Mispronunciation Detection and Diagnosis (MDD) for Modern Standard Arabic (MSA). Building on the previous edition, this iteration introduces \textbf{Iqra\_Extra\_IS26}, a new dataset of authentic human mispronounced speech, complementing the existing training and evaluation resources. Submitted systems employed a diverse range of approaches, spanning CTC-based self-supervised learning models, two-stage fine-tuning strategies, and using large audio-language models. Compared to the first edition, we observe a substantial jump of \textbf{0.28 in F1-score}, attributable both to novel architectures and modeling strategies proposed by participants and to the additional authentic mispronunciation data made available. These results demonstrate the growing maturity of Arabic MDD research and establish a stronger foundation for future work in Arabic pronunciation assessment.
Abstract (translated)
URL
https://arxiv.org/abs/2603.29087
PDF
-
Constructing Composite Features for Interpretable Music-Tagging
2026-03-30 16:25:58Chenhao Xue, Weitao Hu, Joyraj Chakraborty, Zhijin Guo, Kang Li, Tianyu Shi, Martin Reed, Nikolaos ThomosAbstract
Combining multiple audio features can improve the performance of music tagging, but common deep learning-based feature fusion methods often lack interpretability. To address this problem, we propose a Genetic Programming (GP) pipeline that automatically evolves composite features by mathematically combining base music features, thereby capturing synergistic interactions while preserving interpretability. This approach provides representational benefits similar to deep feature fusion without sacrificing interpretability. Experiments on the MTG-Jamendo and GTZAN datasets demonstrate consistent improvements compared to state-of-the-art systems across base feature sets at different abstraction levels. It should be noted that most of the performance gains are noticed within the first few hundred GP evaluations, indicating that effective feature combinations can be identified under modest search budgets. The top evolved expressions include linear, nonlinear, and conditional forms, with various low-complexity solutions at top performance aligned with parsimony pressure to prefer simpler expressions. Analyzing these composite features further reveals which interactions and transformations tend to be beneficial for tagging, offering insights that remain opaque in black-box deep models.
Abstract (translated)
结合多种音频特征可以提升音乐标签任务的性能，但常见的基于深度学习的特征融合方法往往缺乏可解释性。针对这一问题，我们提出一种遗传编程（GP）流程，该流程通过数学方式组合基础音乐特征，自动演化复合特征，从而在保持可解释性的同时捕捉协同交互效应。该方法在提供与深度特征融合相似的表示优势的同时，并未牺牲可解释性。在MTG-Jamendo和GTZAN数据集上的实验表明，与最先进系统相比，该方法在不同抽象层级的基础特征集上均取得持续改进。值得注意的是，大部分性能提升出现在前几百次GP评估中，表明在适度的搜索预算下即可识别出有效的特征组合。排名靠前的演化表达式包含线性、非线性及条件形式，其中多种低复杂度方案因受精简压力影响而优先采用更简洁的表达式，同时保持顶尖性能。进一步分析这些复合特征可揭示哪些交互与变换对标签任务有益，而这些洞见在黑盒深度模型中仍不透明。
URL
https://arxiv.org/abs/2603.28644
PDF
