arXiv_SD
-
A Probabilistic Generative Model for Spectral Speech Enhancement
2026-03-30 13:44:04Marco Hidalgo-Araya, Rapha\"el Tr\'esor, Bart Van Erp, Wouter W. L. Nuijten, Thijs Van De Laar, Bert De VriesAbstract
Speech enhancement in hearing aids remains a difficult task in nonstationary acoustic environments, mainly because current signal processing algorithms rely on fixed, manually tuned parameters that cannot adapt in situ to different users or listening contexts. This paper introduces a unified modular framework that formulates signal processing, learning, and personalization as Bayesian inference with explicit uncertainty tracking. The proposed framework replaces ad hoc algorithm design with a single probabilistic generative model that continuously adapts to changing acoustic conditions and user preferences. It extends spectral subtraction with principled mechanisms for in-situ personalization and adaptation to acoustic context. The system is implemented as an interconnected probabilistic state-space model, and inference is performed via variational message passing in the \texttt{this http URL} probabilistic programming environment, enabling real-time Bayesian processing under hearing-aid constraints. Proof-of-concept experiments on the \emph{VoiceBank+DEMAND} corpus show competitive speech quality and noise reduction with 85 effective parameters. The framework provides an interpretable, data-efficient foundation for uncertainty-aware, adaptive hearing-aid processing and points toward devices that learn continuously through probabilistic inference.
Abstract (translated)
URL
https://arxiv.org/abs/2603.28436
PDF
-
On the Usefulness of Diffusion-Based Room Impulse Response Interpolation to Microphone Array Processing
2026-03-30 09:30:54Sagi Della Torre, Mirco Pezzoli, Fabio Antonacci, Sharon GannotAbstract
Room Impulse Responses estimation is a fundamental problem in spatial audio processing and speech enhancement. In this paper, we build upon our previously introduced diffusion-based inpainting framework for Room Impulse Response interpolation and demonstrate its applicability to enhancing the performance of practical multi-microphone array processing tasks. Furthermore, we validate the robustness of this method in interpolating real-world Room Impulse Responses.
Abstract (translated)
房间脉冲响应估计是空间音频处理与语音增强领域的一个基础性问题。本文在我们先前提出的基于扩散的修复框架基础上，进一步构建了用于房间脉冲响应插值的方法，并展示了其在提升实际多麦克风阵列处理任务性能方面的应用潜力。此外，我们还验证了该方法在真实世界房间脉冲响应插值任务中的鲁棒性。
URL
https://arxiv.org/abs/2603.28209
PDF
-
Audio Language Model for Deepfake Detection Grounded in Acoustic Chain-of-Thought
2026-03-30 04:27:59Runkun Chen, Yixiong Fang, Pengyu Chang, Yuante Li, Massa Baali, Bhiksha RamakrishnanAbstract
Deepfake speech detection systems are often limited to binary classification tasks and struggle to generate interpretable reasoning or provide context-rich explanations for their decisions. These models primarily extract latent embeddings for authenticity detection but fail to leverage structured acoustic evidence such as prosodic, spectral, and physiological attributes in a meaningful manner. This paper introduces CoLMbo-DF, a Feature-Guided Audio Language Model that addresses these limitations by integrating robust deepfake detection with explicit acoustic chain-of-thought reasoning. By injecting structured textual representations of low-level acoustic features directly into the model prompt, our approach grounds the model's reasoning in interpretable evidence and improves detection accuracy. To support this framework, we introduce a novel dataset of audio pairs paired with chain-of-thought annotations. Experiments show that our method, trained on a lightweight open-source language model, significantly outperforms existing audio language model baselines despite its smaller scale, marking a significant advancement in explainable deepfake speech detection.
Abstract (translated)
深度伪造语音检测系统往往局限于二分类任务，难以生成可解释的推理或提供富含上下文的决策依据。这些模型主要提取潜在嵌入以进行真实性检测，却未能以有意义的方式利用韵律、频谱和生理属性等结构化声学证据。本文提出CoLMbo-DF——一种特征引导式音频语言模型，通过将稳健的深度伪造检测与显式声学思维链推理相结合，解决了上述局限性。该方法将低层声学特征的文本化结构化表示直接注入模型提示词，使模型推理有据可依，并提升了检测准确率。为支持该框架，我们引入了首个包含思维链标注的音频对数据集。实验表明，即使在轻量级开源语言模型上训练，我们的方法仍显著优于现有音频语言模型基线，标志着可解释深度伪造语音检测领域的重大进展。
URL
https://arxiv.org/abs/2603.28021
PDF
-
Advancing Multi-Instrument Music Transcription: Results from the 2025 AMT Challenge
2026-03-29 05:41:42Ojas Chaturvedi, Kayshav Bhardwaj, Tanay Gondil, Benjamin Shiue-Hal Chou, Kristen Yeon-Ji Yun, Yung-Hsiang Lu, Yujia Yan, Sungkyun ChangAbstract
This paper presents the results of the 2025 Automatic Music Transcription (AMT) Challenge, an online competition to benchmark progress in multi-instrument transcription. Eight teams submitted valid solutions; two outperformed the baseline MT3 model. The results highlight both advances in transcription accuracy and the remaining difficulties in handling polyphony and timbre variation. We conclude with directions for future challenges: broader genre coverage and stronger emphasis on instrument detection.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27528
PDF
-
Investigation on the Robustness of Acoustic Foundation Models on Post Exercise Speech
2026-03-29 04:17:09Xiangyuan Xue, Yuyu Wang, Ruijie Yao, Xiaoyue Ni, Xiaofan Jiang, Jingping NieAbstract
Automatic speech recognition (ASR) has been extensively studied on neutral and stationary speech, yet its robustness under post-exercise physiological shift remains underexplored. Compared with resting speech, post-exercise speech often contains micro-breaths, non-semantic pauses, unstable phonation, and repetitions caused by reduced breath support, making transcription more difficult. In this work, we benchmark acoustic foundation models on post-exercise speech under a unified evaluation protocol. We compare sequence-to-sequence models (Whisper and FunASR/Paraformer) and self-supervised encoders with CTC decoding (Wav2Vec2, HuBERT, and WavLM), under both off-the-shelf inference and post-exercise in-domain fine-tuning. Across the Static/Post-All benchmark, most models degrade on post-exercise speech, while FunASR shows the strongest baseline robustness at 14.57% WER and 8.21% CER on Post-All. Fine-tuning substantially improves several CTC-based models, whereas Whisper shows unstable adaptation. As an exploratory case study, we further stratify results by fluent and non-fluent speakers; although the non-fluent subset is small, it is consistently more challenging than the fluent subset. Overall, our findings show that post-exercise ASR robustness is strongly model-dependent, that in-domain adaptation can be highly effective but not uniformly stable, and that future post-exercise ASR studies should explicitly separate fluency-related effects from exercise-induced speech variation.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27508
PDF
-
SHroom: A Python Framework for Ambisonics Room Acoustics Simulation and Binaural Rendering
Abstract
Spherical Harmonics ROOM), an open-source Python library for room acoustics simulation using Ambisonics, available at this https URL and installable via \texttt{pip install pyshroom}. \textbf{shroom} projects image-source contributions onto a Spherical Harmonics (SH) basis, yielding a composable pipeline for binaural decoding, spherical array simulation, and real-time head rotation. Benchmarked against \texttt{pyroomacoustics} with an $N=30$ reference, \textbf{shroom} with Magnitude Least Squares (MagLS) achieves perceptual transparency (2.02~dB Log Spectral Distance (LSD) at $N=5$, within the 1--2~dB Just Noticeable Difference (JND)) while its fixed-once decode amortises over multiple sources ($K=1$-to-$8$: slowdown narrows from $7\times$ to $3.1\times$). For dynamic head rotation, \textbf{shroom} applies a Wigner-D multiply at $<1$~ms/frame, making it the only architecturally viable real-time choice.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27342
PDF
-
Two-Stage Acoustic Adaptation with Gated Cross-Attention Adapters for LLM-Based Multi-Talker Speech Recognition
2026-03-28 09:14:41Hao Shi, Yuan Gao, Xugang Lu, Tatsuya KawaharaAbstract
Large Language Models (LLMs) are strong decoders for Serialized Output Training (SOT) in two-talker Automatic Speech Recognition (ASR), yet their performance degrades substantially in challenging conditions such as three-talker mixtures. A key limitation is that current systems inject acoustic evidence only through a projected prefix, which can be lossy and imperfectly aligned with the LLM input space, providing insufficient fine-grained grounding during decoding. Addressing this limitation is crucial for robust multi-talker ASR, especially in three-talker mixtures. This paper improves LLM-based multi-talker ASR by explicitly injecting talker-aware acoustic evidence into the decoder. We first revisit Connectionist Temporal Classification (CTC)-derived prefix prompting and compare three variants with increasing acoustic content. The CTC information is obtained using the serialized CTC proposed in our previous works. While acoustic-enriched prompts outperform the SOT-only baseline, prefix-only conditioning remains inadequate for three-talker mixtures. We therefore propose a lightweight gated residual cross-attention adapter and design a two-stage acoustic adaptation framework based on low-rank updates (LoRA). In Stage 1, we insert gated cross-attention adapters after the self-attention sub-layer to stably inject acoustic embeddings as external memory. In Stage 2, we refine both the cross-attention adapters and the pretrained LLM's self-attention projections using parameter-efficient LoRA, improving robustness for large backbones under limited data; the learned updates are merged into the base weights for inference. Experiments on Libri2Mix/Libri3Mix under clean and noisy conditions show consistent gains, with particularly large improvements in three-talker settings.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27205
PDF
-
Diachronic Modeling of Tonal Coherence on the Tonnetz Across Classical and Popular Repertoires
Abstract
How do different musical traditions achieve tonal coherence? Most computational measures to date have analysed tonal coherence in terms of a single dimension, whereas a multi-dimensional analyses have not been sufficiently explored. We propose a new model drawing on the concept of the Tonnetz -- we define two partially independent measures: \emph{tonal focus}, the concentration of pitch content near a tonal center; and \emph{tonal connection}, the degree to which pitch content reflects structured intervallic pathways back to that center. Analyzing over 2,800 pieces from Western classical and popular traditions, we find that these traditions occupy overlapping yet distinguishable regions of the two-dimensional space. Popular music shows higher tonal focus, while classical music exhibits higher tonal connection. Our complementary measures ground the differences between different tonal styles in quantitative evidence, and offer interpretable dimensions for computational music analysis and controllable generation.
Abstract (translated)
不同的音乐传统如何实现调性统一？迄今为止大多数计算方法仅从单一维度分析调性统一，而多维度分析尚未得到充分探索。我们借鉴音网（Tonnetz）概念提出新模型，定义两个部分独立的度量指标：**调性聚焦**（tonal focus），指音高内容在调性中心附近的集中程度；**调性关联**（tonal connection），指音高内容反映回归该中心的规则化音程路径的程度。通过分析2800余首西方古典与流行音乐作品，我们发现这些传统在二维空间中占据重叠但可区分的区域。流行音乐表现出更高的调性聚焦，而古典音乐呈现更高的调性关联。这些互补性度量指标为不同调性风格的差异提供了量化依据，并为计算音乐分析与可控生成提供了可解释的维度。
URL
https://arxiv.org/abs/2603.27035
PDF
-
Algo P'art: An Algorithmic Reconstruction of Arvo P'art's Summa
Abstract
Arvo Pärt is one of the most popular contemporary composers, known for his highly original tintinnabuli style. Works in this style are typically composed according to precise procedures and have even been described as algorithmic compositions. To understand how algorithmic Pärt's music exactly is, this paper presents an analysis by synthesis: it proposes an algorithm that almost completely reconstructs the score of Summa, his "most strictly constructed and most encrypted work," according to Pärt himself in 1994. The piece is analyzed and then formalized using so-called tintinnabuli processes. An implementation of the resulting algorithm generates a musical score matching Summa in over 93% of the notes. Due to interdependencies between the voices, only half of the mistakes (3.5%) need to be corrected to reproduce the original score faithfully. This study shows that Summa is a largely algorithmic composition and offers new perspectives on the music of Arvo Pärt.
Abstract (translated)
阿尔沃·帕尔特是当代最受欢迎的作曲家之一，以其高度原创的钟鸣风格著称。该风格的作品通常依据精确的程序创作，甚至被称为算法作曲。为探究帕尔特音乐的算法性究竟如何，本文采用分析性合成方法：提出一种算法，据帕尔特1994年所言，该算法几乎完整重构了他"结构最严格、加密最深"的作品《总和》的乐谱。研究首先分析该作品，继而运用所谓钟鸣过程进行形式化建模。所生成算法的实现结果可产出与《总和》超过93%音符匹配的乐谱。由于声部间的相互关联，仅需修正半数错误（3.5%）即可忠实再现原谱。本研究证明《总和》本质上是算法作曲，并为理解阿尔沃·帕尔特的音乐提供了新视角。
URL
https://arxiv.org/abs/2603.26989
PDF
-
Rhythmic segment analysis: Conceptualizing, visualizing, and measuring rhythmic data
Abstract
This paper develops a framework for conceptualizing, visualizing, and measuring regularities in rhythmic data. I propose to think about rhythmic data in terms of interval segments: fixed-length groups of consecutive intervals, which can be decomposed into a duration and a pattern (the ratios between the intervals). This simple conceptual framework unifies three rhythmic visualization methods and yields a fourth: the pattern-duration plot. When paired with a cluster transition network, it intuitively reveals regularities in both synthetic and real-world rhythmic data. Moreover, the framework generalizes two common measures of rhythmic structure: rhythm ratios and the normalized pairwise variability index (nPVI). In particular, nPVI can be reconstructed as the average distance from isochrony, and I propose a more general measure of anisochrony to replace it. Finally, the novel concept of quantality may shed light on wider debates regarding small-integer-ratio rhythms.
Abstract (translated)
URL
https://arxiv.org/abs/2603.26988
PDF
-
Probabilistic Multilabel Graphical Modelling of Motif Transformations in Symbolic Music
2026-03-27 14:42:15Ron Taieb, Yoel Greenberg, Barak SoberAbstract
Motifs often recur in musical works in altered forms, preserving aspects of their identity while undergoing local variation. This paper investigates how such motivic transformations occur within their musical context in symbolic music. To support this analysis, we develop a probabilistic framework for modeling motivic transformations and apply it to Beethoven's piano sonatas by integrating multiple datasets that provide melodic, rhythmic, harmonic, and motivic information within a unified analytical representation. Motif transformations are represented as multilabel variables by comparing each motif instance to a designated reference occurrence within its local context, ensuring consistent labeling across transformation families. We introduce a multilabel Conditional Random Field to model how motif-level musical features influence the occurrence of transformations and how different transformation families tend to co-occur. Our goal is to provide an interpretable, distributional analysis of motivic transformation patterns, enabling the study of their structural relationships and stylistic variation. By linking computational modeling with music-theoretical interpretation, the proposed framework supports quantitative investigation of musical structure and complexity in symbolic corpora and may facilitate the analysis of broader compositional patterns and writing practices.
Abstract (translated)
主题常以变化形式在音乐作品中反复出现，在保持其身份特征的同时经历局部变异。本文研究符号音乐中此类主题变换如何在音乐语境中发生。为支持该分析，我们构建了一个用于模拟主题变换的概率框架，并将其应用于贝多芬钢琴奏鸣曲，通过整合提供旋律、节奏、和声及主题信息的多个数据集，形成统一的分析表征。通过将每个主题实例与其局部语境中的指定参考出现进行比较，主题变换被表示为多标签变量，从而确保变换类别间标签的一致性。我们引入多标签条件随机场来建模主题层面的音乐特征如何影响变换的发生，以及不同变换类别如何倾向于共现。我们的目标是提供一种可解释的、分布式的主题变换模式分析，以研究其结构关系与风格变异。通过将计算建模与音乐理论阐释相结合，所提出的框架支持对符号语料库中音乐结构与复杂性的量化研究，并可能促进对更广泛作曲模式与写作实践的分析。
URL
https://arxiv.org/abs/2603.26478
PDF
-
CA-TCN: A Causal-Anticausal Temporal Convolutional Network for Direct Auditory Attention Decoding
2026-03-27 13:21:28I\~nigo Garc\'ia-Ugarte, Rub\'en Eguinoa, Ricardo San Mart\'in, Daniel Paternain, Carmen VidaurreAbstract
A promising approach for steering auditory attention in complex listening environments relies on Auditory Attention Decoding (AAD), which aim to identify the attended speech stream in a multiple speaker scenario from neural recordings. Entrainment-based AAD approaches, typically assume access to clean speech sources and electroencephalography (EEG) signals to exploit low-frequency correlations between the neural response and the attended stimulus. In this study, we propose CA-TCN, a Causal-Anticausal Temporal Convolutional Network that directly classifies the attended speaker. The proposed architecture integrates several best practices from convolutional neural networks in sequence processing tasks. Importantly, it explicitly aligns auditory stimuli and neural responses by employing separate causal and anticausal convolutions respectively, with distinct receptive fields operating in opposite temporal directions. Experimental results, obtained through comparisons with three baseline AAD models, demonstrated that CA-TCN consistently improved decoding accuracy across datasets and decision windows, with gains ranging from 0.5% to 3.2% for subject-independent models and from 0.8% to 2.9% for subject-specific models compared with the next best-performing model, AADNet. Moreover, these improvements were statistically significant in four of the six evaluated settings when comparing Minimum Expected Switch Duration distributions. Beyond accuracy, the model demonstrated spatial robustness across different conditions, as the EEG spatial filters exhibited stable patterns across datasets. Overall, this work introduces an accurate and unified AAD model that outperforms existing methods while considering practical benefits for online processing scenarios. These findings contribute to advancing the state of AAD and its applicability in real-world systems.
Abstract (translated)
在复杂听觉环境中引导听觉注意力的一个前景方法依赖于听觉注意力解码（AAD），其旨在从多说话人场景的神经记录中识别被注意的语音流。基于锁相效应的AAD方法通常假设可获取纯净语音源和脑电图（EEG）信号，以利用神经响应与被注意刺激之间的低频相关性。本研究提出CA-TCN，一种因果-反因果时序卷积网络，可直接分类被注意的说话人。所提出的架构整合了序列处理任务中卷积神经网络的多项最佳实践。关键的是，该模型通过分别采用因果卷积与反因果卷积，以相反时间方向上具有不同感受野的方式，实现了听觉刺激与神经响应的显式对齐。通过与三种基线AAD模型对比获得的实验结果表明，CA-TCN在跨数据集和决策窗口的解码精度上均持续提升：相较于次优模型AADNet，其针对受试者独立模型的增益范围为0.5%至3.2%，针对受试者专属模型的增益范围为0.8%至2.9%。此外，在比较最小预期切换时长分布时，六个评估设置中的四个显示这些改进具有统计显著性。除精度外，该模型在不同条件下展现出空间鲁棒性，因为脑电空间滤波器在各数据集间呈现稳定模式。总体而言，本研究提出了一种精确且统一的AAD模型，在优于现有方法的同时兼顾了在线处理场景的实际效益。这些发现有助于推动AAD技术发展及其在真实系统中的适用性。
URL
https://arxiv.org/abs/2603.26394
PDF
-
LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling
2026-03-27 12:45:35Xiaoyu Fan, Huizhi Xie, Wei Zou, Yunzhang ChenAbstract
Large language model (LLM)-based text-to-speech (TTS) systems achieve remarkable naturalness via autoregressive (AR) decoding, but require N sequential steps to generate N speech tokens. We present LLaDA-TTS, which replaces the AR LLM with a masked diffusion model that completes generation in a fixed number of parallel steps, decoupling inference latency from sequence length. Remarkably, using only 50 hours of fine-tuning data, we successfully transfer a pretrained AR checkpoint to the masked diffusion paradigm via bidirectional attention. At 64 steps, LLaDA-TTS achieves 0.98% CER (zh) and 1.96% WER (en) on Seed-TTS-Eval, matching the original CosyVoice 3 baseline performance while delivering a 2x LLM-stage speedup--a notable acceleration achieved despite the absence of KV cache, an optimization the AR baseline heavily relies on. Beyond acceleration, the bidirectional architecture naturally enables zero-shot speech editing--including word-level insertion, deletion, and substitution--without any additional training. Theoretically, we prove that AR-pretrained weights are near-optimal for bidirectional masked prediction under the locality property of acoustic tokens, explaining this rapid convergence. This general method modifies only the attention mask and objective, applying seamlessly to any LLM-based AR TTS system. Code and audio samples will be available at this https URL.
Abstract (translated)
基于大语言模型（LLM）的文本转语音（TTS）系统通过自回归（AR）解码实现了卓越的自然度，但需要N个连续步骤来生成N个语音标记。我们提出的LLaDA-TTS用掩码扩散模型取代了AR LLM，以固定数量的并行步骤完成生成，从而将推理延迟与序列长度解耦。值得注意的是，仅使用50小时的微调数据，我们通过双向注意力成功地将预训练的AR检查点迁移至掩码扩散范式。在64步推理下，LLaDA-TTS在Seed-TTS-Eval上实现了0.98%（中文）和1.96%（英文）的CER/WER，匹配了原始CosyVoice 3基线性能，同时实现了2倍的LLM阶段加速——这一显著加速是在未使用KV缓存（AR基线严重依赖的优化）的情况下达成的。除加速外，双向架构天然支持零样本语音编辑——包括词级插入、删除和替换——而无需额外训练。理论上，我们证明在声学标记局部性假设下，AR预训练权重对双向掩码预测近乎最优，这解释了其快速收敛。这一通用方法仅需修改注意力掩码和目标函数，即可无缝应用于任何基于LLM的AR TTS系统。代码及音频样本将在此链接发布。
URL
https://arxiv.org/abs/2603.26364
PDF
-
A Power-Weighted Noncentral Complex Gaussian Distribution
Abstract
The complex Gaussian distribution has been widely used as a fundamental spectral and noise model in signal processing and communication. However, its Gaussian structure often limits its ability to represent the diverse amplitude characteristics observed in individual source signals. On the other hand, many existing non-Gaussian amplitude distributions derived from hyperspherical models achieve good empirical fit due to their power-law structures, while they do not explicitly account for the complex-plane geometry inherent in complex-valued observations. In this paper, we propose a new probabilistic model for complex-valued random variables, which can be interpreted as a power-weighted noncentral complex Gaussian distribution. Unlike conventional hyperspherical amplitude models, the proposed model is formulated directly on the complex plane and preserves the geometric structure of complex-valued observations while retaining a higher-dimensional interpretation. The model introduces a nonlinear phase diffusion through a single shape parameter, enabling continuous control of the distributional geometry from arc-shaped diffusion along the phase direction to concentration of probability mass toward the origin. We formulate the proposed distribution and analyze the statistical properties of the induced amplitude distribution. The derived amplitude and power distributions provide a unified framework encompassing several widely used distributions in signal modeling, including the Rice, Nakagami, and gamma distributions. Experimental results on speech power spectra demonstrate that the proposed model consistently outperforms conventional distributions in terms of log-likelihood.
Abstract (translated)
复高斯分布作为基础频谱与噪声模型在信号处理与通信领域被广泛使用。然而，其高斯结构常限制了对单个源信号中多样化幅度特性的表征能力。另一方面，许多基于超球面模型推导出的非高斯幅度分布虽因幂律结构而具有良好经验拟合效果，却未能明确考虑复值观测数据所固有的复平面几何特性。本文提出一种针对复值随机变量的新概率模型，可视为幂加权非中心复高斯分布。与传统超球面幅度模型不同，该模型直接在复平面上构建，在保留复值观测几何结构的同时维持高维解释性。模型通过单一形状参数引入非线性相位扩散，实现了从相位方向弧状扩散到概率质量向原点聚集的连续几何控制。本文建立了该分布的数学形式并分析了导出幅度分布的统计特性。推导出的幅度与功率分布为信号建模中广泛使用的莱斯分布、中岛分布及伽马分布提供了统一框架。在语音功率谱上的实验结果表明，该模型在对数似然指标上持续优于传统分布。
URL
https://arxiv.org/abs/2603.26344
PDF
-
Cinematic Audio Source Separation Using Visual Cues
2026-03-27 06:54:41Kang Zhang, Suyeon Lee, Arda Senocak, Joon Son ChungAbstract
Cinematic Audio Source Separation (CASS) aims to decompose mixed film audio into speech, music, and sound effects, enabling applications like dubbing and remastering. Existing CASS approaches are audio-only, overlooking the inherent audio-visual nature of films, where sounds often align with visual cues. We present the first framework for audio-visual CASS (AV-CASS), leveraging visual context to enhance separation quality. Our method formulates CASS as a conditional generative modeling problem using conditional flow matching, enabling multimodal audio source separation. To address the lack of cinematic datasets with isolated sound tracks, we introduce a training data synthesis pipeline that pairs in-the-wild audio and video streams (e.g., facial videos for speech, scene videos for effects) and design a dedicated visual encoder for this dual-stream setup. Trained entirely on synthetic data, our model generalizes effectively to real-world cinematic content and achieves strong performance on synthetic, real-world, and audio-only CASS benchmarks. Code and demo are available at \url{this https URL}.
Abstract (translated)
电影音频源分离（CASS）旨在将混合的电影音频分解为语音、音乐和音效，以支持配音和重制等应用。现有CASS方法仅依赖音频，忽略了电影固有的音视特性——声音常与视觉线索相关联。我们首次提出了音视联合的CASS框架（AV-CASS），利用视觉上下文提升分离质量。该方法将CASS建模为基于条件流匹配的条件生成问题，实现多模态音频源分离。针对缺乏带独立音轨的电影数据集问题，我们设计了训练数据合成流程，将野外采集的音频-视频流进行配对（如面部视频对应语音，场景视频对应音效），并为此双流架构设计了专用视觉编码器。模型完全基于合成数据训练，能有效泛化至真实电影内容，在合成、真实及纯音频CASS基准测试中均表现优异。代码及演示已公开于\url{此https链接}。
URL
https://arxiv.org/abs/2603.26113
PDF
-
CLAR: CIF-Localized Alignment for Retrieval-Augmented Speech LLM-Based Contextual ASR
2026-03-26 13:59:49Shangkun Huang, Huan Shen, Wei Zou, Yunzhang ChenAbstract
Speech LLM-based ASR often struggles with named entities and long-tail words due to strong internal language-model priors. Retrieval-augmented biasing can help, but its effectiveness depends on accurate hotword localization in full-utterance speech under weak supervision. We propose CLAR, a dual-encoder speech-text retriever that uses Continuous Integrate-and-Fire (CIF) to learn monotonic token-level alignments without timestamps. With length-aware localized matching, CLAR anchors short-entity acoustic cues and reduces representation dilution and attention drift. The retriever is trained with a multi-granularity objective combining global and local segment-level contrastive losses and a CIF quantity constraint. At inference, top-ranked hotwords are injected as contextual prompts for the Speech LLM, improving recognition without shallow fusion. Experiments show that CLAR significantly improves hotword retrieval and reduces both CER and B-WER against strong contextual ASR baselines.
Abstract (translated)
基于语音大语言模型（Speech LLM）的自动语音识别（ASR）系统，常因其内部强大的语言模型先验，在识别命名实体和长尾词汇时表现不佳。检索增强偏置（Retrieval-augmented biasing）虽能提供帮助，但其效果依赖于在弱监督条件下对整句语音中热词的精确定位。本文提出CLAR，一种双编码器语音-文本检索器，它利用连续积分放电（CIF）机制学习无时间戳的单调词级对齐。通过长度感知的局部化匹配，CLAR能够锚定短实体声学线索，并减少表征稀释与注意力漂移。该检索器通过结合全局与局部片段级对比损失以及CIF数量约束的多粒度目标进行训练。推理时，排名靠前的热词被注入作为语音大语言模型的上下文提示，从而在不采用浅层融合的情况下提升识别性能。实验表明，CLAR显著改善了热词检索效果，并在关键错误率（CER）和带偏置词错误率（B-WER）上均优于强上下文感知ASR基线。
URL
https://arxiv.org/abs/2603.25460
PDF
-
CoDeTT: A Context-Aware Decision Benchmark for Turn-Taking Evaluation
2026-03-26 13:26:41Huan Shen, Yingao Wang, Shangkun Huang, Wei Zou, Yunzhang ChenAbstract
Turn-taking modeling is fundamental to spoken dialogue systems, yet its evaluation remains fragmented and often limited to binary boundary detection under narrow interaction settings. Such protocols hinder systematic comparison and obscure model weaknesses across conversational conditions. We present CoDeTT, a context-aware decision benchmark for turn-taking evaluation. CoDeTT formulates turn-taking as a structured decision problem and constructs a multi-scenario dataset with fine-grained decision categories and controlled context variations. Under a unified evaluation protocol, we assess representative existing models and observe substantial performance disparities across decision types and interaction scenarios. CoDeTT provides a standardized benchmark for systematic and context-aware evaluation of turn-taking systems. The benchmark dataset and evaluation toolkit are available at this https URL.
Abstract (translated)
轮次建模是口语对话系统的基础，但其评估方法仍较为零散，且常局限于狭窄交互场景下的二元边界检测。此类评估协议阻碍了系统性对比，并掩盖了模型在不同对话条件下的缺陷。本文提出CoDeTT，一个面向轮次评估的上下文感知决策基准。CoDeTT将轮次转化为结构化决策问题，并构建了包含细粒度决策类别与可控上下文变化的多场景数据集。在统一的评估协议下，我们测试了代表性现有模型，观察到不同决策类型与交互场景间存在显著性能差异。CoDeTT为轮次系统的系统性、上下文感知评估提供了标准化基准。该基准数据集与评估工具包可通过此https URL获取。
URL
https://arxiv.org/abs/2603.25434
PDF
-
Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition
2026-03-26 12:28:44Yuhang Dai, Haopeng Lin, Jiale Qian, Ruiqi Yan, Hao Meng, Hanke Xie, Hanlin Wen, Shunshun Yin, Ming Tao, Xie Chen, Lei Xie, Xinsheng WangAbstract
Large Audio-Language Models (LALMs) have demonstrated remarkable performance in end-to-end speaker diarization and recognition. However, their speaker discriminability remains limited due to the scarcity of large-scale conversational data and the absence of explicit speaker representation optimization. To address this, we propose GLSC-SDR, a paradigm that jointly trains speaker classification with diarization and recognition. We further introduce a Global-Local Speaker Classification strategy, which uses clustered speakers as global labels and re-encoded intra-cluster speakers as local labels. This hierarchical design enhances fine-grained speaker discrimination while preserving semantic transcription accuracy. Experiments on AliMeeting, AISHELL-4, and AMI-SDM demonstrate that GLSC-SDR achieves competitive or superior performance compared to simulation-based and multi-encoder approaches, without relying on large-scale real conversational data.
Abstract (translated)
大型音频语言模型（LALMs）在端到端说话人分离与识别任务中展现出卓越性能。然而，由于大规模对话数据的稀缺以及缺乏显式说话人表征优化，其说话人区分能力仍较为有限。针对这一问题，我们提出了GLSC-SDR范式，通过联合训练说话人分类、分离与识别任务。进一步引入全局-局部说话人分类策略，利用聚类后的说话人作为全局标签，并对簇内说话人进行重编码作为局部标签。该分层设计在保持语义转写准确性的同时增强了细粒度说话人区分能力。在AliMeeting、AISHELL-4和AMI-SDM数据集上的实验表明，GLSC-SDR无需依赖大规模真实对话数据，即可达到与基于模拟方法及多编码器方法相当或更优的性能。
URL
https://arxiv.org/abs/2603.25377
PDF
-
YingMusic-Singer: Controllable Singing Voice Synthesis with Flexible Lyric Manipulation and Annotation-free Melody Guidance
2026-03-25 17:58:11Chunbo Hao, Junjie Zheng, Guobin Ma, Yuepeng Jiang, Huakang Chen, Wenjie Tian, Gongyu Chen, Zihao Chen, Lei XieAbstract
Regenerating singing voices with altered lyrics while preserving melody consistency remains challenging, as existing methods either offer limited controllability or require laborious manual alignment. We propose YingMusic-Singer, a fully diffusion-based model enabling melody-controllable singing voice synthesis with flexible lyric manipulation. The model takes three inputs: an optional timbre reference, a melody-providing singing clip, and modified lyrics, without manual alignment. Trained with curriculum learning and Group Relative Policy Optimization, YingMusic-Singer achieves stronger melody preservation and lyric adherence than Vevo2, the most comparable baseline supporting melody control without manual alignment. We also introduce LyricEditBench, the first benchmark for melody-preserving lyric modification evaluation. The code, weights, benchmark, and demos are publicly available at this https URL.
Abstract (translated)
在保持旋律一致性的同时重新生成歌词已改唱的歌声仍具挑战性，因为现有方法要么可控性有限，要么需要繁琐的手动对齐。我们提出了YingMusic-Singer，一个完全基于扩散的模型，能够实现旋律可控的歌声合成与灵活的歌词编辑。该模型接受三个输入：可选的音色参考、提供旋律的歌声片段以及修改后的歌词，无需手动对齐。通过课程学习与组相对策略优化进行训练，YingMusic-Singer在旋律保持和歌词贴合度上均优于Vevo2——这是目前唯一支持无需手动对齐的旋律可控基线模型。我们还引入了LyricEditBench，这是首个用于评估旋律保持型歌词修改的基准测试。代码、权重、基准测试及演示均已在以下链接公开。
URL
https://arxiv.org/abs/2603.24589
PDF
-
Iterate to Differentiate: Enhancing Discriminability and Reliability in Zero-Shot TTS Evaluation
2026-03-25 15:36:43Shengfan Shen, Di Wu, Xingchen Song, Dinghao Zhou, Liumeng Xue, Meng Meng, Jian Luan, Shuai WangAbstract
Reliable evaluation of modern zero-shot text-to-speech (TTS) models remains challenging. Subjective tests are costly and hard to reproduce, while objective metrics often saturate, failing to distinguish SOTA systems. To address this, we propose Iterate to Differentiate (I2D), an evaluation framework that recursively synthesizes speech using the model's own outputs as references. Higher-quality models exhibit greater resilience to the distributional shift induced by iterative synthesis, resulting in slower performance degradation. I2D exploits this differential degradation to amplify performance gaps and reveal robustness. By aggregating objective metrics across iterations, I2D improves discriminability and alignment with human judgments, increasing system-level SRCC from 0.118 to 0.464 for UTMOSv2. Experiments on 11 models across Chinese, English, and emotion datasets demonstrate that I2D enables more reliable automated evaluation for zero-shot TTS.
Abstract (translated)
对现代零样本文本转语音（TTS）模型的可靠评估仍具有挑战性。主观测试成本高昂且难以复现，而客观指标常趋于饱和，无法区分最先进系统。为此，我们提出“迭代区分法”（I2D）评估框架，该框架递归使用模型自身输出作为参考来合成语音。更高质量的模型对迭代合成引起的分布偏移表现出更强的鲁棒性，导致性能下降更缓慢。I2D利用这种差异化的性能退化来放大系统间差距并揭示鲁棒性差异。通过聚合迭代间的客观指标，I2D提升了判别能力并与人类判断的一致性，使UTMOSv2的系统级SRCC从0.118提升至0.464。在中文、英文及情感数据集上对11个模型的实验表明，I2D能够实现更可靠的零样本文本转语音自动化评估。
URL
https://arxiv.org/abs/2603.24430
PDF
