arXiv_SD
-
Semantic-Aware Interruption Detection in Spoken Dialogue Systems: Benchmark, Metric, and Model
2026-03-25 10:11:33Kangxiang Xia, Bingshen Mu, Xian Shi, Jin Xu, Lei XieAbstract
Achieving natural full-duplex interaction in spoken dialogue systems (SDS) remains a challenge due to the difficulty of accurately detecting user interruptions. Current solutions are polarized between "trigger-happy" VAD-based methods that misinterpret backchannels and robust end-to-end models that exhibit unacceptable response delays. Moreover, the absence of real-world benchmarks and holistic metrics hinders progress in the field. This paper presents a comprehensive frame-work to overcome these limitations. We first introduce SID-Bench, the first benchmark for semantic-aware interruption detection built entirely from real-world human dialogues. To provide a rigorous assessment of the responsiveness-robustness trade-off, we propose the Average Penalty Time (APT) metric, which assigns a temporal cost to both false alarms and late responses. Building on this framework, we design an LLM-based detection model optimized through a novel training paradigm to capture subtle semantic cues of intent. Experimental results show that our model significantly outperforms mainstream baselines, achieving a nearly threefold reduction in APT. By successfully resolving the long-standing tension between speed and stability, our work establishes a new state-of-the-art for intelligent interruption handling in SDS. To facilitate future research, SID-Bench and the associated code are available at: this https URL.
Abstract (translated)
在口语对话系统中实现自然的全双工交互仍是一项挑战，这源于准确检测用户中断的困难。现有方案在"过于敏感"的基于语音活动检测的方法（易误判反馈性插话）与响应延迟不可接受的端到端模型之间两极分化。此外，真实场景基准和整体性指标的缺失也阻碍了该领域发展。本文提出一个综合性框架以突破这些限制：首先引入SID-Bench——首个完全基于真实人类对话构建的语义感知中断检测基准；为严谨评估响应速度与鲁棒性的权衡，提出平均惩罚时间（APT）指标，该指标为误报和延迟响应均赋予时间代价；基于此框架，我们设计了一种通过新型训练范式优化、旨在捕捉细微语义意图线索的基于大语言模型的检测模型。实验结果表明，本模型显著优于主流基线，将APT降低近三倍。通过成功解决速度与稳定性之间的长期矛盾，本研究为口语对话系统中的智能中断处理建立了新的最先进水平。为促进后续研究，SID-Bench及相关代码已公开于：this https URL。
URL
https://arxiv.org/abs/2603.24144
PDF
-
ACAVCaps: Enabling large-scale training for fine-grained and diverse audio understanding
2026-03-25 07:49:39Yadong Niu, Tianzi Wang, Heinrich Dinkel, Xingwei Sun, Jiahao Zhou, Gang Li, Jizhong Liu, Junbo Zhang, Jian LuanAbstract
General audio understanding is a fundamental goal for large audio-language models, with audio captioning serving as a cornerstone task for their development. However, progress in this domain is hindered by existing datasets, which lack the scale and descriptive granularity required to train truly versatile models. To address this gap, we introduce ACAVCaps, a new large-scale, fine-grained, and multi-faceted audio captioning dataset. Derived from the ACAV100M collection, ACAVCaps is constructed using a multi-expert pipeline that analyzes audio from diverse perspectives-including speech, music, and acoustic properties-which are then synthesized into rich, detailed descriptions by a large language model. Experimental results demonstrate that models pre-trained on ACAVCaps exhibit substantially stronger generalization capabilities on various downstream tasks compared to those trained on other leading captioning datasets. The dataset is available at this https URL.
Abstract (translated)
通用音频理解是大型音频-语言模型的基本目标，而音频描述任务则是其发展的基石。然而，该领域的进展受到现有数据集的制约——这些数据集缺乏训练真正通用模型所需的规模和描述粒度。为填补这一空白，我们推出了ACAVCaps，这是一个全新的大规模、细粒度、多角度的音频描述数据集。该数据集源自ACAV100M集合，通过多专家流水线构建：该流水线从语音、音乐及声学特性等多个维度分析音频，再由大语言模型综合生成丰富详尽的描述。实验结果表明，在ACAVCaps上预训练的模型，相较于使用其他领先描述数据集训练的模型，在各类下游任务中展现出显著更强的泛化能力。数据集已通过此链接公开。
URL
https://arxiv.org/abs/2603.24038
PDF
-
Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning
2026-03-25 00:40:02Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen, Noboru Harada, Nobutaka OnoAbstract
Since the introduction of Masked Autoencoders, various improvements to masking techniques have been explored. In this paper, we rethink masking strategies for audio representation learning using masked prediction-based self-supervised learning (SSL) on general audio spectrograms. While recent informed masking techniques have attracted attention, we observe that they incur substantial computational overhead. Motivated by this observation, we propose dispersion-weighted masking (DWM), a lightweight masking strategy that leverages the spectral sparsity inherent in the frequency structure of audio content. Our experiments show that inverse block masking, commonly used in recent SSL frameworks, improves audio event understanding performance while introducing a trade-off in generalization. The proposed DWM alleviates these limitations and computational complexity, leading to consistent performance improvements. This work provides practical guidance on masking strategy design for masked prediction-based audio representation learning.
Abstract (translated)
自掩码自编码器提出以来，研究者们对掩码技术进行了多种改进。本文针对通用音频频谱图，重新思考基于掩码预测的自监督学习（SSL）中的掩码策略。尽管近期信息感知掩码技术备受关注，但我们发现其计算开销显著。基于此观察，本文提出分散加权掩码（DWM）——一种轻量级掩码策略，利用音频内容频率结构中固有的频谱稀疏性。实验表明，近期SSL框架中常用的反向块掩码虽能提升音频事件理解性能，却以牺牲泛化能力为代价。所提出的DWM缓解了这些限制并降低计算复杂度，实现了稳定的性能提升。本研究为基于掩码预测的音频表征学习中的掩码策略设计提供了实用指导。
URL
https://arxiv.org/abs/2603.23810
PDF
-
Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
2026-03-24 21:19:45Jakob Kienegger, Timo GerkmannAbstract
Deep spatially selective filters achieve high-quality enhancement with real-time capable architectures for stationary speakers of known directions. To retain this level of performance in dynamic scenarios when only the speakers' initial directions are given, accurate, yet computationally lightweight tracking algorithms become necessary. Assuming a frame-wise causal processing style, temporal feedback allows for leveraging the enhanced speech signal to improve tracking performance. In this work, we investigate strategies to incorporate the enhanced signal into lightweight tracking algorithms and autoregressively guide deep spatial filters. Our proposed Bayesian tracking algorithms are compatible with arbitrary deep spatial filters. To increase the realism of simulated trajectories during development and evaluation, we propose and publish a novel dataset based on the social force model. Results validate that the autoregressive incorporation significantly improves the accuracy of our Bayesian trackers, resulting in superior enhancement with none or only negligibly increased computational overhead. Real-world recordings complement these findings and demonstrate the generalizability of our methods to unseen, challenging acoustic conditions.
Abstract (translated)
深度空间选择性滤波器通过适用于静态声源且已知方向的实时架构实现高质量增强。当仅已知声源初始方向时，为在动态场景中保持同等性能，需要精确且计算轻量的跟踪算法。假设采用逐帧因果处理模式，时间反馈可利用增强后的语音信号来提升跟踪性能。本研究探索了将增强信号融入轻量跟踪算法并自回归引导深度空间滤波器的策略。所提出的贝叶斯跟踪算法兼容任意深度空间滤波器。为提升开发与评估中模拟轨迹的真实性，我们基于社会力模型提出并发布了一个新数据集。结果验证了自回归融合显著提高了贝叶斯跟踪器的精度，从而在几乎不增加计算开销的情况下实现更优的增强效果。真实世界录音补充了这些发现，并展示了该方法在未见且具有挑战性的声学条件中的泛化能力。
URL
https://arxiv.org/abs/2603.23723
PDF
-
Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition
2026-03-24 19:27:02Lucas H. Ueda, Jo\~ao G. T. Lima, Paula D. P. CostaAbstract
Speech Emotion Recognition (SER) in real-world scenarios remains challenging due to severe class imbalance and the prevalence of spontaneous, natural speech. While recent approaches leverage self-supervised learning (SSL) representations and multimodal fusion of speech and text, most existing methods apply supervision only at the final classification layer, limiting the discriminative power of intermediate representations. In this work, we propose Crab (Contrastive Representation and Multimodal Aligned Bottleneck), a bimodal Cross-Modal Transformer architecture that integrates speech representations from WavLM and textual representations from RoBERTa, together with a novel \textit{Multi Layer Contrastive Supervision} (MLCS) strategy. MLCS injects multi-positive contrastive learning signals at multiple layers of the network, encouraging emotionally discriminative representations throughout the model without introducing additional parameters at inference time. To further address data imbalance, we adopt weighted cross-entropy during training. We evaluate the proposed approach on three benchmark datasets covering different degrees of emotional naturalness: IEMOCAP, MELD, and MSP-Podcast 2.0. Experimental results demonstrate that Crab consistently outperforms strong unimodal and multimodal baselines across all datasets, with particularly large gains under naturalistic and highly imbalanced conditions. These findings highlight the effectiveness of \textit{Multi Layer Contrastive Supervision} as a general and robust strategy for SER. Official implementation can be found in this https URL.
Abstract (translated)
在真实场景中，语音情感识别（SER）仍面临严重类别不平衡和自发自然语音普遍存在的挑战。尽管近期方法利用自监督学习（SSL）表示以及语音与文本的多模态融合，但现有方法大多仅在最终分类层施加监督，限制了中间表征的判别能力。本研究提出Crab（对比表征与多模态对齐瓶颈），一种双模态跨模态Transformer架构，集成来自WavLM的语音表征和来自RoBERTa的文本表征，并引入新颖的\textit{多层对比监督}（MLCS）策略。MLCS在网络多层注入多正例对比学习信号，促使模型各层生成具有情感判别力的表征，且推理时不引入额外参数。为应对数据不平衡，训练中采用加权交叉熵。我们在涵盖不同情感自然度程度的三个基准数据集IEMOCAP、MELD和MSP-Podcast 2.0上评估所提方法。实验结果表明，Crab在所有数据集上持续优于强单模态与多模态基线，尤其在自然且高度不平衡条件下提升显著。这些发现凸显了\textit{多层对比监督}作为SER通用鲁棒策略的有效性。官方实现代码见此https URL。
URL
https://arxiv.org/abs/2603.23673
PDF
-
Rewriting TTS Inference Economics: Lightning V2 on Tenstorrent Achieves 4x Lower Cost Than NVIDIA L40S
2026-03-24 13:02:58Ranjith M. S., Akshat Mandloi, Sudarshan KamathAbstract
Text-to-Speech (TTS) models are significantly more numerically fragile than Large Language Models (LLMs) due to their continuous waveform generation and perceptual sensitivity to small numerical perturbations. While aggressive precision reduction techniques such as BlockFloat8 (BFP8) and low-fidelity (LoFi) compute have been widely adopted in language models, applying similar strategies to TTS systems often results in audible artifacts, phase instability, and spectral distortion. In this work, we present Lightning V2, a production-grade TTS model co-optimized for Tenstorrent hardware. Through precision-aware architectural design and hardware-software co-optimization, we achieve over 95% LoFi computational fidelity and more than 80% BlockFloat8 deployment without measurable degradation in audio quality. Leveraging Tenstorrent's Network-on-Chip (NoC), distributed SRAM, and deterministic execution model, we reduce memory movement and redundant weight fetches, enabling efficient low-precision inference. Compared to an NVIDIA L40S baseline, Lightning V2 achieves approximately 4x lower on-prem accelerator cost at equivalent throughput, while maintaining production audio fidelity. Our results demonstrate that precision co-design, combined with hardware-aware optimization, can fundamentally reshape the economics of real-time speech inference.
Abstract (translated)
文本转语音（TTS）模型由于其连续的波形生成特性以及对微小数值扰动的感知敏感度，在数值上显著比大语言模型（LLMs）更为脆弱。尽管诸如BlockFloat8（BFP8）和低精度（LoFi）计算等激进的精度缩减技术已在语言模型中广泛应用，但将类似策略应用于TTS系统往往会导致可听见的伪影、相位不稳定和频谱失真。在本研究中，我们推出了Lightning V2——一个为Tenstorrent硬件协同优化的生产级TTS模型。通过精度感知的架构设计与软硬件协同优化，我们实现了超过95%的LoFi计算保真度以及超过80%的BlockFloat8部署，且音频质量无 measurable 降级。借助Tenstorrent的片上网络（NoC）、分布式SRAM和确定性执行模型，我们减少了内存移动和冗余权重获取，从而实现了高效的低精度推理。相较于NVIDIA L40S基线，Lightning V2在相同吞吐量下将本地加速器成本降低了约4倍，同时保持了生产级音频保真度。我们的成果表明，精度协同设计结合硬件感知优化，能够从根本上改变实时语音推理的经济性。
URL
https://arxiv.org/abs/2604.03279
PDF
-
The Interspeech 2026 Audio Encoder Capability Challenge for Large Audio Language Models
2026-03-24 02:47:24Heinrich Dinkel, Jiahao Zhou, Guanbo Wang, Yadong Niu, Junbo Zhang, Yufeng Hao, Ying Liu, Ke Li, Wenwu Wang, Zhiyong Wu, Jian LuanAbstract
This paper presents the Interspeech 2026 Audio Encoder Capability Challenge, a benchmark specifically designed to evaluate and advance the performance of pre-trained audio encoders as front-end modules for Large Audio Language Models (LALMs). While LALMs have shown remarkable understanding of complex acoustic scenes, their performance depends on the semantic richness of the underlying audio encoder representations. This challenge addresses the integration gap by providing a unified generative evaluation framework, XARES-LLM, which assesses submitted encoders across a diverse suite of downstream classification and generation tasks. By decoupling encoder development from LLM fine-tuning, the challenge establishes a standardized protocol for general-purpose audio representations that can effectively be used for the next generation of multimodal language models.
Abstract (translated)
本文介绍了Interspeech 2026音频编码器能力挑战赛，这是一项专门为评估和提升预训练音频编码器作为大型音频语言模型前端模块的性能而设计的基准测试。尽管大型音频语言模型已展现出对复杂声学场景的卓越理解能力，但其性能仍取决于底层音频编码器表征的语义丰富度。本挑战赛通过提供统一的生成式评估框架XARES-LLM，解决了这一集成难题——该框架通过多样化的下游分类与生成任务对提交的编码器进行综合评估。通过将编码器开发与大型语言模型微调解耦，本挑战赛为通用音频表征建立了标准化协议，这些表征可有效应用于下一代多模态语言模型。
URL
https://arxiv.org/abs/2603.22728
PDF
-
Velocity Potential Neural Field for Efficient Ambisonics Impulse Response Modeling
2026-03-23 21:28:28Yoshiki Masuyama, Francois G. Germain, Gordon Wichern, Chiori Hori, Jonathan Le RouxAbstract
First-order Ambisonics (FOA) is a standard spatial audio format based on spherical harmonic decomposition. Its zeroth- and first-order components capture the sound pressure and particle velocity, respectively. Recently, physics-informed neural networks have been applied to the spatial interpolation of FOA signals, regularizing the network outputs based on soft penalty terms derived from physical principles, e.g., the linearized momentum equation. In this paper, we reformulate the task so that the predicted FOA signal automatically satisfies the linearized momentum equation. Our network approximates a scalar function called velocity potential, rather than the FOA signal itself. Then, the FOA signal can be readily recovered through the partial derivatives of the velocity potential with respect to the network inputs (i.e., time and microphone position) according to physics of sound propagation. By deriving the four channels of FOA from the single-channel velocity potential, the reconstructed signal follows the physical principle at any time and position by construction. Experimental results on room impulse response reconstruction confirm the effectiveness of the proposed framework.
Abstract (translated)
一阶Ambisonics（FOA）是一种基于球谐分解的标准空间音频格式，其零阶与一阶分量分别捕捉声压与质点速度。近期，物理信息神经网络已被应用于FOA信号的空间插值，通过基于物理原理（如线性化动量方程）推导的软惩罚项对网络输出进行正则化。本文重构了该任务，使预测的FOA信号自动满足线性化动量方程。我们的网络不直接预测FOA信号本身，而是近似一个称为速度势函数的标量函数。随后，根据声传播物理原理，FOA信号可通过速度势函数关于网络输入（即时间与麦克风位置）的偏导数直接恢复。通过从单通道速度势函数推导出FOA的四通道信号，重构的信号在构造上即遵循任意时间与位置的物理原理。关于房间脉冲响应重建的实验结果验证了所提框架的有效性。
URL
https://arxiv.org/abs/2603.22589
PDF
-
MSP-Conversation: A Corpus for Naturalistic, Time-Continuous Emotion Recognition
2026-03-23 19:58:17Luz Martinez-Lucas, Pravin Mote, Abinay Reddy Naini, Mohammed Abdelwahab, Carlos BussoAbstract
Affective computing aims to understand and model human emotions for computational systems. Within this field, speech emotion recognition (SER) focuses on predicting emotions conveyed through speech. While early SER systems relied on limited datasets and traditional machine learning models, recent deep learning approaches demand largescale, naturalistic emotional corpora. To address this need, we introduce the MSP-Conversation corpus: a dataset of more than 70 hours of conversational audio with time-continuous emotional annotations and detailed speaker diarizations. The time-continuous annotations capture the dynamic and contextdependent nature of emotional expression. The annotations in the corpus include fine-grained temporal traces of valence, arousal, and dominance. The audio data is sourced from publicly available podcasts and overlaps with a subset of the isolated speaking turns in the MSP-Podcast corpus to facilitate direct comparisons between annotation methods (i.e., in-context versus out-of-context annotations). The paper outlines the development of the corpus, annotation methodology, analyses of the annotations, and baseline SER experiments, establishing the MSP-Conversation corpus as a valuable resource for advancing research in dynamic SER in naturalistic settings.
Abstract (translated)
情感计算旨在为计算系统理解和建模人类情感。在这一领域中，语音情感识别（SER）专注于预测语音中传达的情感。早期的SER系统依赖有限的数据集和传统机器学习模型，而近年来的深度学习方法则需要大规模、自然化的情感语料库。为满足这一需求，我们推出了MSP-Conversation语料库：该数据集包含超过70小时的对话式音频，具有时间连续情感标注和详细的说话人分离标注。时间连续标注能够捕捉情感表达的动态性和情境依赖性。语料库中的标注包含效价、唤醒度和支配度的细粒度时序轨迹。音频数据源自公开可用的播客节目，并与MSP-Podcast语料库中部分孤立的说话段重叠，以促进不同标注方法（即情境内标注与情境外标注）之间的直接对比。本文概述了该语料库的开发过程、标注方法、标注分析以及基线SER实验，确立了MSP-Conversation语料库作为推动自然场景下动态语音情感识别研究的重要资源。
URL
https://arxiv.org/abs/2603.22536
PDF
-
SelfTTS: cross-speaker style transfer through explicit embedding disentanglement and self-refinement using self-augmentation
2026-03-23 17:45:03Lucas H. Ueda, Jo\~ao G. T. Lima, Pedro R. Corr\^ea, Fl\'avio O. Sim\~oes, M\'ario U. Neto, Paula D. P. CostaAbstract
This paper presents SelfTTS, a text-to-speech (TTS) model designed for cross-speaker style transfer that eliminates the need for external pre-trained speaker or emotion encoders. The architecture achieves emotional expressivity in neutral speakers through an explicit disentanglement strategy utilizing Gradient Reversal Layers (GRL) combined with cosine similarity loss to decouple speaker and emotion information. We introduce Multi Positive Contrastive Learning (MPCL) to induce clustered representations of speaker and emotion embeddings based on their respective labels. Furthermore, SelfTTS employs a self-refinement strategy via Self-Augmentation, exploiting the model's voice conversion capabilities to enhance the naturalness of synthesized speech. Experimental results demonstrate that SelfTTS achieves superior emotional naturalness (eMOS) and robust stability in target timbre and emotion compared to state-of-the-art baselines.
Abstract (translated)
本文提出SelfTTS，一种旨在实现跨说话人风格迁移的文本转语音（TTS）模型，该模型无需依赖外部预训练的说话人或情感编码器。其架构通过结合梯度反转层（GRL）与余弦相似度损失的显式解耦策略，将说话人与情感信息分离，从而在中性说话人中实现情感表达性。我们引入多正例对比学习（MPCL），基于说话人与情感的标签诱导其嵌入表示的聚类。此外，SelfTTS采用通过自增强（Self-Augmentation）实现的自我精炼策略，利用模型的语音转换能力提升合成语音的自然度。实验结果表明，与最先进的基线方法相比，SelfTTS在目标音色与情感控制上具有更优的情感自然度（eMOS）和稳健稳定性。
URL
https://arxiv.org/abs/2603.22252
PDF
-
AnimalCLAP: Taxonomy-Aware Language-Audio Pretraining for Species Recognition and Trait Inference
2026-03-23 14:53:02Risa Shinoda, Kaede Shiohara, Nakamasa Inoue, Hiroaki Santo, Fumio OkuraAbstract
Animal vocalizations provide crucial insights for wildlife assessment, particularly in complex environments such as forests, aiding species identification and ecological monitoring. Recent advances in deep learning have enabled automatic species classification from their vocalizations. However, classifying species unseen during training remains challenging. To address this limitation, we introduce AnimalCLAP, a taxonomy-aware language-audio framework comprising a new dataset and model that incorporate hierarchical biological information. Specifically, our vocalization dataset consists of 4,225 hours of recordings covering 6,823 species, annotated with 22 ecological traits. The AnimalCLAP model is trained on this dataset to align audio and textual representations using taxonomic structures, improving the recognition of unseen species. We demonstrate that our proposed model effectively infers ecological and biological attributes of species directly from their vocalizations, achieving superior performance compared to CLAP. Our dataset, code, and models will be publicly available at this https URL.
Abstract (translated)
动物发声为野生动物评估提供了关键见解，尤其在森林等复杂环境中，有助于物种识别和生态监测。近年来深度学习的进展已实现基于发声的自动物种分类。然而，对训练中未出现过的物种进行分类仍具挑战性。针对这一局限，我们提出了AnimalCLAP——一个融合分层生物信息的分类学感知音文框架，包含新数据集与模型。具体而言，我们的发声数据集包含4,225小时录音，覆盖6,823个物种，并标注了22项生态特征。AnimalCLAP模型基于该数据集训练，通过分类学结构对齐音频与文本表征，提升了未见过物种的识别能力。研究表明，我们提出的模型能直接从动物发声推断物种的生态与生物属性，性能优于CLAP。数据集、代码及模型将在此网址公开。
URL
https://arxiv.org/abs/2603.22053
PDF
-
LipsAM: Lipschitz-Continuous Amplitude Modifier for Audio Signal Processing and its Application to Plug-and-Play Dereverberation
Abstract
The robustness of deep neural networks (DNNs) can be certified through their Lipschitz continuity, which has made the construction of Lipschitz-continuous DNNs an active research field. However, DNNs for audio processing have not been a major focus due to their poor compatibility with existing results. In this paper, we consider the amplitude modifier (AM), a popular architecture for handling audio signals, and propose its Lipschitz-continuous variants, which we refer to as LipsAM. We prove a sufficient condition for an AM to be Lipschitz continuous and propose two architectures as examples of LipsAM. The proposed architectures were applied to a Plug-and-Play algorithm for speech dereverberation, and their improved stability is demonstrated through numerical experiments.
Abstract (translated)
深度神经网络（DNNs）的鲁棒性可通过其利普希茨连续性进行认证，这使得构建利普希茨连续的DNN成为一个活跃的研究领域。然而，由于与现有成果兼容性较差，音频处理领域的DNN尚未成为主要研究方向。本文针对处理音频信号的流行架构——振幅调制器（AM），提出了其利普希茨连续变体，称之为LipsAM。我们证明了AM满足利普希茨连续的充分条件，并提出了两种作为LipsAM实例的架构。所提出的架构被应用于语音去混响的即插即用算法中，并通过数值实验验证了其改进后的稳定性。
URL
https://arxiv.org/abs/2603.21684
PDF
-
Enterprise Sales Copilot: Enabling Real-Time AI Support with Automatic Information Retrieval in Live Sales Calls
2026-03-22 21:54:51Jielin Qiu, Liangwei Yang, Ming Zhu, Wenting Zhao, Zhiwei Liu, Juntao Tan, Zixiang Chen, Roshan Ram, Akshara Prabhakar, Rithesh Murthy, Shelby Heinecke, Caiming Xiong, Silvio Savarese, Huan WangAbstract
During live sales calls, customers frequently ask detailed product questions that require representatives to manually search internal databases and CRM systems. This process typically takes 25-65 seconds per query, creating awkward pauses that hurt customer experience and reduce sales efficiency. We present SalesCopilot, a real-time AI-powered assistant that eliminates this bottleneck by automatically detecting customer questions, retrieving relevant information from the product database, and displaying concise answers on the representative's dashboard in seconds. The system integrates streaming speech-to-text transcription, large language model (LLM)-based question detection, and retrieval-augmented generation (RAG) over a structured product database into a unified real-time pipeline. We demonstrate SalesCopilot on an insurance sales scenario with 50 products spanning 10 categories (2,490 FAQs, 290 coverage details, and 162 pricing tiers). In our benchmark evaluation, SalesCopilot achieves a measured mean response time of 2.8 seconds with 100% question detection rate, representing a 14xspeedup compared to manual CRM search in an internal study. The system is domain-agnostic and can be adapted to any enterprise sales domain by replacing the product database.
Abstract (translated)
在实时销售通话中，客户常会提出详细的产品问题，这要求销售代表手动查询内部数据库和CRM系统。此过程通常每项查询需耗时25-65秒，造成尴尬的停顿，损害客户体验并降低销售效率。我们推出了SalesCopilot——一款实时AI助手，通过自动检测客户问题、从产品数据库检索相关信息，并在数秒内将简明答案呈现在代表仪表板上，从而消除这一瓶颈。该系统将流式语音转文本转录、基于大语言模型（LLM）的问题检测，以及对结构化产品数据库的检索增强生成（RAG）整合为统一的实时流水线。我们在一个涵盖50种产品、10个类别（2,490条常见问题、290项保障详情、162个定价层级）的保险销售场景中演示了SalesCopilot。在基准评估中，SalesCopilot实现了平均2.8秒的响应时间，问题检测率达100%，相比内部研究中手动CRM搜索提速14倍。该系统不限定领域，只需替换产品数据库即可适配任何企业销售领域。
URL
https://arxiv.org/abs/2603.21416
PDF
-
HELIX: Scaling Raw Audio Understanding with Hybrid Mamba-Attention Beyond the Quadratic Limit
Abstract
Audio representation learning typically evaluates design choices such as input frontend, sequence backbone, and sequence length in isolation. We show that these axes are coupled, and conclusions from one setting often do not transfer to others. We introduce HELIX, a controlled framework comparing pure Mamba, pure attention, and a minimal hybrid with a single attention bottleneck. All models are parameter-matched at about 8.3M parameters to isolate architectural effects. Across six datasets, we find that the preferred input representation depends on the backbone, and that attention hurts performance on short, stationary audio but becomes important at longer sequence lengths. On a 5-minute speaker identification task with 30,000 tokens, pure attention fails with out-of-memory errors, while HELIX closes an 11.5-point gap over pure Mamba.
Abstract (translated)
音频表示学习通常孤立地评估输入前端、序列主干和序列长度等设计选择。我们证明这些维度相互关联，且某一设置下的结论往往无法迁移至其他场景。我们引入HELIX——一个受控框架，用于比较纯Mamba、纯注意力机制以及仅含单一注意力瓶颈的最小混合模型。所有模型均进行参数匹配（约830万参数），以隔离架构影响。在六个数据集上的实验表明：首选输入表示取决于主干架构；注意力机制在短时平稳音频上会损害性能，但在更长序列长度下变得至关重要。在包含30,000个token的5分钟说话人识别任务中，纯注意力因内存不足而失败，而HELIX相较于纯Mamba缩小了11.5个百分点的性能差距。
URL
https://arxiv.org/abs/2603.21316
PDF
-
Emotion-Aware Quantization for Discrete Speech Representations: An Analysis of Emotion Preservation
2026-03-22 13:29:47Haoguang Zhou, Siyi Wang, Jingyao Wu, James Bailey, Ting DangAbstract
Modern speech systems increasingly use discretized self-supervised speech representations for compression and integration with token-based models, yet their impact on emotional information remains unclear. We study how residual vector quantization (RVQ) reshapes emotional information in discrete speech representations from both representation- and task-level perspectives. Our analysis shows that aggressive compression disproportionately degrades emotion, with uneven loss across emotion classes and model architectures. To address this, we introduce emotion-aware quantization using emotion-specific and emotion-biased codebooks, improving the preservation of both hard and soft emotion perception. We further propose Emo-Q, a lightweight routed quantization method that selects emotion-specialized codebooks, improving emotion recognition performance at lower bitrates. These results highlight the importance of emotion-aware discretization for robust affective speech processing.
Abstract (translated)
现代语音系统越来越多地采用离散化自监督语音表征以实现压缩并与基于令牌的模型集成，然而这些表征对情感信息的影响尚不明确。本研究从表征层面和任务层面探讨残差向量量化（RVQ）如何重塑离散语音表征中的情感信息。分析表明，激进的压缩会不成比例地损害情感信息，且不同情感类别和模型架构的损失程度不均。为应对此问题，我们引入了基于情感感知的量化方法，使用情感专属和情感偏置码本，从而改善硬性及软性情感感知的保留效果。进一步提出Emo-Q——一种轻量级路由量化方法，通过选择情感专业化码本，在更低码率下提升情感识别性能。这些结果凸显了情感感知离散化对于鲁棒性情感语音处理的重要性。
URL
https://arxiv.org/abs/2603.21224
PDF
-
ERM-MinMaxGAP: Benchmarking and Mitigating Gender Bias in Multilingual Multimodal Speech-LLM Emotion Recognition
2026-03-22 04:16:51Zi Haur Pang, Xiaoxue Gao, Tatsuya Kawahara, Nancy F. ChenAbstract
Speech emotion recognition (SER) systems can exhibit gender-related performance disparities, but how such bias manifests in multilingual speech LLMs across languages and modalities is unclear. We introduce a novel multilingual, multimodal benchmark built on MELD-ST, spanning English, Japanese, and German, to quantify language-specific SER performance and gender gaps. We find bias is strongly language-dependent, and multimodal fusion does not reliably improve fairness. To address these, we propose ERM-MinMaxGAP, a fairness-informed training objective, which augments empirical risk minimization (ERM) with a proposed adaptive fairness weight mechanism and a novel MinMaxGAP regularizer on the maximum male-female loss gap within each language and modality. Building upon the Qwen2-Audio backbone, our ERM-MinMaxGAP approach improves multilingual SER performance by 5.5% and 5.0% while reducing the overall gender bias gap by 0.1% and 1.4% in the unimodal and multimodal settings, respectively.
Abstract (translated)
语音情感识别（SER）系统可能表现出与性别相关的性能差异，但此类偏见在多语言语音大语言模型中如何跨语言和模态体现尚不明确。我们基于MELD-ST构建了一个新颖的多语言多模态基准，涵盖英语、日语和德语，以量化特定语言的SER性能及性别差距。研究发现偏见高度依赖语言，且多模态融合并不能可靠地提升公平性。针对这些问题，我们提出了ERM-MinMaxGAP——一种考虑公平性的训练目标，它在经验风险最小化（ERM）基础上增加了提出的自适应公平权重机制，以及一种针对每种语言和模态内男女损失最大差距的新型MinMaxGAP正则化器。基于Qwen2-Audio主干网络，我们的ERM-MinMaxGAP方法在单模态和多模态设置中分别将多语言SER性能提升5.5%和5.0%，同时将整体性别偏见差距降低0.1%和1.4%。
URL
https://arxiv.org/abs/2603.21050
PDF
-
The Binding Effect: Analyzing How Multi-Dimensional Cues Form Gender Bias in Instruction TTS
2026-03-21 10:17:13Kuan-Yu Chen, Yi-Cheng Lin, Po-Chung Hsieh, Huang-Cheng Chou, Chih-Fan Hsu, Jeng-Lin Li, Hung-yi Lee, Jian-Jiun DingAbstract
Current bias evaluations in Instruction Text-to-Speech (ITTS) often rely on univariate testing, overlooking the compositional structure of social cues. In this work, we investigate gender bias by modeling prompts as combinations of Social Status, Career stereotypes, and Persona descriptors. Analyzing open-source ITTS models, we uncover systematic interaction effects where social dimensions modulate one another, creating complex bias patterns missed by univariate baselines. Crucially, our findings indicate that these biases extend beyond surface-level artifacts, demonstrating strong associations with the semantic priors of pre-trained text encoders and the skewed distributions inherent in training data. We further demonstrate that generic diversity prompting is insufficient to override these entrenched patterns, underscoring the need for compositional analysis to diagnose latent risks in generative speech.
Abstract (translated)
当前指令文本转语音（ITTS）中的偏差评估通常依赖单变量测试，忽视了社会线索的组合结构。在本研究中，我们通过将提示词建模为社会地位、职业刻板印象与人格描述符的组合来探究性别偏差。分析开源ITTS模型后，我们发现了系统性交互效应——社会维度会相互调节，形成单变量基线方法所遗漏的复杂偏差模式。关键的是，我们的研究结果表明这些偏差不仅停留在表面特征，更与预训练文本编码器的语义先验以及训练数据中固有的分布偏差存在强关联。我们还证明，通用多样性提示不足以覆盖这些根深蒂固的模式，这凸显了采用组合分析来诊断生成式语音中潜在风险的必要性。
URL
https://arxiv.org/abs/2603.20743
PDF
-
End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
2026-03-20 18:05:23Philippe Gonzalez, Vera Margrethe Frederiksen, Torsten Dau, Tobias MayAbstract
A multi-task learning framework is proposed for optimizing a single deep neural network (DNN) for joint noise reduction (NR) and hearing loss compensation (HLC). A distinct training objective is defined for each task, and the DNN predicts two time-frequency masks. During inference, the amounts of NR and HLC can be adjusted independently by exponentiating each mask before combining them. In contrast to recent approaches that rely on training an auditory-model emulator to define a differentiable training objective, we propose an auditory model that is inherently differentiable, thus allowing end-to-end optimization. The audiogram is provided as an input to the DNN, thereby enabling listener-specific personalization without the need for retraining. Results show that the proposed approach not only allows adjusting the amounts of NR and HLC individually, but also improves objective metrics compared to optimizing a single training objective. It also outperforms a cascade of two DNNs that were separately trained for NR and HLC, and shows competitive HLC performance compared to a traditional hearing-aid prescription. To the best of our knowledge, this is the first study that uses an auditory model to train a single DNN for both NR and HLC across a wide range of listener profiles.
Abstract (translated)
本研究提出一种多任务学习框架，用于优化单个深度神经网络（DNN），以实现噪声抑制（NR）与听力损失补偿（HLC）的联合处理。为每个任务分别定义训练目标，该DNN预测两个时频掩码。在推理阶段，通过分别对每个掩码进行指数运算后再合并，可独立调整NR和HLC的强度。与近期依赖训练听觉模型模拟器来定义可微分训练目标的方法不同，我们提出了一种本身具有可微性的听觉模型，从而实现端到端优化。听力图作为输入提供给DNN，使系统能够实现听者个性化定制而无需重新训练。结果表明，所提方法不仅支持单独调节NR和HLC强度，且在客观指标上优于单一训练目标优化方案。该方法还超越了分别针对NR和HLC训练的两个级联DNN，并在HLC性能上与传统助听器验配方案具有竞争力。据我们所知，这是首个利用听觉模型训练单个DNN，以覆盖广泛听者画像同时实现NR和HLC的研究。
URL
https://arxiv.org/abs/2603.20387
PDF
-
Audio Avatar Fingerprinting: An Approach for Authorized Use of Voice Cloning in the Era of Synthetic Audio
Abstract
With the advancements in AI speech synthesis, it is easier than ever before to generate realistic audio in a target voice. One only needs a few seconds of reference audio from the target, quite literally putting words in the target person's mouth. This imposes a new set of forensics-related challenges on speech-based authentication systems, videoconferencing, and audio-visual broadcasting platforms, where we want to detect synthetic speech. At the same time, leveraging AI speech synthesis can enhance the different modes of communication through features such as low-bandwidth communication and audio enhancements - leading to ever-increasing legitimate use-cases of synthetic audio. In this case, we want to verify if the synthesized voice is actually spoken by the user. This will require a mechanism to verify whether a given synthetic audio is driven by an authorized identity, or not. We term this task audio avatar fingerprinting. As a step towards audio forensics in these new and emerging situations, we analyze and extend an off-the-shelf speaker verification model developed outside of forensics context for the task of fake speech detection and audio avatar fingerprinting, the first experimentation of its kind. Furthermore, we observe that no existing dataset allows for the novel task of verifying the authorized use of synthetic audio - a limitation which we address by introducing a new speech forensics dataset for this novel task.
Abstract (translated)
随着AI语音合成技术的进步，生成目标声音的真实音频变得前所未有的容易。只需几秒的目标参考音频，就能让目标人物说出特定话语。这给基于语音的身份验证系统、视频会议和音视频广播平台带来了全新的法证相关挑战——我们需要在这些场景中检测合成语音。与此同时，利用AI语音合成也能通过低带宽通信和音频增强等功能提升不同通信模式，导致合成音频的合法使用场景不断增加。在这种情况下，我们需要验证合成语音是否确实由用户发出，这就需要一种机制来判断给定合成音频是否由授权身份驱动。我们将此任务称为“音频化身指纹”。作为面向这些新兴场景的音频法证研究一步，我们分析并扩展了一个原本非法证领域的现成说话人验证模型，用于伪造语音检测和音频化身指纹任务——这是该领域的首次尝试。此外，我们观察到目前尚无数据集支持合成音频授权验证这一新型任务，为此我们专门构建了一个针对该任务的新语音法证数据集以解决这一局限。
URL
https://arxiv.org/abs/2603.20165
PDF
-
BioDCASE 2026 Challenge Baseline for Cross-Domain Mosquito Species Classification
2026-03-20 16:41:35Yuanbo Hou, Vanja Zdravkovic, Marianne Sinka, Yunpeng Li, Wenwu Wang, Mark D. Plumbley, Kathy Willis, Stephen RobertsAbstract
Mosquito-borne diseases affect more than one billion people each year and cause close to one million deaths. Traditional surveillance methods rely on traps and manual identification that are slow, labor-intensive, and difficult to scale. Audio-based mosquito monitoring offers a non-destructive, lower-cost, and more scalable complement to trap-based surveillance, but reliable species classification remains difficult under real-world recording conditions. Mosquito flight tones are narrow-band, often low in signal-to-noise ratio, and easily masked by background noise, and recordings for several epidemiologically relevant species remain limited, creating pronounced class imbalance. Variation across devices, environments, and collection protocols further increases the difficulty of robust classification. Such variation can cause models to rely on domain-specific recording artefacts rather than species-relevant acoustic cues, which makes transfer to new acquisition settings difficult. The BioDCASE 2026 Cross-Domain Mosquito Species Classification (CD-MSC) challenge is designed around this deployment problem by evaluating performance on both seen and unseen domains. This paper presents the official baseline system and evaluation pipeline as a simple, fully reproducible reference for the CD-MSC challenge task. The baseline uses log-mel features and a multitemporal resolution convolutional neural network (MTRCNN) with species and auxiliary domain outputs, together with complete training and test scripts. The baseline system performs strongly on seen domains but degrades markedly on unseen domains, showing that cross-domain generalisation, rather than within-domain recognition, is the central challenge for practical mosquito species classification from multi-source bioacoustic recordings.
Abstract (translated)
蚊媒疾病每年影响超过十亿人，并导致近一百万人死亡。传统监测方法依赖诱捕器和人工鉴定，过程缓慢、劳动密集且难以规模化。基于音频的蚊虫监测为诱捕监测提供了一种非侵入式、成本更低且更易扩展的补充方案，但在真实录音条件下实现可靠的物种分类仍十分困难。蚊虫飞行音调为窄带信号，通常信噪比较低，且易受背景噪声干扰。此外，多种与流行病学相关的蚊种录音数据仍然有限，导致明显的类别不平衡。设备、环境及采集协议间的差异进一步增加了稳健分类的难度。这种差异可能导致模型依赖特定领域的录音伪影而非物种相关的声学特征，从而难以迁移到新的采集场景中。 BioDCASE 2026跨域蚊种分类挑战赛（CD-MSC）正是围绕这一部署难题设计，通过评估模型在已见域和未见域上的性能来考察其泛化能力。本文介绍了官方基线系统和评估流程，作为CD-MSC任务的简单且完全可复现的参考方案。该基线采用对数梅尔特征及多时间分辨率卷积神经网络（MTRCNN），并引入物种与辅助域双输出结构，同时提供完整的训练与测试脚本。基线系统在已见域上表现良好，但在未见域上性能显著下降，这表明跨域泛化能力——而非域内识别能力——才是从多源生物声学录音中实现实用蚊种分类的核心挑战。
URL
https://arxiv.org/abs/2603.20118
PDF
