arXiv_SD
-
How Should We Evaluate Naturalness for Synthetic Speech with a Foreign Accent?
2026-10-09 08:01:49Kentaro Onda, Satoru Fukayama, Daisuke Saito, Nobuaki MinematsuAbstract
Recent advances in speech synthesis technology have enabled the generation of speech with diverse characteristics. One such characteristic is a foreign accent. In our preliminary experiments, we found that UTMOS, a widely used automatic speech quality assessment system, tended to assign lower scores to speech with stronger foreign accents, even though it is produced by a human speaker. This observation raised a question: How do humans evaluate the naturalness of foreign-accented speech? We conducted a large-scale subjective evaluation involving both native and non-native listeners under different evaluation instructions. The results showed that humans also considered accentedness when evaluating naturalness, but that defining naturalness in terms of human-likeness and selecting raters who have greater exposure to the target accent may enable more appropriate evaluation. This study aims to provide a starting point for future discussions on fairer and more appropriate evaluation of synthetic speech with a foreign accent.
Abstract (translated)
URL
https://arxiv.org/abs/2610.13175
PDF
-
LiveSynth: A Streaming Neural Synthesizer for Instrument Cloning and Text-to-Instrument
2026-10-09 06:05:45Kyungsu Kim, Yejin Kim, Kyogu LeeAbstract
Recently, generative neural synthesizers have enabled zero-shot instrument cloning from reference recordings and timbre generation from text prompts. However, existing systems generate complete audio segments offline, precluding their use as playable instruments. We present LiveSynth, a streaming neural synthesizer with flexible timbre control and interactive generation capabilities, including zero-shot instrument cloning and text-to-instrument synthesis. A feedback-free causal Transformer conditioned on MIDI events and a CLAP timbre embedding outputs continuous audio latents at 100 Hz, which a causal variational autoencoder decodes into audio. Frame-wise timbre conditioning enables continuous timbre morphing, while MIDI-conditioning dropout enables autonomous continuation from a performed prefix. On the NSynth and Slakh datasets, LiveSynth achieved timbre similarity and spectral reconstruction quality comparable to offline baselines. On an M3 MacBook Air, LiveSynth required 8.8 ms of computation per 10 ms audio frame (an RTF of 0.88) with 10 ms algorithmic latency, enabling real-time interaction with a performer.
Abstract (translated)
URL
https://arxiv.org/abs/2610.13054
PDF
-
E3-System: Expressive EmoData, Explicitly Controllable, Easy-to-Use TTS
2026-10-09 04:55:49Yiqiao Huang, Xiaolong Lin, Zhenpeng Zhan, Zhizheng WuAbstract
Recent zero-shot Text-to-Speech (TTS) systems can clone unseen speakers, yet controllable expressiveness remains limited by a gap between semantic intent and acoustic realization. Text descriptions offer a natural control interface but provide weak acoustic grounding, whereas speech style prompts provide concrete prosody but entangle style with the prompt speaker's timbre. To bridge this gap, we propose E3-System, a unified intent-to-speech framework for expressive zero-shot TTS that couples three requirements: Expressive style modeling, Explicit timbre-style control, and Easy-to-use natural-language interaction. Given a target speaker and either a natural-language style description or a speech style prompt, E3-System generates expressive speech while preserving the target timbre. The key idea is to close the full control loop: building a large expressive style space, grounding user intent into usable acoustic style references, and synthesizing speech with disentangled timbre and style. To realize this loop, EmoData provides a thousand-hour multilingual emotional speech dataset, Style-RAG grounds text descriptions into high-quality acoustic style prompts from these data, and Dual-TTS transfers the grounded style to arbitrary target speakers through VC-augmented timbre-style disentanglement. Experiments show that E3-System achieves strong speech quality, speaker preservation, and explicit expressive controllability.
Abstract (translated)
URL
https://arxiv.org/abs/2610.12993
PDF
-
Emo-BPO: Emotion Bidirectional Preference Optimization for Diffusion-based Emotional TTS
2026-10-09 03:56:01Jiacheng Shi, Hongfei Du, Xinyuan Song, Y. Alicia Hong, Yanfu Zhang, Ye GaoAbstract
Diffusion-based TTS has demonstrated strong capability in emotional speech generation. However, existing emotion alignment methods primarily reinforce target emotional expressions while overlooking modeling of competing emotional modes, limiting fine-grained controllability. This issue becomes more pronounced under classifier-free guidance (CFG), which relies on contrasting conditional trajectories to shape generation. We propose Emotion Bidirectional Preference Optimization (Emo-BPO), a simple effective framework that jointly learns emotion-aligned and emotion-contrastive score functions from reordered same-text emotional pairs. Our method requires neither auxiliary reward models nor additional annotations, and can be seamlessly integrated into existing diffusion-based TTS systems to improve emotional controllability and expressiveness.
Abstract (translated)
URL
https://arxiv.org/abs/2610.12958
PDF
-
Lightweight Systems for Spatial Semantic Segmentation of Sound Scenes
2026-10-08 20:19:46Mayank Mishra, Paul Magron, Romain SerizelAbstract
Spatial semantic segmentation of sound scenes (S5) jointly performs audio source separation and tagging on a multichannel audio mixture. Therefore, an S5 system can be decomposed into an audio tagging model and a separation model. The baseline S5 system in task~4 of the DCASE 2025 challenge uses M2D and ResUNet models for tagging and separation, respectively. These are large in overall size and therefore not appropriate for use in embedded devices. The main aim of this paper is to assess the performance loss in an S5 system when using lightweight tagging and separation models instead. To this end, we first compare the tagging performance of lightweight audio tagging models, namely PANNs and DyMNs, in terms of label prediction accuracy against the baseline M2D model. Secondly, we assess separation performance in terms of the scale-invariant signal-to-distortion ratio of the baseline ResUNet separator relative to that of a convolutional-based separator using SuDoRMRF. Finally, we combine the tagging and separation models across various configurations and assess the overall performance of the S5 system using the class-aware signal-to-distortion ratio.
Abstract (translated)
URL
https://arxiv.org/abs/2610.12693
PDF
-
Input Representation, Not Model Size or Pretraining Scale, Shapes Speaker Verification Robustness to Temporal Degradation
2026-10-08 13:42:20Erdem Baha Topbas, Srikanth Madikeri, Elisa Pellegrino, Volker DellwoAbstract
Speech remains intelligible to listeners under heavy temporal degradation, but whether speaker identity survives is largely unexplored. We evaluate six automatic speaker verification (ASV) models, trained on temporally intact speech, and tested under three degradations that remove different combinations of temporal cues. The models form three pairs, one per architecture family (convolutional, transformer, conformer), each pairing a waveform-input with a mel filterbank-input model. All models degrade, but robustness tracks input representation rather than pretraining scale or model size. In all nine family-by-condition comparisons the filterbank-input model degrades less, with eight of nine 95% confidence intervals excluding zero, even when it is smaller or has less pretraining than its waveform counterpart. In the controlled convolutional pair, where the models differ only in input representation, the filterbank-input model's equal error rate rises from 1.1% to 17.1% under the heaviest tested degradation, against 2.9% to29.7% for its waveform counterpart.
Abstract (translated)
URL
https://arxiv.org/abs/2610.12539
PDF
-
STEMMA: Song-to-Stem Multi-Audio Reasoning for Large Audio Language Models
2026-10-08 12:54:12Hoyeol Sohn, Wonil Kim, Keunhyoung Kim, Sangeun Kum, Taehyoung Kim, Dongjoo Moon, Theerasak Charoenchob, Teeratep Weerapang, Jongpil Lee, Juhan NamAbstract
Music understanding often requires comparing excerpts and reasoning about relationships among songs, sections, and stems. However, existing large audio-language models (LALMs) and music question-answering datasets typically operate on single recordings or compare independently sampled tracks with no known production relationship. We introduce STEMMA, a multi-audio music question-answering framework built around production provenance: whether excerpts originate from the same track or section, and which stems belong to which mixtures. Because such relations are sparse under conventional audio-first sampling, STEMMA adopts a relation-first construction strategy: it first specifies a target relation and then queries the catalog for excerpts that satisfy it and hard negatives that do not. Labels are determined directly from catalog provenance rather than generated by a language model from metadata. We build STEMMA-Bench for evaluation and a track-disjoint training set, STEMMA-Instruct. Fine-tuning two LALMs on STEMMA-Instruct improves multi-audio reasoning, with the largest gains on structural relations directly determined by the catalog, while preserving single-audio music understanding.
Abstract (translated)
URL
https://arxiv.org/abs/2610.11884
PDF
-
Structural and Representational Agency in LLM-Driven Timbre Generation and Interpolation
Abstract
Generative creative interfaces routinely report the user's position within a space of possibilities. Such displays imply that the quantities shown were measured from the artefact; in a growing class of systems they are instead predicted by a language model describing its own output, and the two are indistinguishable on screen. Most creative domains lack an agreed measurement against which such a self-report can be checked; audio admits one. In a measurement study of a working LLM-driven timbre interface, what a user can cause proves separable from what the interface reports. The first is exactly characterisable: the reachable set is a zonotope whose dimension increases by one per sound placed, so the space the model may act within is demonstrably the user's own. The second is inaccurate. The displayed position differs from measurement by approximately three times the width of that region, and the error has two independent sources: model-predicted coordinates, whose accuracy varies twofold between the two models evaluated, and the interpolation step, which arises from combining synthesiser parameters and is unaffected by model quality. End to end, 26 of 36 refinement instructions produced no perceptible change, and under the weaker model the facility performed no action while reporting convergence. Effective agency (what a user can cause) and epistemic agency (whether they can know it) are therefore separable and independently engineered: structurally enforced constraints survive model failure, whereas guarantees resting on the model's own report do not. The failure mode that remains is silence presented as success. We close with a short protocol for building and auditing such displays.
Abstract (translated)
URL
https://arxiv.org/abs/2610.12532
PDF
-
Self-Supervised Speech Representations for Cross-Speaker Dysarthria Detection During Awake Craniotomy
2026-10-08 12:25:15Kanthila Chinmayi (IRDL, LaTIM), Abdallah Nassib (LARIS), Misy Harrison (LaTIM), Panheleux Celine (LaTIM, CHU - BREST), Saliou Vanessa (CHU - BREST), Seizeur Romuald (LaTIM), Dardenne Guillaume (LaTIM)Abstract
Detecting intra-operative speech impairment during awake craniotomy is essential for preserving language function. However, automated detection remains challenging because operating-room recordings contain substantial acoustic interference, clinically relevant speech events are rare, and available cohorts are small and heterogeneous across speakers. This study presents a systematic component-wise evaluation of a pipeline for distinguishing dysarthric from no-trouble speech in the DATABRASE corpus of awake-craniotomy recordings. The pipeline incorporates speaker diarization to isolate patient speech, a multi-view representation combining handcrafted acoustic descriptors with multilayer wav2vec 2.0 embeddings, speaker-conditional normalization and transferability-based feature selection to improve cross-speaker robustness, and a cascaded classifier comprising a gradient-boosted first stage and a neural second stage. Evaluation was conducted under strict speaker-independent conditions using leave-one-speaker-out cross-validation. The results show that cross-speaker performance is influenced more strongly by the speech representation than by classifier choice. The AUCs of three classifiers differed by no more than 4.7%, whereas replacing conventional acoustic descriptors with the multilayer self-supervised representation produced AUC improvements of 18.2%-26.1%. Diarization-conditioned feature extraction and the proposed classifier cascade provided additional consistent gains. These findings indicate that reliable patient-specific speech isolation and strong pretrained representations are more important than increased classifier complexity in low-resource intra-operative settings. They also quantify the potential performance gains that may be achieved through patient-specific preoperative calibration.
Abstract (translated)
URL
https://arxiv.org/abs/2610.11825
PDF
-
Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation
Abstract
Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has achieved notable success with static sound sources, real-world scenarios often involve moving targets, giving rise to the more challenging problem of \textbf{Dynamic Audio-Visual Navigation (D-AVN)}. Existing approaches largely rely on semantic auditory features, which limits generalization to unheard sound categories and hinders fine-grained spatial reasoning. To address these challenges, we propose a novel \textbf{Two-Stage Multi-modal Learning} framework for D-AVN. In the Physical Alignment stage, the proposed \textbf{Intensity-Calibrated Geometry Encoder (ICGE)} performs physics-informed alignment of sound intensity with visual geometry to capture category-agnostic spatial correspondence. In the Scene Aggregation stage, a Transformer-based \textbf{Cross-Modal Scene Aggregator (CMSA)} dynamically integrates multi-modal features to construct a robust state representation for policy learning. Extensive experiments on Replica and Matterport3D demonstrate that our method achieves state-of-the-art success rates, while significantly improving generalization and navigation efficiency in complex environments and under unheard sound scenarios.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14077
PDF
-
Reference-Free Singing Pitch Correction via Music-Constrained Sequence Editing
2026-10-08 08:55:19Biao Dong, Jiajun Li, Binzhen Zhu, Mingwei Yi, Tong Liu, Yuanhao Zhang, Jiqing Han, Yongjun HeAbstract
Existing singing pitch correction approaches rely on target melodies or accompaniment tracks, which may be unavailable in practice. We formulate reference-free singing pitch correction as a music-constrained sequence editing task that determines whether and how each note should be corrected from the input performance alone. A pretrained symbolic music encoder with lightweight singing-domain adapters produces contextual representations of the singing MIDI. Based on these representations, two lightweight correction heads jointly model correction necessity and signed pitch modification through a factorized pitch-editing distribution. An input-dependent tonal prior derived from the estimated key distribution then reranks the candidate offsets, favoring tonally compatible corrections without target melodies or ground-truth key annotations. Experiments on real paired amateur and professional singing recordings show that the method improves note-level pitch accuracy from 73.25 to 83.43, outperforming a context-based baseline by 3.99 percentage points while balancing error correction and preservation of correctly performed notes.
Abstract (translated)
URL
https://arxiv.org/abs/2610.11524
PDF
-
Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS
2026-10-08 07:59:57Junchuan Zhao, Chenglin Xu, Wei Zeng, Haoyang Li, Yiwen Guo, Ye WangAbstract
Instruction-based text-to-speech (TTS) offers control over voice characteristics and speech expression through interfaces including voice cloning and text-based voice design. Voice cloning reproduces a reference voice, whereas text-based voice design creates a voice from a natural-language description. However, neither interface directly enables users to modify the timbre of a given reference and synthesize speech with the modified voice. Meanwhile, utterance-level expressive instructions leave changes across individual text segments underspecified. We introduce \textbf{EDICT}, a framework that unifies global timbre editing and local expressive control by using an edited acoustic reference to anchor voice identity across segments. To enable synthesis with an instruction-edited voice, EDICT combines reference audio with structured timbre edits to generate an edited reference in codec-token space. This representation serves as a shared voice anchor for a frozen TTS backbone, allowing segment-specific natural-language instructions to guide expression. To accommodate instruction changes while supporting acoustic continuity, EDICT rebuilds the KV cache at each segment boundary, refreshing instruction conditioning while retaining bounded acoustic context from previously generated speech. Evaluations on our proposed TimbreEdit-Bench and IntraTTS-Bench demonstrate improved timbre editing and a favorable balance between local instruction adherence, speaker consistency, and transition quality. Audio demos are available.
Abstract (translated)
URL
https://arxiv.org/abs/2610.11437
PDF
-
A computational framework for the acoustic characterization of Suikinkutsu
2026-10-08 04:14:09Paul Haimes, Alarith Uhde, Daniel Moritz MarutschkeAbstract
Suikinkutsu are partially concealed acoustic devices traditionally installed adjacent to stone water basins (tsukubai) in Japanese temple gardens. While previous research has examined their physical acoustics and cultural significance, computational methods for documenting and comparing their acoustic characteristics remain underexplored. This exploratory study introduces a computational framework for analyzing suikinkutsu using spectral and temporal audio features extracted from field recordings. Rather than estimating the physical geometry of individual suikinkutsu, the framework characterizes their acoustic profiles using a multidimensional set of computational descriptors, enabling systematic comparisons between recordings across different sites and installations. Audio recordings from nine suikinkutsu located at temples, shrines, and gardens in the Kansai region of Japan were analyzed using a feature-extraction pipeline that incorporates Fast Fourier Transform (FFT), Mel-Frequency Cepstral Coefficients (MFCCs), Root Mean Square (RMS) energy, spectral centroid, spectral bandwidth, and spectral contrast. The extracted features indicate that, although the suikinkutsu share broadly similar spectral and timbral characteristics, each exhibits a measurably distinct acoustic profile; a supplementary analysis of within-site and between-site variability suggests these profiles reflect a bounded range of characteristic resonant behaviour rather than a fixed acoustic signature. The proposed framework provides one of the first computational approaches for systematically characterizing and comparing suikinkutsu, contributing a practical foundation for future research and the digital preservation of this distinctive form of Japanese environmental design.
Abstract (translated)
URL
https://arxiv.org/abs/2610.11219
PDF
-
MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness
2026-10-08 03:17:17Jinbo Hu, Hang Su, Lichun Fan, Heinrich Dinkel, Gang Li, Zhanchen Dai, Yiru Zhang, Chang Liu, Peng Wang, Junnan Wu, Jian Luan, Cong Zou, Heng QuAbstract
Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at this https URL and this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.11156
PDF
-
SEER: Source-Conditioned Emotion Enhancement via Retrieval for Cochlear-Implant Speech
2026-10-08 02:56:52Hsing-Hang Chou, Yun-Shao Lin, Ching-Chin Sung, Chi-Chun LeeAbstract
Cochlear implants (CIs) restore speech access but weaken cues needed for vocal emotion recognition. Prior CI-oriented enhancement requires parallel normal/strong recordings and intensity labels. We propose SEER, a retrieval-based framework that learns which same-emotion reference helps each source remain recognizable after CI processing. A source-conditioned retriever learns CI-aware utility from sampled emotional voice conversion outcomes, while uncertainty-guided exploration avoids exhaustive pair evaluation; neither parallel recordings nor intensity labels are required. SEER improves Source macro-F1 at N8 by 7.30 points on RAVDESS and 11.66 points on ESD, with significant ESD gains across N4/N8/N16. Sixteen-listener RAVDESS gains are significant across all conditions. Exhaustive analysis finds an aggregate benefit from stronger references but little effect from matching gender or content.
Abstract (translated)
URL
https://arxiv.org/abs/2610.11131
PDF
-
Cross-Lingual Speaker Verification with Self-Supervised Pre-Trained Models
2026-10-08 02:13:07Jinghan Peng, Yu Zheng, Weiqiang Wang, Jian LiuAbstract
Speaker verification (SV) performance degrades under language mismatch due to the entanglement of speaker identity with language-specific acoustic cues. To address this problem, we leverage large-scale self-supervised pre-trained models (PTMs) to learn language-agnostic speaker representations. We utilize PTMs as robust front-end feature extractors, capitalizing on their rich acoustic and linguistic knowledge acquired from vast, diverse audio data. These generalized features are then used to train a downstream speaker embedding network, effectively disentangling speaker identity from language-specific characteristics. We validate our approach on the TidyVoice2026 benchmark, which benchmarks SV under language mismatch. Our proposed system (team T02) achieves equal error rates (EERs) of 2.21% on tv26_eval-A and 2.99% on tv26_eval-U.
Abstract (translated)
URL
https://arxiv.org/abs/2610.11099
PDF
-
Robust Prediction of Internal Wave-Affected Multi-Scale Sound Speed Distribution Using Lightweight Kolmogorov-Arnold Networks with Hybrid Basis Functions
2026-10-07 20:29:05Wei Huang, Junpeng Lu, Tianhe Xu, Hao Zhang, Feng YinAbstract
The underwater sound speed distribution directly governs acoustic propagation paths, rendering it critically important for underwater acoustic communication and target localization. Conventional sound speed profile (SSP) prediction methods provide a good way to estimate the underwater sound speed distribution without on-site data measurement, thus breaking through the coverage area constraints of sonar observation equipment and making the model universal in most marine areas. However, underwater sound speed exhibits multi-scale variations, such as diurnal, quarterly, and intermittent fluctuations caused by ocean processes such as internal waves. This makes it difficult for the fixed structure models in existing methods to have good generalization ability for multi-scale sound speed distribution patterns. To tackle this problem, we proposed a lightweight hybrid basis-function empowered Kolmogorov-Arnold network (LHBF-KAN) model for multi-scale sound speed prediction. We aim to construct a multi-branch representation layer in which different basis functions respond to distinct temporal patterns, from slowly varying background trends to rapid fluctuations induced by dynamic ocean processes, allowing the model to naturally accommodate the inherently multi-scale evolution of sound speed at different depths. To prevent the multi-branch structure from increasing model size, a pruning strategy is further introduced to suppress branches with consistently low contribution during training, yielding a compact architecture, suitable for deployment on resource constrained underwater platforms.
Abstract (translated)
URL
https://arxiv.org/abs/2610.10877
PDF
-
AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text
2026-10-07 18:31:29Tristan Wu, Daniel Chin, Liwei Lin, Junan Zhang, Gus XiaAbstract
Audio generation models can translate natural-language descriptions into sound, but their outputs are typically waveforms. Their audio quality is constrained by audio compression, and their outputs do not readily support direct edits to notes, timbral parameters, or modulation relationships. We present AutoSynth, which represents MIDI performance events, fixed synthesizer parameters, and variable-length modulation routes as a unified sequence for a synthesizer, and learns their dependencies with an audio-conditioned autoregressive model. A single model supports both tasks. Given reference audio, the model directly predicts a synthesizer program; given text, it uses a pretrained audio generation model and converts the generated audio into a program. Training consists of two stages: supervised learning on large-scale audio-program pairs automatically constructed from a small set of native presets, followed by group-relative policy optimization with a mixed reward combining semantic similarity, pitch-related features, acoustic similarity, and sound usefulness. The pipeline requires neither paired text-target-program annotations nor a differentiable synthesizer. Experiments show that AutoSynth produces complete, editable synthesizer programs and achieves competitive results in both synthesizer inversion and text-driven generation. Audio demos and source code are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.10774
PDF
-
Detection-Guided Adaptive Purification with Diffusion Models for Robust Audio Deepfake Detection
2026-10-07 18:19:55Muhammed Salih Kayhan, Qiben YanAbstract
Audio deepfake detectors remain vulnerable to adversarial perturbations that suppress the acoustic cues used for detection, allowing manipulated utterances to evade the detector. Although existing defenses can improve robustness, they require retraining the detector or introduce additional distortion. Diffusion-based purification instead leaves the pretrained detector unchanged, but existing methods use the same purification strength for all inputs, creating a trade-off between removing adversarial perturbations and preserving the subtle spoofing cues needed for detection. In this paper, we propose Detection-Guided Adaptive Purification (DGAP), a diffusion-based defense that adjusts purification strength per input. Building on the observation that a light purification perturbs the detector score of an adversarial input far more than that of a benign one, the framework uses the resulting score shift as a reference-free indicator of adversarial manipulation. Inputs with small shifts are passed unchanged, whereas flagged inputs undergo stronger purification before final detection. We evaluate the framework against three adversarial attack settings across three deepfake detectors, and compare it with nine existing defenses. Our results show that DGAP achieves the best defense performance across all detectors while leaving benign inputs nearly unaffected, and remains effective under the defense-aware adaptive attack.
Abstract (translated)
URL
https://arxiv.org/abs/2610.10752
PDF
-
Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs
2026-10-07 18:14:58Pooneh Mousavi, Mirco Ravanelli, Cem SubakanAbstract
Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.
Abstract (translated)
URL
https://arxiv.org/abs/2610.10749
PDF
