arXiv_SD
-
ForestIR: Physics-Informed Forest Sound Simulation for Array-Based Bioacoustic Remote Sensing
2026-07-07 14:04:25Xin Shen, Jennifer N. Kampe, Changwoo J. Lee, Braden Scherting, Panu Somervuo, Ari Lehti\"o, Sandro von Brandenburg, Ossi Nokelainen, Otso Ovaskainen, David B. DunsonAbstract
Microphone array-based passive acoustic monitoring is increasingly used for biodiversity sensing in forests. However, design and evaluation of array systems and configurations remains difficult since field recordings are costly, difficult to reproduce, and provide limited control over forest and atmospheric conditions. We present ForestIR, a physics-informed and reproducible simulation framework that links forest and environmental conditions to microphone-array recordings for bioacoustic remote sensing. Through a more realistic sound propagation method and a systematic control over array design and environmental factors, ForestIR provides a practical simulation framework for optimizing array-based monitoring systems, especially for sound source localization purposes. ForestIR generates source-microphone impulse responses (IRs) under user-controlled forest and atmospheric conditions, and renders synthetic array recordings by convolving test signals with controlled background noise. We evaluate and demonstrate realistic features of ForestIR through experiments based on localization sensitivity to forest layout and atmospheric conditions, and also comparison between simulated IRs with sine-sweep IR measurements from a field experiment. ForestIR provides a practical way to test how forest and ground conditions, atmospheric state, and array geometry affect bioacoustic localization, and can support microphone-array design, robustness testing, and synthetic-data generation for passive acoustic monitoring.
Abstract (translated)
URL
https://arxiv.org/abs/2607.06299
PDF
-
Learning-based Physics-Constrained Neural Kernel for Sound Field Estimation With Source-Position-Dependent Directional Weighting
Abstract
A learning-based physics-constrained neural kernel for sound field estimation is proposed. Sound field estimation aims to estimate the spatial distribution of an acoustic field from a discrete set of microphone measurements, which have a wide range of applications. Among existing sound field estimation methods, kernel-regression-based methods offer a flexible and principled framework for incorporating physical constraints and allow inference through linear operation. It is also possible to adapt the kernel function to the target acoustic environment by representing the directional weighting function as an implicit neural representation (INR) and optimizing hyperparameters using measurements. However, the kernel function is generally optimized for single snapshot measurements of the microphones, which can lead to strong overfitting and poor generalization. We propose a source-position-dependent INR for the directional weighting function, enabling the kernel function to capture common directional patterns and to generalize to unseen source positions in the target acoustic environment. Experimental results indicate that our proposed method outperforms the snapshot-based method by estimating a directional weighting function that matches the directivity of the target sound field.
Abstract (translated)
URL
https://arxiv.org/abs/2607.06274
PDF
-
Flow Matching-Based Speech Source Separation with Best-of-N Biometric Sampling
2026-07-07 10:00:23Anastasia Zorkina, Alexandr Anikin, Nikita Khmelev, Anastasiya Korenevskaya, Sergey Novoselov, Vladimir Volokhov, Maxim Korenevsky, Yuriy MatveevAbstract
Single-channel speech separation remains challenging for real-world deployment due to source permutation ambiguity, sampling variability of generative models, and the difficulty of processing long recordings with chunk-wise inference. We address these issues with a conditional flow-matching-based method that produces an ordered two-source output conditioned on the mixture. A frozen speaker encoder defines the source order during training and is reused at inference for biometric best-of-$N$ candidate selection and chunk-level channel alignment. We evaluate separation quality on Libri2Mix benchmark using SI-SDR, PESQ, and ESTOI, and measure downstream impact using cpWER for automatic speech recognition and EER for speaker verification. The results show that the proposed Transformer U-Net variant is competitive with strong baselines in objective separation metrics and achieves the lowest downstream automatic speech recognition and speaker verification error rates in all evaluated settings.
Abstract (translated)
URL
https://arxiv.org/abs/2607.06088
PDF
-
Determinantal point process sampling for bioacoustic active learning
Abstract
Eco-acoustic monitoring generates vast volumes of audio data, making active learning a promising approach for reducing annotation effort while efficiently training reliable biodiversity classifiers. This report presents CARE-DPP, a batch active-learning acquisition method submitted to BioDCASE Active Learning for Bioacoustics 2026 challenge. The method combines class-balanced predictive uncertainty with embedding-space novelty, while a determinantal point process (DPP) objective selects a high-quality and non-redundant acquisition batch. The uncertainty-novelty balance is annealed over the annotation budget: early cycles emphasize geometric coverage, whereas later cycles increasingly exploit classifier uncertainty. To mitigate unreliable early scores, the DPP candidate pool mixes top-quality candidates with a decreasing proportion of random exploration. An adaptive acquisition schedule uses smaller batches early and larger batches later. Evaluated over five repeats on the BirdSet HSN, POW and UHH subsets and on ATBFL, CARE-DPP obtains a mean development AULC of 0.50 for macro mAP, compared with 0.46 for the official CoreSet baseline. Ablations identify DPP batch diversification and the adaptive acquisition schedule as the largest contributors.
Abstract (translated)
URL
https://arxiv.org/abs/2607.06063
PDF
-
Fr'echet Distance Loss on Speech Representations for Text-to-Speech Synthesis
2026-07-07 09:07:30Ho-Lam Chung, Kuan-Po Huang, Bo-Ru Lu, Hung-yi LeeAbstract
Few-step diffusion and flow-matching text-to-speech (TTS) models are usually trained with local objectives, such as conditional flow matching, reconstruction, and stop prediction. These losses provide stable optimization, but they never ask whether sampled speech follows the distribution of high-quality speech. We propose Speech Representation Fr'echet Distance loss (SR-FD), a training-time distributional regularizer for tokenizer-free flow-matching autoregressive TTS. During fine-tuning, the model synthesizes speech with the same few-step sampler used at deployment, and SR-FD matches the mean and covariance of frozen Whisper and CTC features of this speech to reference statistics computed offline from three complementary content targets. The loss requires no discriminator and no inference-time computation. On Seed-TTS English, four-step SR-FD fine-tuning reduces WER from the original four-step VoxCPM2 baseline's 2.2279% to 1.4147%, a 36.5% relative reduction, and also surpasses the original ten-step baseline at 1.7366%; both gains are significant under an utterance-level paired bootstrap. Speaker similarity and objective quality proxies are preserved at the ten-step level, and an error analysis shows the gain comes from content substitutions across all prompt lengths. SR-FD is thus an intelligibility-improving distributional regularizer for few-step TTS.
Abstract (translated)
URL
https://arxiv.org/abs/2607.06027
PDF
-
Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on Any Music
2026-07-07 08:57:34Tom\'a\v{s} Sourada, Katia Vendrame, Jan Haji\v{c} jrAbstract
Music represents a cornerstone of human culture, existing digitally across diverse modalities, including audio, symbolic encodings (e.g., MIDI, MusicXML), and sheet music. Despite the advancement of Multimodal Large Language Models (MLLMs), current music benchmarks face three major limitations. First, large static benchmarks are resource-intensive to evaluate, and it remains unclear how their results transfer to diverse kinds of music beyond those included in the benchmark. Second, benchmarks claiming to measure "music understanding" often fail to require music perception. Third, they do not support systematic performance comparisons across musical modalities. To overcome these issues, we introduce the Music I Care About Meta-Benchmark (MusICA-MetaBench), a framework that automatically derives on-demand benchmarks directly from user-provided data. By leveraging structured symbolic representations (e.g., MusicXML) and our pre-defined question templates, we build multiple-choice question-answer pairs that probe music perception competencies, aligned with music pedagogy, across audio, music notation images, and symbolic files. We demonstrate our framework with the ChoraleBricks dataset, and experimentally determine benchmark sizes that ensure statistically reliable model comparisons for this setup. By comparing against text-only and white-noise baselines, we show our questions do measure music perception. Ultimately, MusICA-MetaBench represents a significant advancement in the cross-modal assessment of music perception for MLLMs. By proposing a dataset-specific benchmarking paradigm, it enables efficient on-demand evaluation of music perception capabilities.
Abstract (translated)
URL
https://arxiv.org/abs/2607.06015
PDF
-
Escaping the Procrustean Bed: Groupwise Orthogonal Connectors for Audio-Language Models
2026-07-07 08:57:00Ho-Lam Chung, Ke-Han Lu, Yi-Cheng Lin, Guan-Ting Lin, Yiming Chen, Hung-yi LeeAbstract
Audio-language models compress a speech encoder's output through a Querying Transformer (Q-Former) connector before feeding it to a large language model. We identify two failures in this compression. The connector's output vectors collapse to a single direction, and different speakers produce nearly indistinguishable outputs, with paralinguistic cues such as speaker identity, gender, and prosody lost along the way. Our method, ORCA, reverses this collapse by splitting the queries into groups whose outputs are constrained to point in different directions. On SAKURA multi-hop reasoning, ORCA gains 26.4 points over an identically trained 4B baseline, reaching 75.2% (vs. 49.0% for the 8B Audio Flamingo-3). At the connector level, the same change cuts query redundancy by 12x and raises cross-speaker variance by 75x.
Abstract (translated)
URL
https://arxiv.org/abs/2607.06014
PDF
-
Multimodal Video-to-Music Recommendation via Semantic Retrieval and Temporal Reranking
2026-07-07 08:04:56Seungheon Doh, Minhee Lee, Sangmoon Lee, Ben Sangbae Chon, Juhan NamAbstract
We present VTMR, a two-stage framework for Video-To-Music Recommendation. In Stage~1, VTMR aligns comprehensive video and music signals in a joint audio-visual-text representation space and efficiently retrieves semantically compatible candidates using coarse global embeddings. In Stage~2, it reranks the retrieved candidates by attending to the temporal sequences of both video and music, thereby capturing fine-grained temporal correspondence. Evaluated on the video-to-music recommendation task, the multimodal retrieval stage improves R@10 from 14.2 to 15.9 and Median Rank from 75 to 58 over the strongest baseline; the temporal reranker further boosts R@10 to 18.3 and Median Rank to 46, demonstrating complementary gains from richer query encoding and temporal alignment. A human preference study confirms that VTMR is on par with a commercial baseline in overall preference, while outperforming a generative baseline in music quality.
Abstract (translated)
URL
https://arxiv.org/abs/2607.05971
PDF
-
A Self-Supervised Approach for Minimal-Annotation Hydroacoustic Data Exploration
2026-07-07 07:47:16Pierre-Yves Raumer, Axel Marmoret, Dorian Cazau, Anatole Gros-Martial, Richard Dreo, Maelle Torterotot, Sara Bazin, Flore Samaran, Jean-Yves RoyerAbstract
Passive hydroacoustic monitoring often generates large volumes of continuous recordings that are only partially exploited due to the cost of manual annotation. Supervised detection methods perform well but require large labeled datasets, seldom available for rare signals or understudied environments. This work proposes a self-supervised exploration pipeline to address this limitation in low-frequency settings. A Masked AutoEncoder (MAE) is pre-trained on a reconstruction pretext task, then used to extract patch-level representations from spectrograms. Within each spectrogram, adjacent informative patches are aggregated into event-level embeddings, enabling the disentanglement of overlapping events. These embeddings are then clustered at the dataset scale using the dimension reduction algorithm UMAP and the clustering algorithm HDBSCAN to identify hydroacoustic patterns. The pipeline was applied to a multi-year hydroacoustic dataset collected near Mayotte Island, Indian Ocean, containing marine mammal vocalizations, seismo-volcanic signals, and anthropogenic noise. The 317 clusters were manually mapped to 15 hydroacoustic classes or noise in less than one hour. The method was evaluated in two ways. Quantitatively, when used as a classifier, it achieved performance comparable to two existing detectors. Qualitatively, it recovered known seasonal patterns of marine mammal acoustic activity. It also identified patterns of previously unstudied signals, thereby demonstrating its practical value.
Abstract (translated)
URL
https://arxiv.org/abs/2607.07733
PDF
-
Context-Aware ASR for Mandarin Technical Lectures
2026-07-06 13:31:09Ho-Lam Chung, Yiming Chen, Hung-yi LeeAbstract
Technical lectures mix Mandarin speech with English technical terms. These terms carry the core meaning of the lecture, yet they occupy few characters. Character error rate (CER) therefore hides their recognition failures. We study whether lecture context helps recognize these terms. We build a term-rich Mandarin AI/ML lecture benchmark, and we define term-centric metrics that measure technical-term recognition directly. We then propose a two-pass, reference-free decoding method. The first pass runs segment-only ASR. We extract the most frequent technical terms from the first-pass hypotheses, and we prompt the recognizer with this self-built glossary in the second pass. Across five ASR backbones, the first-pass glossary raises term recall for every model and holds or lowers CER on all five. On Breeze-ASR-25 it lifts term recall from 52.50% to 60.13% while lowering CER, and a hybrid that adds a small external term list reaches 62.05% recall and 82.73% term precision. Lecture context, recovered from the model's own output, is a practical signal for technical-term recognition. Term-centric evaluation exposes errors that CER misses.
Abstract (translated)
URL
https://arxiv.org/abs/2607.05058
PDF
-
Listen, Think, Transcribe: Continuous Latent Test-Time Scaling for ASR
2026-07-06 13:25:33Ho Lam Chung, Yiming Chen, Dau-Cheng Lyu, Hsiao-Tsung Hung, Hung-yi LeeAbstract
End-to-end ASR models transcribe in a single pass, leaving no room for the decoder to revisit hard inputs. We propose LatentASR, a parameter-efficient method that adds continuous latent test-time scaling to a frozen ASR backbone. Two small trainable modules drive it: a Latent Adapter that iteratively refines a few latent prefix positions through bounded, stabilized updates, and a Value Head that predicts whether extra computation will help and halts the loop early. The Qwen3-ASR-0.6B backbone stays fully frozen, and we train only ~4M extra parameters. We activate this loop with a deliberately small, diverse 500-utterance training set. Under this minimal-data regime, standard adaptation methods all regress: full fine-tuning, LoRA, and prompt tuning each increase WER. LatentASR is the only tested method that reduces WER on both clean benchmarks (FLEURS -2.54% and VoxPopuli -0.47% relative). The reductions are concentrated on intrinsically hard inputs. On accented and code-switched speech (ASCEND), LatentASR achieves a 16.0% relative CER reduction. Across 30 FLEURS languages (23,049 utterances), the multilingual WER decreases uniformly across resource tiers, confirming that the adapter generalizes without overfitting. Dynamic halting preserves most of the clean-set reduction at a fraction of the compute, skipping roughly half of all utterances at the entry gate. Our results show that a small, carefully chosen activation set can switch on test-time scaling inside a frozen ASR model without corrupting the model itself, converting fixed per-utterance compute into input-dependent compute where it is most needed.
Abstract (translated)
URL
https://arxiv.org/abs/2607.05051
PDF
-
Towards Robust Uncertainty-Aware Speaker Modeling
Abstract
Speaker embeddings aggregate frame-level acoustic features into compact representations for speaker recognition. Recent uncertainty-aware speaker modeling approaches further characterize the reliability of speaker embeddings by estimating their associated uncertainty. However, existing methods often suffer from inaccurate uncertainty estimation and uncertainty miscalibration under domain shifts. To address these challenges, we propose a robust uncertainty modeling framework from both estimation and adaptation perspectives. Specifically, we introduce an Inter- and Intra-Speaker-Aware Uncertainty Softmax that incorporates both inter-speaker separability and intra-speaker variability into uncertainty learning, enabling uncertainty estimates to better capture the reliability of speaker embeddings. Furthermore, we propose an Uncertainty-Calibrated Domain Adaptation (UCDA) framework to mitigate uncertainty miscalibration caused by domain mismatch. Extensive experiments on both in-domain and cross-domain benchmarks demonstrate that the proposed approach consistently improves uncertainty reliability and speaker recognition robustness.
Abstract (translated)
URL
https://arxiv.org/abs/2607.04937
PDF
-
Adaptive Diversity-Uncertainty Active Learning with Redundancy Control for Bioacoustic Event Classification
2026-07-06 09:42:51Gabriel Dubus, Hugo Magaldi, Anatole Gros-MartialAbstract
Active learning is a promising framework for reducing annotation costs in large-scale bioacoustic monitoring, where expert labeling is expensive and data distributions are highly heterogeneous across environments. However, existing sample selection strategies often rely on static criteria that do not adapt to the evolving reliability of model predictions during training. This limitation can lead to suboptimal exploration-exploitation trade-offs and redundant sample selection. We propose an active learning strategy for multilabel bioacoustic event classification that jointly models predictive uncertainty, embedding-space diversity, and intra-batch redundancy. The method introduces an adaptive weighting scheme that progressively shifts from diversity-driven exploration in high-uncertainty regimes toward uncertainty-driven exploitation as the model becomes more confident, reflecting the increasing reliability of the classifier. To further improve annotation efficiency, a greedy Maximum Marginal Relevance (MMR) procedure is used to enforce diversity among selected samples within each acquisition batch. We evaluate the proposed approach within the BioDCASE 2026 Task 4 active learning framework on terrestrial (BirdSet) and marine (ATBFL) benchmarks using pretrained audio embeddings and a fixed annotation budget. Experimental results show consistent improvements in learning efficiency and competitive in terms of macro mean Average Precision (mAP) and Area Under the Learning Curve (AULC) across heterogeneous acoustic domains. The gains are particularly pronounced on structured terrestrial soundscapes, while performance remains competitive under noisier marine conditions. These findings demonstrate that adaptive acquisition strategies combining uncertainty estimation, embedding-space diversity, and redundancy-aware batch construction provide an effective and robust solution for [...].
Abstract (translated)
URL
https://arxiv.org/abs/2607.04868
PDF
-
Ranking the Impact of Contextual Specialization in Neural Speech Enhancement
2026-07-06 08:59:07Peter Leer, Svend Feldt, Zheng-Hua Tan, Jan {\O}stergaard, Jesper JensenAbstract
We systematically investigate neural speech enhancement systems, ranging from very small ($\sim$10\,k parameters) to medium-large ($\sim$2-5\,M parameters), which specialize to acoustic conditions using contextual information such as speaker identity, noise type, speaker gender, spoken language, and SNR. By fine-tuning generalist models on specific data subsets, we find that specializing to a speaker's identity consistently yields the largest gains in estimated speech intelligibility and quality. In contrast, specializing to SNR, noise type, or gender offers only marginal benefits. Crucially, we show that a small model specialized to both a specific speaker and a specific noise type can match or exceed the performance of a generalist model ten times its size. Further, cross-lingual tests reveal that models specialized to a target language outperform multilingual generalists, suggesting that language is a salient feature for specialization. These findings highlight the potential of small, adaptive models for resource-constrained applications like hearing aids, which specialize on-the-fly to contextual information.
Abstract (translated)
URL
https://arxiv.org/abs/2607.04826
PDF
-
Extending Xenakis: From Architectural Geometry to Sonification of the Philips Pavilion
2026-07-06 06:30:45Changda Ma, Sunshiyu Wang, Canting Zhu, Alexandria SmithAbstract
Architecture and music have been linked through proportion and temporal structure, yet architectural geometry is rarely viewed as a source of generative music. Revisiting Xenakis' one-directional transformation from string glissandi in Metastaseis to the ruled surfaces of the Philips Pavilion, we invert this workflow and sonify the completed Pavilion as a temporal composition. We reconstruct the Pavilion as nine ruled surfaces, extract their governing ruling lines, and subdivide each surface into structural lines and spatial sampling points. Four evenly spaced ruling lines per surface generate continuous string glissandi, while 3357 sampled points develop five density-based energy blocks and a sparse brass and woodwind subsequence. Implemented in Python, the system produces MIDI rendered in Ableton Live, accompanied by a real-time 3D visualization that reveals architectural motion, stasis, and structural contrast through sound and image. In general, this work paves the way for the transfer of architectural geometry as a performable musical structure, extending Xenakis's architectural and musical thinking to sonification and interactive music practice.
Abstract (translated)
URL
https://arxiv.org/abs/2607.06589
PDF
-
Weakly Guided and Autoregressive Beamformer Parameterization for Generalizable Moving Speaker Extraction in Higher-Order Ambisonics
2026-07-05 19:32:57Jakob Kienegger, Tal Peer, Sina Khanagha, Timo GerkmannAbstract
Linear spatial filters (beamformers) enable robust, generalizable and interpretable speech enhancement with performance guarantees under ideal parameterization. Modern beamformers are often parameterized by deep neural networks, whose performance degrades in dynamic scenarios with multiple moving speakers of unknown directions. We propose a data-driven beamforming pipeline, which only requires an estimate of the target's initial direction. Building on a higher-order ambisonics representation, we show that neural temporal-spectral processing can be decoupled from linear spatial processing, and thereby achieve generalizable and array-agnostic enhancement. By incorporating autoregression into a frame-wise causal framework, we maintain consistent performance throughout fast speaker motion and long recordings. Evaluation on synthetic data demonstrates robust enhancement under challenging conditions with closely spaced and crossing speakers. Real-world recordings in a dynamic office meeting scenario complement these findings and show generalizability across varying ambisonics orders.
Abstract (translated)
URL
https://arxiv.org/abs/2607.04471
PDF
-
Sampling Bias Compensation for Robust Evaluation of Audio Classification Systems with Partially Labeled Evaluation Datasets
2026-07-05 19:08:01Javier Naranjo-Alcazar, Annamaria Mesaros, Tuomas Virtanen, Pedro ZuccarelloAbstract
The performance of acoustic machine learning systems is commonly evaluated using fully annotated test sets. In real-world deployments, however, exhaustively labeling large volumes of continuously collected audio data is often infeasible. Consequently, performance assessment typically relies on a small labeled subset of the available data, introducing a sampling bias that can severely distort evaluation metrics. This paper studies methods for compensating the bias in evaluation-labeled subsets under strict annotation-budget constraints. We study whether importance weighting techniques can mitigate this discrepancy by compensating for the selection bias. Specifically, we implement and compare three density-ratio estimation methods: kernel density estimation (KDE), logistic regression, and k-nearest neighbors (kNN), utilizing feature-space representations of the deployed audio. To emulate realistic deployment scenarios, the labeled subsets are generated using five distinct sampling strategies based on active learning techniques. Experiments conducted on an audio scene classification (ASC) benchmark demonstrate that importance weighting consistently yields more realistic accuracy estimates, significantly reducing the gap between subset-based metrics and the true evaluation performance.
Abstract (translated)
URL
https://arxiv.org/abs/2607.04463
PDF
-
Doppelganger: Sound Effects and Their Synthetic Twins
2026-07-05 14:35:25Elliott AshAbstract
Audio-conditioned generators now produce synthetic sound effects from real recordings, so the real and synthetic versions of an event increasingly coexist in sound libraries and in the corpora used to train audio models -- yet no benchmark measures whether a representation can match a synthetic clip to the specific real recording it was generated from. I introduce Doppelganger, a benchmark for matching sound effects across the synthetic-real boundary, pairing 10,420 real clips across 34 everyday sound events each with an audio-conditioned synthetic twin, alongside a controlled 7-class corpus. Off-the-shelf audio encoders do not cross the boundary cleanly. Making the embedding ignore the boundary the standard way -- training it on sound-event labels -- works on familiar sounds but backfires on new ones, dropping below the untrained encoder. Training on the pairs instead -- a clip and its own synthetic twin -- generalizes. On sound events held out of training, it recovers the exact real source about 80% of the time (up from 61% untrained; chance 0.03%), whereas no objective meaningfully improves category-level recognition on those unseen events. The learned matching is specific to one generator -- it survives changes to that generator's settings but not a switch to a different generator, and collapses for the text-only generators tested. A human annotation baseline (49 listeners) lands well above chance but below the models on the same trials. Synthetic twins fool people into calling them real about 29% of the time, yet a generator-specific detector separates these audio-conditioned twins from real recordings perfectly.
Abstract (translated)
URL
https://arxiv.org/abs/2607.04337
PDF
-
MOSAIC: Interpretable Multi-Token Cross-Attention of Biophonetic and Self-Supervised Representations for Unified Voice Anti-Spoofing
2026-07-05 14:02:09Yugwon WonAbstract
The dominant trend in voice anti-spoofing fuses self-supervised (SSL) backbones (e.g., WavLM) with handcrafted features, yet such fusion typically lacks transparency in cue-to-layer interactions, and simple concatenation limits cross-modal learning. We propose MOSAIC (Multi-token Oriented Speech Anti-spoofing via Integrated Cross-attention), an interpretable multi-token cross-attention framework that splits a 152-dimensional biophonetic feature vector into six semantic-group query tokens (Praat, phase, LFCC mean/std, sub-band mean/std) and attends them over thirteen mean-std pooled WavLM-Large transformer layers as keys/values. The resulting 6x13 attention matrix visualizes cue-to-layer alignment; a z-score analysis of the per-token activations shows that biophonetic/phase tokens activate more on bona fide speech while spectral/channel tokens activate more on spoofed speech -- yielding per-cue, per-layer attribution that extends prior fusion approaches. Trained jointly with focal loss, a dual LA/PA domain-adversarial classifier, and a bona-fide-only VAE regularizer, MOSAIC attains EER 1.93% / 1.98% on ASVspoof 2019 LA / PA -- a single unified model that approaches the PA-specialized SOTA (LFCC-CMR, 1.34%) while remaining competitive on LA -- and 9.28% / 6.21% / 40.09% on ASVspoof 2021 LA / DF / PA.
Abstract (translated)
URL
https://arxiv.org/abs/2607.04314
PDF
-
EEG-Based Imagined Speech Decoding Using a Hybrid CNN-SNN Architecture
2026-07-04 12:23:46Fatima Shalhoub, Mariam Al Mawla, Kabalan Chaccour, Iv\'an L\'opez-Espejo, Hoda FaresAbstract
Imagined speech decoding using EEG signals has emerged as a promising frontier in brain-computer interface (BCI) research, particularly to restore communication for individuals with severe speech impairments. However, decoding imagined speech remains a complex task due to the non-stationary, low-amplitude, and highly variable nature of EEG signals. Existing methods often rely on classical machine learning or deep learning models that fail to exploit spike-based temporal dynamics or event-driven firing mechanisms of biological neurons, which are naturally modeled by spiking neural networks (SNNs). In this study, we propose a hybrid decoding pipeline that extracts temporal representations using convolutional neural networks (CNNs) followed by biologically inspired temporal classification via SNNs. To our knowledge, this is the first study to integrate SNNs into EEG-based imagined speech decoding. Experimental results show that the proposed CNN-SNN architecture achieves an accuracy of 80.13% on the 2020 BCI Competition III benchmark, surpassing existing methods reported in the literature (up to 70.19%) under comparable evaluation settings. These findings demonstrate the effectiveness of spike-based temporal decoding for imagined speech, highlighting the promise of biologically grounded pipelines for next generation neuromorphic BCI applications.
Abstract (translated)
URL
https://arxiv.org/abs/2607.03844
PDF
