arXiv_SD
-
Synthesizing Ultrasonic Echo Profiles from Text for Open-Vocabulary Silent Speech Recognition
2026-10-11 14:49:19Jiacheng Liu, Ruidong Zhang, Fran\c{c}ois Guimbreti\`ere, Cheng ZhangAbstract
Eyewear that senses articulation with inaudible ultrasound can now recognize silent speech over an open vocabulary, but its accuracy is limited by the cost of training data. We present SoniGAN, which synthesizes the ultrasonic echo profiles of silent articulation from text and uses them as synthetic supervision. A voice-cloned text-to-speech (TTS) model reads a text corpus, and a two-stage generator synthesizes echo profiles from representations of the spoken content of that audio. Pretraining a recognizer on 104 hours of the synthesized data improves the word error rate (WER) of a public open-vocabulary benchmark by 2.3 points. Adding the sessions the corpus discards for a faulty microphone brings the system to 22.2% without any language model. Lexicon-constrained decoding, LLM rescoring, and ensembling further bring it to 14.9%, the first minimally obtrusive silent speech interface to fall inside the error range in which users report no loss of willingness to use silent speech input.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15671
PDF
-
AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency
2026-10-11 14:45:42Yunjia Zhang, Junan Zhang, Jing Yang, Xueyao Zhang, Fan Fan, Zhizheng WuAbstract
We present AnyEdit, a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments, including noise, reverberation, and background music. Recent unified generative models already achieve strong speech and singing generation in clean conditions, but real-world vocal editing remains challenging because the model must revise content while simultaneously preserving speaker identity, prosody, and the surrounding acoustic scene. Directly modeling noisy audio end-to-end entangles semantic editing with environmental acoustics, weakening the core vocal generation capability. We therefore adopt a two-stage design that preserves clean-domain editing ability while deferring acoustic rendering to a separate stage: an instruction-guided autoregressive editor first predicts clean prosody and content-style tokens, supported by a teacher-student noise-robust prosody tokenizer that distills clean prosodic tokens from degraded recordings; a flow-matching acoustic model then performs in-context rendering conditioned on the source acoustic context, so the edited segment inherits surrounding noise, reverberation, and background sound while maintaining consistent speaker timbre. To stabilize unedited regions, we introduce unmodified-area-aware supervision; after pre-training, we further apply direct preference optimization on mined real-to-real singing pairs. We also release AnyEditBench, a benchmark covering both speech and singing voice editing under clean, simulated, and in-the-wild acoustic conditions. Experiments on Chinese and English speech and singing datasets showed that AnyEdit consistently outperforms strong baselines in content accuracy, subjective quality, and background consistency under noisy and reverberant conditions. Demo: this https URL. AnyEditBench: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15665
PDF
-
Sleep Sound Event Detection Powered by Learnable Multi-Resolution Adaptive Line Enhancer
2026-10-11 11:10:08Chanwoo Park, Chanwoo KimAbstract
Sleep-disordered breathing, including snoring, hypopnea, and obstructive apnea, is the hallmark of obstructive sleep apnea, whose severity is quantified by the Apnea-Hypopnea Index (AHI). As conventional polysomnography is costly and clin- ically demanding, microphone-only detection has emerged as a noninvasive alternative for screening. We propose ALE- Confidence Fusion Sound Event Detection (ACF-SED), the first framework that directly injects the per-frame confidence map of an Adaptive Line Enhancer (ALE) into Transformer atten- tion biasing and feature modulation, rather than as mere pre- processing. A Multi-Resolution ALE Bank aggregates three parallel NLMS filters at distinct decorrelation delays, while a Confidence-Guided Cross-Path block couples event and noise streams via gated cross-attention. On the Audio-Polygraphy Dataset for Sleep Apnea Analysis, ACF-SED achieves state- of-the-art detection across Event-F1, Segment-F1, and PSDS, enabling end-to-end AHI estimation.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15532
PDF
-
Sori: Streamable and Controllable Model for Non-musical Audio to Symbolic Music Transformation
2026-10-11 09:26:59Kyungsu Kim, Yejin Kim, Kyogu LeeAbstract
We introduce Non-musical Audio to Symbolic Music Transformation (NASMT), a novel task that converts non-musical audio into symbolic music representations. Unlike automatic music transcription, which assumes musical input and a unique correct score, NASMT is inherently one-to-many and requires evaluation along two axes: input correspondence and musicality. We present Sori, an encoder--decoder Transformer that addresses this task through domain adversarial training (DAT), which bridges the gap between musical and non-musical audio in the encoder's feature space, and classifier-free guidance (CFG) with a continuous parameter~$\gamma$ that controls the trade-off between correspondence and musicality at inference time. Across objective metrics and a listening study, Sori provides effective continuous control via $\gamma$ and compares favorably with instrument-agnostic transcription models. Sori supports real-time streaming on a consumer CPU. Audio examples are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15474
PDF
-
BridgeCodec: Mamba Enhanced Neural Audio Codec with Schr'odinger Bridge at Low Bitrate
2026-10-11 08:54:53Zijian Lin, Jing Yang, Jinghao Luo, Zhuo Wang, Fan Fan, Zhiyong WuAbstract
Neural audio codecs achieve remarkable performance but operate as closed ecosystems with rigidly co-trained encoderdecoder pairs, severely hindering interoperability. To resolve this, we propose BridgeCodec, a versatile framework that decouples these pairs by translating latent representations between mismatched, frozen endpoints. We formulate this cross-codec translation as a Schrödinger Bridge optimal transport problem, enabling robust mapping between disparate distributions without predefined noise priors. To efficiently capture longrange speech temporal dependencies, we integrate a Mambaenhanced U-Net architecture. We validate BridgeCodec on an extreme deployment scenario: mapping a lightweight 8 kHz source encoder directly to a 48 kHz target decoder at an ultralow bitrate of 1 kbps. Experimental results demonstrate that BridgeCodec achieves superior wideband reconstruction and high perceptual quality, successfully bridging heavily bottlenecked, heterogeneous codecs.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15453
PDF
-
Learning from Real Stuttering: Synthetic Mandarin Stuttering Speech for Data Augmentation in Speech Recognition
2026-10-11 07:18:30Yao Lu, Jiaming Zhou, Zhongyuan Fu, Yong QinAbstract
Mandarin stuttered speech remains under-resourced, limiting the development of robust speech processing systems. We present a Mandarin stuttering speech synthesis and data augmentation framework that adapts both autoregressive and non-autoregressive TTS models using naturally stuttered speech. Specifically, we adapt CosyVoice2 and MaskGCT to support controllable generation of specific stuttering events while also modeling broader utterance-level characteristics of stuttered speech. Using the adapted models, we construct synthetic data resources for downstream augmentation. Evaluations show that the generated speech closely approaches real stuttered speech. Extensive ASR experiments demonstrate consistent augmentation gains and further observe a domain-gap--diversity trade-off between synthetic and real data. These results demonstrate the value of learning synthetic stuttering data directly from real dysfluency for robust Mandarin speech processing.
Abstract (translated)
URL
https://arxiv.org/abs/2606.21457
PDF
-
LiveAnon: Improving Privacy and Utility in Streaming Speaker Anonymization with Codec Language Models
2026-10-11 06:05:01Yejin Kim, Kyungsu Kim, Seungu Han, Kyogu LeeAbstract
Speaker anonymization (SA) aims to conceal speaker identity while preserving linguistic content and emotion, but existing streaming systems often lose emotion information through discrete content bottlenecks or require emotion-specific modeling. We present LiveAnon, a voice-conversion-based streaming SA system that directly conditions an autoregressive codec language model on hidden features from a pretrained streaming ASR encoder, preserving emotion without emotion-specific supervision. To reduce codec-generation latency, we introduce capped RVQ decoding, which preserves dependencies among coarse codebooks while predicting the remaining codebooks in parallel. Under VPC24, LiveAnon improves intelligibility and emotion preservation over fixed-target Stream-Voice-Anon (SVA), a strong prior streaming SA system, matches emotion-prompted SVA in emotion preservation, and achieves the strongest semi-informed privacy among streaming systems. Under VPC26, it ranks second across all metrics among the evaluated anonymization systems, demonstrating a strong privacy-utility balance.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15379
PDF
-
JSIR: An Audio-Aligned, Multi-Task Dataset of Jazz Standard Recordings
Abstract
Beat tracking, chord estimation, and structure analysis are staple tasks of music information retrieval. Their datasets are drawn largely from Western popular music, and jazz remains a repertoire on which state-of-the-art systems underperform. We present JSIR, the Jazz Standards dataset for Information Retrieval. Its 267 recordings of 44 jazz standards span 25.6 hours, 3.7 times the audio of JAAH, the closest existing resource. For each recording, we provide annotations for beat and downbeat positions, chords, and sections, all produced by a jazz musician and revised in several passes. In baseline experiments, models trained with JSIR also improve on other jazz datasets: the downbeat F-measure of a state-of-the-art beat tracker rises from 87.7% to 89.2%, and the seventh-chord accuracy of a pop-trained chord estimator on JAAH from 34% to 41%. We publicly release the annotations, the annotation tool used to create them, and a web application that displays them in sync with the audio.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15356
PDF
-
Gugak-VocalSet: A Multi-Genre Studio-Quality Dataset of Korean Traditional Singing with Sigimsae Annotations
2026-10-11 05:14:13Danbinaerin Han, Wonil Kim, Seah Hong, Dongjoo Moon, Jaeok Lee, Jongpil Lee, Chaewon Kim, Juhan NamAbstract
We present Gugak-VocalSet, the first publicly available audio dataset for Korean traditional singing with dense annotations of sigimsae, a structured pitch contour system of vocal expressions. Gugak-VocalSet contains 24.5 hours of unaccompanied solo vocal recordings across 305 tracks of representative repertoire curated by the National Gugak Center, performed by 20 professional singers in a controlled studio environment with fixed key and tempo. In addition to genre, lyrics, and mood/timbre tags, the dataset provides a unified sigimsae ontology and approximately 68,000 segments (71,700 instances including multi-label) of audio-grounded sigimsae annotations; the ontology adopts a functional-position-based hierarchy designed for consistent application across genres, and 14 expert annotators specializing in Korean traditional music performed segment-level labeling accordingly. Baseline experiments on sigimsae classification and event detection show that domain-informed F0 representations remain competitive with, and on classification surpass, frozen probing of a general-purpose music foundation model, while pitch and spectral cues play complementary roles in event-level detection. Companion page: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2610.15350
PDF
-
SIMPLE: Semantic Structure-Aligned Prompt Learning for Open-Vocabulary Audio-Visual Event Localization
2026-10-11 04:48:59Hanyu Xuan, Haoran Xu, Min Yin, Shuo Wang, Cheng Zhao, Kun Li, Fei Wang, Zhiliang Wu, Guanghui Yue, Hehe FanAbstract
Open-vocabulary audio-visual event localization aims to infer explicit segment-level categories for both seen and unseen audio-visual events. Existing solutions rely heavily on hand-crafted prompt templates, whose predictions are highly sensitive to subtle wording variations. Learnable prompts remove this manual tuning burden, but when trained solely on seen events, they tend to overfit to seen categories and distort the inter-event semantic topology required for generalization to unseen categories, a failure we term semantic topology drift. In this paper, we propose SIMPLE, a semantic structure-aligned prompt learning framework that aligns cross-category semantic topology between learnable and hand-crafted prompts across asynchronous audio and visual streams. Specifically, our SIMPLE combines Learnable Continuous Prompt Tuning (LCPT) with Semantic Topology Consistency Regularization (STCR). The LCPT automates prompt engineering through data-driven optimization, while the STCR module aligns the category-wise similarity distributions of learnable and hand-crafted prompts under random category subset sampling to alleviate semantic topology drift. Comprehensive experiments and analyses on the OV-AVEBench demonstrate that our SIMPLE framework achieves state-of-the-art performance on the seen and unseen splits, while using roughly half the trainable parameters and roughly one-third the training time.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15342
PDF
-
CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks
2026-10-11 04:05:18Mengzhe Geng, Max W. Y. Lam, Yuewen Cao, Shoukang Hu, Jing Xu, Marc Tessier, Xu LiAbstract
Removing audio can shift a visual predictor's output without showing whether the task uses the intended audio information. We introduce Conditional Input and Visual Interpretation Checks (CIViC), a four-gate protocol that links an interpretation to an intervention, held-out task comparison, or endpoint control. We demonstrate CIViC on actor-disjoint face-raster tests, a synthetic audio-feature mixture, and source-video-disjoint SpeakerVid proxy tasks. In the static face-raster test, cross-actor audio replacement meets the registered detection rule with a small effect; on a separate temporal endpoint, constant-audio fits score better than audio-trained fits in four matched seeds. In the synthetic mixture, the condition-augmented fit has higher 1,000-step RMSE than the audio-only fit (0.1807 vs. 0.1707), with the gap shrinking during training. In the SpeakerVid proxy, the five-feature-zero fit (five low-level audio summaries zeroed; identity, dialogue, and condition inputs retained) has lower whole-frame grayscale RMSE than aligned audio across four held-out conditions and five paired seeds, while aligned audio has lower segment-drift error. The results show that output activation, held-out error, and endpoint controls support different interpretations across these proxy tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15327
PDF
-
HE-NAC: High-Efficiency Neural Audio Coding via Hierarchical and Residual Multi-Band Quantization
2026-10-11 00:33:45Darius P\'etermann, Wootaek Lim, Seungkwon Beack, Minje KimAbstract
High-fidelity audio coding requires faithful reconstruction of perceptually important high-frequency components, which are often lost due to the aggressive temporal decimation and quantization of neural codecs. State-of-the-art neural audio coding methods rely on adversarial learning to hallucinate these missing components, yielding perceptually plausible but imprecise reconstructions; a limitation for high-fidelity music applications. As a remedy, we propose high-efficiency neural audio coding (HE-NAC), which provides a multi-band extension of the popular residual vector quantization (RVQ) scheme. HE-NAC introduces an effective training mechanism that harmonizes a bitrate-free blind upsampling process, residual learning to compensate it with minimal bitrate, and a multi-head decoder architecture for band-specific reconstruction. As a result, HE-NAC achieves a balanced trade-off, reliably reconstructing the core-band ($\leq$ 3 kHz) component while preserving overall sound fidelity. Subjective listening tests and objective evaluations demonstrate that HE-NAC outperforms both the DAC baseline and MPEG HE-AAC at equal or lower bitrates.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15243
PDF
-
Word-Aligned Prosody Reflects Speaker Differences More Than Depression or PTSD Severity
2026-10-10 21:01:51Hsiang-Chen Yeh, Aurosweta Mahapatra, Shreeram Suresh Chandra, Ryan L. Boyd, Berrak SismanAbstract
Word-aligned prosody is increasingly used to detect depression and posttraumatic stress disorder (PTSD), yet the stability and clinical relevance of these measurements remain unclear. We examine stability at two levels. At the word level, we ask whether a speaker produces the same word in a similar speaker-specific way across different utterance contexts. At the speaker level, we ask whether averaging prosody across many words produces consistent speaker scores across separate recordings. Drawing on principles of psychological measurement, we compare these representations in four corpora spanning acted, spontaneous, and clinical speech. Word-specific deviations show little cross-context recurrence (0.01-0.16), whereas speaker summaries are highly consistent, reaching 0.997 in ESD, 0.73-0.90 in MSP-Podcast, and 0.84-0.996 in DAIC-WOZ and PTSD-STOP. Despite this consistency, associations with depression or PTSD severity do not exceed 0.15; equivalence tests exclude effects of 0.30 in all analyses and 0.20 in DAIC-WOZ, while within-person associations with daily PTSD symptoms are near zero ($|\rho|\leq0.04$). Thus, aggregation yields stable speaker-level prosodic measures, but these measures do not detectably track depression or PTSD severity in the studied corpora. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15181
PDF
-
GAMW: Generation-Aware Mel-Domain Watermarking for TTS Traceability
2026-10-10 17:30:02Yihang Zhang, Weicai Yan, Qifei Zhang, Yangyang Wu, Tao JinAbstract
As text-to-speech (TTS) systems continue to advance, provenance tracking and security verification of synthesized speech have become increasingly important. Audio watermarking provides active provenance by embedding imperceptible, machine-recoverable information into speech. However, existing post-hoc methods typically modify the waveform after synthesis, leaving the watermark vulnerable to compound perturbations. To address this limitation, we propose GAMW, a watermark injection framework integrated into the TTS generation process. Before waveform synthesis, GAMW embeds a content-adaptive multi-bit watermark residual into the generated representation, and its decoding objective combines focal modulation with dynamic per-bit weighting. We train and evaluate GAMW separately on two TTS backbones. With a 16-bit payload, GAMW improves decoding accuracy over the strongest baseline by 9.2 and 23.5 percentage points under individual and compound perturbations, respectively, while maintaining competitive speech quality. Reliable clean-speech decoding is also retained at higher payloads. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15082
PDF
-
How Language-Independent Are Emotional Attributes? A Study on Training Data Scaling and Cross-Lingual Generalization
Abstract
Current state-of-the-art Speech Emotion Recognition (SER) systems rely on lots of training data, which is available only for a few languages such as English. For low-resource languages, using a cross-lingual model or adapting it to the target language might be an alternative to training a mono-lingual model from scratch. However, it is not clear what constitutes a "low-resource", i.e. where model adaptation loses its edge over the mono-lingual approach. We investigate this topic by training and adapting several SER models on English and Mandarin data, focusing on the emotional attributes of arousal, valence and dominance. Our results show that, for arousal, model adaptation is beneficial even with two hours of data, while for valence 10-20 hours of adaptation material is required to match the performance of a mono-lingual model trained with 100 hours of data. This indicates that task-specific model adaptation may be an effective approach in low-resource speech emotion recognition problems.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15059
PDF
-
Learning Where to Split: Dynamic Band Splitting for Audio Source Separation
Abstract
Audio source separation models must preserve fine spectral structure while keeping the cost of the separator tractable. Band-split architectures reduce this cost by manually partitioning the spectrum into predefined frequency bands and encoding each band as a token, but the same boundaries are used for every mixture. Because informative spectral structure varies with source content, fixed boundaries can merge discriminative components or spend tokens where little resolution is needed. We introduce Dynamic Band Splitting (DBS), an end-to-end framework that adapts boundary locations and band widths to each mixture under a fixed band-token budget. DBS predicts content-conditioned boundary scores within frequency blocks, selects a prescribed number of boundaries using hard Top-$K$, and applies content-dependent pooling to obtain compact band tokens. After separation, it smooths and expands these tokens to the original STFT grid, refines them along frequency, and decodes complex masks with a frequency-shared head. DBS improved separation over the corresponding static models on DnR v2, MUSDB18-HQ, and EchoSet, with DBS-BSRNN achieving relative gain of 29.0% in average SI-SDR on DnR v2. These results and boundary analyses on DnR v2 show that content-adaptive frequency partitioning benefits multiple separator backbones under a fixed band-token budget. Code and audio demos are available at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2610.14993
PDF
-
HiPAD: A Unified Gait, Audio-Visual, and Personality Fusion Network for Depression Detection, with a Hierarchical Personality Ablation and the Calibration Limits of Domain-Adapted LLMs
2026-10-10 14:35:22Karol Chlasta, Piotr StruzikAbstract
We present HiPAD, a single fusion network submitted to all six sub-tracks of the MPDD-AVG 2026 depression-detection challenge, covering the Elder and Young cohorts and three modality sets (gait, audio-visual, and both), each combined with personality descriptions. The network encodes each available modality with a shared BiLSTM, zero-pads absent modalities, and applies cross-modal attention only when at least two of gait, audio and video are present. Personality, which dominates our results, is kept out of the attention and fused by concatenation. Because the architecture is identical in every sub-track, differences between them reflect the data rather than the model, and we use this to study three questions. Our main finding is that the personality description, roughly 150 tokens about who the subject is, carries more depression signal than audio, video, and gait combined: gait plus personality alone (TLS 0.647) beats every configuration that includes audio and video. First, a four-level ablation of the personality description (L0-L3) quantifies this: removing every personality-derived feature, including the Big Five scores that earlier MPDD ablations keep, lowers the Track-Level Score by 0.499 to 0.507 and leaves the model at chance. Second, domain-adapting the personality encoder with LoRA (7B) or QLoRA (72B) on PHQ-8/9 data raises binary Cohen's $\kappa$ by 0.13 but lowers the overall score by distorting the PHQ-9 regression; a frozen RoBERTa-large encoder does better at this sample size. Third, at fixed architecture, handcrafted clinical features (eGeMAPS, OpenFace action units) outperform generic deep features (wav2vec2, DenseNet) on the audio-visual tracks. The unified system reaches a cross-track G-Score of 0.482; with under 100 subjects per cohort, the limiting factors are the size of the representation and the calibration of the outputs, not the choice of architecture.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14961
PDF
-
Transfer Learning for Rhythm Game Chart Generation from Music Transcription and Across Games
2026-10-10 14:26:30Sojeong An, Nathan Pruyne, Chris Donahue, Dasaem JeongAbstract
Rhythm games ask the player to act in time with a recording, following a chart: a sequence of actions, each with a time and, depending on the game, a lane, type, or playfield position. Charts are hand-written per recording and difficulty. Automatic generation from audio has been studied with pipeline and end-to-end approaches, and recent work reuses pre-trained audio encoders, but never compares them with one trained from scratch. None of these targets automatic music transcription (AMT), yet most chart times are note onsets, which AMT models locate. We freeze a pre-trained audio encoder, an AMT model in particular, and train only a linear projection and a Transformer decoder on charts. With MuScriptor, this fits human charts more closely than encoders trained from scratch and places more accurate note types than MERT, a music encoder pre-trained without transcription labels. Joint training on three osu! modes raises event F1 on the mode with the fewest charts from 0.492 to 0.506, while charts from Clone Hero, similar in layout, lower it to 0.482. Transfer between games thus appears to depend on charting style and the music covered.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14955
PDF
-
SemiCodec: Bridging Speech Tokenization and High-fidelity Reconstruction with Semi-discrete Representations
2026-10-10 11:52:24Kecan Mao, Nan Xu, Zhaoyang Meng, Zhengyao Ma, Yingming Gao, Yi Zhao, Ya LiAbstract
Speech representation learning requires a discrete modeling interface while preserving high waveform fidelity. Existing approaches rarely achieve both: discrete tokens ease modeling but lose fine-grained acoustic information, while continuous latents preserve fidelity but require specialized generative objectives. We introduce SemiCodec, a novel Semi-discrete neural speech Codec with synchronized discrete tokens and continuous latents. Direct joint optimization may create a reconstruction shortcut through the continuous pathway, leaving the discrete tokens insufficiently grounded. To address this issue, we propose a Preserve-then-Ground training paradigm that first establishes a high-fidelity synthesis space and then grounds SSL-guided discrete tokens within it. Experiments show that SemiCodec outperforms fully discrete codecs in reconstruction quality and intelligibility while remaining competitive with continuous representations in acoustic fidelity and speaker preservation. Audio samples are available at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2610.14834
PDF
-
Cross-Lingual TTS for Indian Languages: A Benchmark and Inference-Time Duration Estimation Strategies
2026-10-10 11:41:56Adarsh Arigala, Arjun Gangwar, Srinivasan UmeshAbstract
Recent alignment-free Text-to-Speech (TTS) models achieve high synthesis quality by removing explicit alignment modules and simplifying training. However, they often struggle in cross-lingual scenarios due to differences in character durations across languages, leading to duration mismatch and degraded speech quality. Existing approaches typically address this issue using explicit duration predictors, increasing model complexity. Systematic evaluation of TTS performance for Indian languages in cross-lingual settings remains under-explored due to the lack of standardized evaluation benchmarks. In this work, we construct a cross-lingual evaluation benchmark covering English and nine Indian languages and propose a language-aware inference-time duration estimation strategy for alignment-free TTS that incorporates language-specific duration-per character statistics without retraining or architectural modifications. We further develop heuristic formulations that refine duration estimation based on language-dependent duration variations. Experiments on bilingual and multilingual A-DMA models demonstrate improved speech naturalness, intelligibility, and speaker similarity while reducing duration mismatch.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14826
PDF
