arXiv_SD
-
Composer2Vec: A Continuous Embedding Space of Composer Style Learned from Symbolic Melody Generation
2026-09-17 05:09:39Sakutaro Nishio, Osamu IchikawaAbstract
We analyze the composer embeddings learned by a composer-conditioned Transformer as a continuous latent space of compositional style, rather than merely as an internal representation for generation. A model that recursively predicts melody continuations was trained on melodic sequences extracted from MIDI data, conditioned on composer identity (124 composers). Principal component analysis of the learned composer embedding matrix (124x128) shows that the first principal component correlates strongly with composer birth year (r = -0.884, p < 0.001, n = 123), a stronger correlation than we obtain by applying the same PC1-birth-year analysis to existing general-purpose audio-text embeddings (CLAP, MuQ-MuLan) trained on unrelated audio-text corpora, not on symbolic melody generation. A shuffle test (2,000 permutations) confirms that the Silhouette score for stylistic-period labels is statistically significant (0.0110, p < 0.001). We further show that vector arithmetic in the embedding space captures meaningful stylistic relationships between composers. These results suggest that composer embeddings, learned without supervision beyond composer identity, form an interpretable latent space that captures musical-historical structure.
Abstract (translated)
URL
https://arxiv.org/abs/2609.20893
PDF
-
A State-Space Model of Figured-Bass Realization: Local Constraints, Coupled Voices, and Polynomial-Time Solvability
Abstract
Figured-bass realization can be described as a sequence of choices constrained both within each sonority and between successive sonorities. This paper gives an explicit mathematical model of a restricted, examination-style four-part realization problem. Pitch spelling, range, chord membership, doubling, omission, spacing, crossing, overlap, melodic motion, consecutive perfect intervals, and selected resolution requirements are expressed as predicates. We distinguish hard constraints from optional preference costs. Four labeled notes are represented visually as the vertices of a quadrilateral and computationally as one ordered voicing state. Legal progressions become paths through a layered graph. We prove that feasibility and minimum-cost realization are polynomial-time problems for a fixed number of voices with explicit finite note domains and fixed local rules. For fixed ranges, a fixed note alphabet, and adjacent-event rules, the number of graph operations is linear in the number of events. Worked two-, four-, and eight-beat examples illustrate legality, optimization, and the failure of a greedy choice. The result concerns the stated formal model; it is not a claim that every musical judgment is captured by local predicates.
Abstract (translated)
URL
https://arxiv.org/abs/2609.19397
PDF
-
Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding
2026-09-16 14:00:38Benedikt Mayrhofer, Enrique Orozco Olivares, Franz Pernkopf, Philipp Aichinger, Martin Hagm\"ullerAbstract
Self-supervised learning (SSL) has improved speech representations, yet performance degrades in pathological domains such as electrolaryngeal (EL) speech, and the computational footprint of SSL models limits their applicability in real-time, on-device deployment. We propose a multi-teacher knowledge distillation framework to train a lightweight, streaming content encoder that generalizes across healthy (HE) and EL speech. Two teachers are distilled progressively: a frozen SSL model providing discrete phonetic cluster targets from HE speech, and an EL-fine-tuned speech recognition model supplying continuous bottleneck feature targets. Evaluated via downstream speech recognition, our approach reduces the EL word error rate to 21.2%, compared to 39.3% for the strongest zero-shot SSL baseline. Among causal convolutional, Transformer, Conformer, and Mamba-based student architectures, a Mel-Conformer achieves the best combination of EL accuracy and computational efficiency. The final encoder contains 21.9,M parameters and runs at a real-time factor of 0.30 under ONNX Runtime on a single CPU core.
Abstract (translated)
URL
https://arxiv.org/abs/2609.18686
PDF
-
TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking
2026-09-16 12:44:11Giorgia Adorni, Michela Papandrea, Battista Rimoldi, Tiziano LeidiAbstract
Text-to-music (TTM) systems are increasingly used to generate musical audio from natural-language descriptions. Robust evaluation is therefore essential, yet reliable performance comparison remains challenging. This difficulty stems from differences in system architecture, supported conditioning information, and access mode, as well as heterogeneous and fragmented metrics that cannot be applied uniformly across systems. To address these challenges, we introduce TTM-Bench, a framework that defines a common protocol for systematic, reproducible performance benchmarking of contemporary TTM systems. It evaluates performance along two dimensions: musical-content alignment, quantified by interpretable semantic, genre, and musical-descriptor agreement scores against a common musical specification and summarized by an aggregate score; and computational efficiency, characterized by generation latency and real-time factor, alongside resource use for local models and cost for hosted services. We demonstrate the framework through a preliminary comparative case study, illustrating the complementary evidence captured by these dimensions. The results show that higher musical-content alignment does not systematically coincide with lower computational demands, highlighting the importance of assessing TTM performance through distinct, interpretable measures rather than a reductive overall indicator.
Abstract (translated)
URL
https://arxiv.org/abs/2609.18585
PDF
-
Encypher: Shared Agency and Social Presence in Collaborative Music Generation for Dance Cyphers
Abstract
Music and dance are social practices of expression and connection, yet most HCI work in human-AI co-creation centers the solo performer. As generative music matures, we ask not only what AI can compose but what social encounters it can organize around sound. We present Encypher, a collaborative generative music system that translates collective movement qualities into text prompts conditioning real-time music generation for dance cyphers. Through five weeks of co-design with local dancers, a user study with unacquainted participants, a public museum event, and a live performance, we found that users developed shared agency, perceiving the music as a response to the room's energy. While newcomers felt uncertain, the system fostered social presence by prompting them to look to each other for cues. By treating sociality as a design concern rather than a downstream effect, we offer a framework and design implications for AI systems for collaborative, embodied expression.
Abstract (translated)
URL
https://arxiv.org/abs/2609.18062
PDF
-
The Unbearable Weight: Scaling Models and Methods for UAV Audio Classification
2026-09-15 22:22:21Andrew P. Berg, Qian Zhang, Mia Y. WangAbstract
As unmanned aerial vehicles (UAVs) become increasingly prevalent in consumer and defense settings, classifying them reliably from limited, modality-specific data is an urgent challenge. The dominant approach, large pretrained networks fully fine-tuned on task data, carries a substantial computational and memory weight that is hard to bear in resource-constrained UAV deployments, where edge inference and rapid retraining for emerging platforms are both required. This paper systematically scales across both model architectures and fine-tuning methods for UAV audio classification, asking when that weight is justified and when lighter alternatives prevail. Using a custom dataset of 3,100 audio clips spanning 31 drone classes, we evaluate transformer (ViT, AST) and convolutional (custom CNN, ResNet-18/152, MobileNet-V3-S/L, EfficientNet-B0/B7) backbones under full fine-tuning, classifier-only fine-tuning, and four parameter-efficient fine-tuning (PEFT) methods: SSF, IA3, OFT, and selective batch-norm tuning. All configurations are evaluated with 5-fold cross-validation across accuracy, training time, trainable-parameter share, and inference-time memory footprint. Selective batch-norm fine-tuning of EfficientNet-B7 with three-fold augmentations achieves the highest validation accuracy (97.65% +- 0.30) while updating under 0.5% of model parameters. Across the sweep, lightweight CNNs consistently outperform transformers on both accuracy and efficiency. For UAV audio classification under data scarcity, scaling the method outperforms scaling the model.
Abstract (translated)
URL
https://arxiv.org/abs/2609.17884
PDF
-
SpiroPhonia: Non-Invasive Respiratory Health Assessment from Spontaneous Speech
2026-09-15 15:49:37Roksana Khanom, Shafia Supty, Nirupam Roy, Ashok AgrawalaAbstract
Chronic Obstructive Pulmonary Disease (COPD) remains a major global health challenge, emphasizing the need for accessible and non-invasive detection. Since speech production is fundamentally linked to respiratory physiology, its disruptions can serve as indirect indicators of pulmonary impairment. This study introduces SpiroPhonia, a machine learning framework that leverages spontaneous speech for respiratory health assessment. We evaluated SpiroPhonia on a new dataset of 201 speakers (102 with COPD, 99 healthy controls). By integrating statistical analysis with recursive feature selection, we identified a compact set of discriminative speech markers. Our best model achieved 78% accuracy, 80% F1-score, and 87% AUC. This performance on spontaneous speech is competitive with methods using controlled laboratory recordings. Findings demonstrate that everyday speech encodes robust respiratory biomarkers, paving the way for continuous health monitoring via voice-enabled technologies.
Abstract (translated)
URL
https://arxiv.org/abs/2609.17350
PDF
-
The Internet Archive Music Dataset
2026-09-15 15:15:18Paraskevas Stamatiadis (S2A, LTCI, IDS), Bernardo V Miranda (S2A, LTCI, IDS), Cl\'ementine Berger (LTCI, IP Paris, S2A, IDS, IMT), Ga\"el Richard (S2A, LTCI, IDS), Mathieu Fontaine (S2A, LTCI, IDS), Slim Essid (IDS, S2A, LTCI)Abstract
We introduce the Internet Archive Music Dataset (IAMD), a large-scale collection of captioned music segments derived from the Internet Archive. To the best of our knowledge, IAMD constitutes the largest publicly available music-caption dataset to date with over 34,000 hours of audio, providing a valuable benchmark for training and evaluating music understanding and generative models. The dataset is built from content declared to be distributed under Creative Commons licenses, and cross-referencing with MusicBrainz is done to improve license information reliability. To annotate IAMD, we present an automatic captioning pipeline that augments base captions produced by an audio-language model (ALM) with textual metadata sourced from the Internet Archive and imputed metadata obtained using audio classification models. Caption quality is assessed objectively and subjectively, and results indicate that the annotation pipeline is reliable and does not degrade caption quality with scaling.
Abstract (translated)
URL
https://arxiv.org/abs/2609.20870
PDF
-
Self-Distilled Pronunciation and Accent Control for Neural Text-to-Speech
Abstract
Text-to-speech that reads raw text has no lexicon: a rare word is read as guessed. Remedies train a reading-and-accent channel on recorded speech or edit words one at a time from exemplars. We do neither. The frozen backbone reads a sentence containing a common word it already says correctly, and its own output then serves as the teacher for the same sentence, with that word replaced by a tagged, accented reading; this training pair is the whole idea. On Sarashina2.2-TTS, screened raters at Fleiss' kappa = 0.85 hear the prescribed accent on 0.89 of unseen words against 0.57 for kana, which cannot express one; kana wins no pair; naturalness is not measurably hurt. Moved untuned to autoregressive, diffusion, and encoder-decoder backbones, it transfers reading, 0.25 to 0.47 above no edit on 319 words, and on CosyVoice 2 accent on two words in three, but not on Irodori; the paper locates why.
Abstract (translated)
URL
https://arxiv.org/abs/2609.17234
PDF
-
Taming Long-form Text-to-Speech
2026-09-15 11:03:52Rongxiang Wang, Berkin Durmus, Aysegul Orhon, Eduardo Pacheco, Atila OrhonAbstract
Long-form text-to-speech (TTS) enables multi-turn conversations with consistent prosody and higher quality voice cloning from longer reference audio. Recent open-weights autoregressive TTS models such as Qwen3-TTS and VoxCPM2 attain state-of-the-art word error rate (WER) and speaker similarity (SIM) on short-form prompts but significantly deteriorate when used with long-form prompts. We propose Localized Attention-Constrained Inference (LACI), an inference-only method to detect TTS errors in near real-time, roll back to the error onset and regenerate with temporary guardrails, adding negligible computational overhead. Using LACI, we improve worst-of-N WER across 10 RNG seeds for Qwen3-TTS-0.6B from 35.2% to 3.4% on prompts longer than 1500 words, even surpassing its short-form reliability of 5.4\% on prompts with fewer than 500 words. To demonstrate the efficacy of LACI on voice cloning reliability, we propose a sliding-window version of the SIM metric that we call wSIM. wSIM exposes several novel failure patterns that are not captured by SIM. LACI improves worst-of-N wSIM from 0.01 to 0.47 on 120 seconds of reference audio while reducing the rate of catastrophic generations with WER above 30% from 26% to below 1%
Abstract (translated)
URL
https://arxiv.org/abs/2609.16989
PDF
-
Multimodal Emergency Vehicle Classification via Audio-Visual Transformers and Knowledge Distillation
2026-09-15 02:34:07Vijay John, Amar DabajaAbstract
Emergency vehicle detection in autonomous driving is a safety-critical perception task that demands robustness under diverse and adverse real-world conditions. Existing approaches rely on a single modality, either audio or video, which leads to systematic failure when that modality is degraded: microphone-based systems fail in noisy urban environments, and camera-based systems fail at night or under occlusion. This report presents AVNet, a multimodal audio-visual transformer that classifies emergency vehicles (ambulance, fire engine, police car) and road background using both audio and video, while gracefully handling the absence of either modality at inference time. AVNet introduces three key contributions: (1) a temporally aligned cross-modal fusion module that performs second-level cross-attention between audio spectrogram tokens and video frame tokens, exploiting their exact temporal correspondence without any learned alignment mechanism; (2) learned null embeddings that substitute for missing modality tokens, enabling a single unified model to operate in audio-only, video-only, or joint audio-visual mode without retraining; and (3) a knowledge distillation training strategy in which specialist unimodal teacher models transfer inter-class dark knowledge into the multimodal student fusion branch via soft probability targets. Evaluated on 281 clips from the Google AudioSet dataset, AVNet achieves 66.6% overall accuracy in audio-visual mode, outperforming the audio-only branch by +10.4% and the video-only branch by +15.0%. The largest per-class gain is observed for the hardest class, Ambulance, where fusion achieves +29.5% over either unimodal branch alone, demonstrating that the two modalities provide complementary information that the aligned cross attention mechanism successfully exploits.
Abstract (translated)
URL
https://arxiv.org/abs/2609.16535
PDF
-
SongCraft: Unified Song Generation and Editing with Reconstructive Learning
2026-09-14 20:25:58Haohe Liu, Varun Nagaraja, Gael Le Lan, Xinhao Mei, Zhaoheng Ni, Vikas Chandra, Abdelrahman Mohamed, Yangyang ShiAbstract
Song generation and editing have mostly been treated as separate tasks. Existing editing methods often require noise injection and regeneration or curated paired training data. We propose a unified approach for song generation and editing based on reconstructive pretraining, in which a model is trained to reconstruct audio from varying numbers of interpretable conditions. With conditions such as text and lyrics, the model learns to generate diverse songs. With dense conditions specifying fine-grained music attributes, the model learns to reconstruct the target and enables editing by modifying any single attribute while keeping others fixed. This leads to SongCraft, a latent flow matching based model trained for both generation and fine-grained editing. To improve song generation quality, we further introduce word-level phoneme alignment that improves pronunciation learning and accelerates convergence, beat conditioning that improves general musicality, and representation alignment on VAE latent space that produces semantically meaningful latents for improved generation quality. Experiments show that SongCraft achieves the lowest word error rate among evaluated song generation baselines while maintaining competitive audio quality. We further show that a single model can support editing of lyrics, vocal melody, beats, and singer identity, and we also study the trade-off between reconstruction quality and editability.
Abstract (translated)
URL
https://arxiv.org/abs/2609.16315
PDF
-
Sectional Structure and Emotional Dynamics in Chinese Pop Songs: An Empirical Analysis of Valence-Arousal Trajectories across 100 Songs
Abstract
Music Emotion Recognition (MER) aims to identify and represent emotional information in music through computational methods and is an important research area within Music Information Retrieval (MIR). To address the limited consideration of song sectional structure in existing dynamic MER research, this study examines 100 Chinese pop songs by aligning 1,046 manually annotated sections with continuous Valence-Arousal (VA) trajectories and analyzing them from the perspectives of section type, adjacent section transitions, repeated sections, and whole-song trajectories. The results show that emotional differences across sections are reflected more strongly in Arousal. Verse typically forms a relatively low-activation baseline, Pre-chorus exhibits a progressive buildup, and Chorus produces a more pronounced high-arousal arrival, while Interlude, Bridge, and Outro tend to show transitional, divergent, and closing functions, respectively. Although whole-song sectional configurations are diverse, high-frequency local transitions are relatively concentrated. High-arousal positions occur more often in the later part of a song, but are not fixed to the final Chorus. Based on these findings, this study summarizes the emotional organization of the selected pop songs as an empirical framework of "local cycles-global accumulation", in which local sectional cycles are accompanied by emotional pullbacks, while the overall trajectory exhibits a later-stage rise in VA and a tendency for high-Arousal positions to occur later in the song.
Abstract (translated)
URL
https://arxiv.org/abs/2609.15675
PDF
-
OpenEnded: An Open-Response Speech Corpus for Speaking Proficiency Assessment with Human Annotations and ALM Supervision
2026-09-14 14:47:18Yu-Wen Chen, Eric Zhou, Evelyn Ding, Tianyi Shen, Zhou Yu, Julia HirschbergAbstract
The development of automated speaking assessment (ASA) is limited by the scarcity of public datasets, with most existing work relying on read-aloud speech, which limits applicability to real-world communication scenarios. In this work, we introduce OpenEnded, a corpus of English practice speech from Mandarin speakers in open-response tasks. Unlike prior open-response datasets that provide only holistic proficiency scores, OpenEnded offers utterance-level assessments of accuracy, fluency, and prosody. Approximately 10,000 utterances are collected and annotated using a hybrid framework: 1,000 are manually labeled via multi-rater scoring with discrepancy resolution to form a high-quality test set, while the remaining are pseudo-labeled by an audio language model (ALM) for training and development sets. We evaluate ALMs and existing ASA models on the OpenEnded test set and introduce VoxPA as an additional baseline. Results show that ALM-generated pseudo-labels improve training over original ALM scoring, while VoxPA achieves the best performance among all baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2609.15666
PDF
-
Graph Attention Design Choices Matter: A Controlled Study of LoRA-Adapted Audio Anti-Spoofing
2026-09-14 14:33:28Haoyu Wang, Jing Yang, Chenyu Liu, Yushan Du, Yifan Liao, Ningning Pan, Gongping Huang, Yu Zhao, Gang Li, Jian LuanAbstract
Audio anti-spoofing systems increasingly combine self-supervised learning, parameter-efficient fine-tuning, and graph-attention-based backends. However, performance gains in such systems are often entangled with concurrent changes in the backbone, fine-tuning strategy, and training protocol, making the independent contribution of graph attention design difficult to isolate. To address this issue, we conduct a systematic controlled study of the graph attention layer under a unified experimental setting. We decompose the layer into three independently testable design dimensions: scoring symmetry, temperature learnability, and routing granularity. These are instantiated as a concat-based scoring branch, a LearnT branch with learnable temperature, and a multi-temperature routing branch, respectively. Each dimension is implemented as an independently gated residual branch, enabling the evaluation of both individual variants and their combinations under the same experimental setting. Experiments on five evaluation sets with five random seeds show that the LearnT branch achieves the best average equal error rate (EER), yielding a 16.1% relative improvement over the baseline. In contrast, the multi-temperature routing branch does not improve average performance on its own, but substantially reduces cross-seed standard deviation when combined with the concat-based scoring branch. Moreover, two individually effective branches degrade performance when used together, resulting in a 25.6% relative deterioration compared with the baseline. This finding reveals strong non-additive interactions among graph attention design dimensions. Overall, the results suggest that, under parameter-constrained fine-tuning, improvements in graph attention layers depend more on capacity allocation and branch interaction than on simply adding more learnable parameters.
Abstract (translated)
URL
https://arxiv.org/abs/2609.15650
PDF
-
Timbre Analysis of the Hulusi, a Southwestern Chinese Free-Reed Instrument, using Machine Learning
Abstract
The hulusi is a wind instrument that was invented in Yunnan Province, China, and has become tremendously popular in recent years. It consists of a mouthpiece, a gourd, and three bamboo tubes, all with free reeds made of copper. The main bamboo tube in the middle has seven finger holes. In this instrument, the pipe length, not the free reed's eigenfrequency, determines the instrument's pitch, unlike, for example, with the Western accordion or the blues harp. In this study, a machine learning model implemented in the COMSAR framework (this https URL) was used to investigate the timbre characteristics of the \emph{hulusi} to cluster different instruments and pitches. The measured \emph{hulusi} pitches C, B, A, G, and F were analyzed according to seven psychoacoustic features, among which only the spectral centroid, sharpness, and fractal correlation dimension are shown to form pitch clusters. These timbre features were used to train Kohonen self-organizing maps (SOMs) for clustering. Brightness and sharpness analysis revealed that the highest pitches were less bright and less sharp than mid- and low-range pitches were. Furthermore, the fractal correlation dimension, which mainly determines the chaoticity of the initial transients, was the best-clustering timbre feature for the hulusi, with the highest pitches showing the least chaoticity. This result is supported by defining a cluster quality index for the SOMs.
Abstract (translated)
URL
https://arxiv.org/abs/2609.17612
PDF
-
Building a Dataset for Music Sample Identification
2026-09-14 12:27:55R. Oguz Araz, Xavier Lizarraga, Xavier Serra, Dmitry BogdanovAbstract
Sample identification (SI) is the task of matching an element of a musical work to its musically transformed versions used to create new works. The task has received little attention and lacks large-scale publicly available data. In this work, we mine sampling annotations from a music database and split them for training and evaluation. The resulting dataset is nearly three orders of magnitude larger than the existing SI benchmarks, with training, validation, and test sets of 114 k, 6 k, and 10 k tracks. We find that naively splitting the annotations places the same tracks in different sets. To avoid this, we construct a graph from the annotations and split it over connected components. We further find that a single mega-component contains half of the annotations, making component-wise splitting incompatible with balanced splits; we trim it, yielding a leakage-aware pipeline. We share the dataset for non-commercial scientific research purposes only and make the data-analysis and splitting code publicly available. We hope that our work fosters research on SI.
Abstract (translated)
URL
https://arxiv.org/abs/2609.15465
PDF
-
Listening for Airway Stenosis: A Foundation Model-Based Method for Rapid and Accessible Detection
2026-09-14 12:20:23Jean Groeninger, Zihao Zhao, Juliana de Castilhos, Sven Nebelung, Daniel TruhnAbstract
Airway stenosis can cause severe respiratory complications, yet its detection often relies on specialized examinations and medical imaging. This study explores the potential of acoustic AI for rapid and accessible airway stenosis detection using readily acquired patient voice recordings. We systematically investigate whether acoustic foundation models (AFMs) can extract acoustic representations associated with airway stenosis-related speech patterns. Experiments are conducted on a cohort of 748 participants from the Bridge2AI-Voice dataset, 134 with airway stenosis and 614 without. The best-performing model achieves an AUROC of 0.952 and an accuracy of 0.924 (means over five-fold cross-validation), highlighting the potential of AFMs to transfer beyond general-purpose speech applications to clinical diagnostic tasks. Further analysis reveals that the model primarily relies on connected-speech recordings rather than isolated acoustic tasks, such as sustained phonation and breathing. Overall, these results suggest that voice-based acoustic AI could complement existing diagnostic workflows by enabling rapid, low-burden, and widely accessible screening for airway stenosis.
Abstract (translated)
URL
https://arxiv.org/abs/2609.15453
PDF
-
BioDCASE: Active Learning for Bioacoustics
2026-09-14 09:14:50Ben McEwen, Rupa Kurinchi-Vendhan, Shiqi Zhang, Lukas Rauch, Marek Herde, Sara BeeryAbstract
Ecological monitoring increasingly relies on machine learning models, whose performance depends on the quality and quantity of labelled data. However, obtaining these labels is costly, particularly in passive acoustic monitoring, where vast amounts of data are collected but only a small proportion can feasibly be annotated. Active learning addresses this bottleneck by prioritizing which samples should be labelled. However, progress is difficult to measure, because published methods are evaluated under different models, budgets, evaluation metrics and datasets. To address this challenge, we present the 2026 Active Learning for Bioacoustics BioDCASE challenge: a systematic evaluation of sampling methods designed to identify effective AL strategies. Participant methods were evaluated across four subsets composed of terrestrial and marine data. Across ten proposed sampling methods from seven teams, the top-ranked method achieved an area under the learning curve 26.4 % higher than random sampling at the same annotation budget, averaged over four data subsets. Significant variation in performance was observed across subsets, with the top-performing submission achieving a 67.1 % gain for the HSN subset over random sampling and a gain of 8 % for the ATBFL subset. Top-ranking submissions combined multiple acquisition signals, and diversity-based selection outperformed pure uncertainty sampling. Furthermore, there is evidence that transitioning from diversity-based to uncertainty-based selection and explicitly reducing redundancy within acquisition batches improve model training. There is also initial evidence that larger acquisition batch sizes may be increasingly beneficial later in the labelling process.
Abstract (translated)
URL
https://arxiv.org/abs/2609.15255
PDF
-
Generating the Unheard: Phylogeny-Guided Latent Generation for Ancestral Sound Reconstruction
2026-09-14 08:57:25Tianyi Xu, Shrinaath Narasimhan, Evan Gorstein, Santiago Perea, Yunyi Shen, Claudia Sol\'is-LemusAbstract
What did an ancestral bird species sound like? Existing ancestral state reconstruction methods can infer low-dimensional traits such as morphological characters at internal nodes of a phylogenetic tree, but no one has tried to produce rich perceptual signals such as audio. Some of the challenges include inferred representations that are either too low-dimensional to decode or lie in non-generative feature spaces, so no method to date can produce ancestral audio. We introduce the first framework that generates plausible ancestral vocalizations. Our pipeline encodes bird recordings into a VAE latent space, learns a low-dimensional trait projection aligned with phylogenetic distances, performs ancestral inference in this trait space, and recovers decodable latents through an anchored inverse lift before emitting novel waveforms for each ancestral node. Because the entire pipeline stays within a decodable latent space, every internal node receives a genuinely new audio output representing plausible intermediate ancestral sounds unavailable to retrieval-based alternatives. Experiments on two phylogenetically distant bird clades, 21-species Tyrannidae and 19-species Paridae, show that our method is the only approach that simultaneously achieves genuine generation, phylogenetic consistency, and naturalistic audio quality across both datasets.
Abstract (translated)
URL
https://arxiv.org/abs/2609.15240
PDF
