Abstract
Music representation learning is central to music information retrieval and generation. While recent advances in multimodal learning have improved alignment between text and audio for tasks such as cross-modal music retrieval, text-to-music generation, and music-to-text generation, existing models often struggle to capture users' expressed intent in natural language descriptions of music. This observation suggests that the datasets used to train and evaluate these models do not fully reflect the broader and more natural forms of human discourse through which music is described. In this paper, we introduce MusicSem, a dataset of 32,493 language-audio pairs derived from organic music-related discussions on the social media platform Reddit. Compared to existing datasets, MusicSem captures a broader spectrum of musical semantics, reflecting how listeners naturally describe music in nuanced and human-centered ways. To structure these expressions, we propose a taxonomy of five semantic categories: descriptive, atmospheric, situational, metadata-related, and contextual. In addition to the construction, analysis, and release of MusicSem, we use the dataset to evaluate a wide range of multimodal models for retrieval and generation, highlighting the importance of modeling fine-grained semantics. Overall, MusicSem serves as a novel semantics-aware resource to support future research on human-aligned multimodal music representation learning.
Abstract (translated)
音乐表示学习是音乐信息检索和生成的核心。虽然近期在多模态学习方面的进展提升了文本与音频之间的对齐能力，特别是在跨模式音乐检索、从文本到音乐的生成以及从音乐到文本的生成等任务中，现有的模型往往难以捕捉用户在其自然语言描述中的意图表达。这一观察表明，用于训练和评估这些模型的数据集未能充分反映人类通过广泛且自然的语言形式来描述音乐的方式。 在本文中，我们介绍了MusicSem数据集，该数据集包含从社交媒体平台Reddit上收集的32,493对语言-音频配对，这些内容源自有机的音乐相关讨论。与现有的数据集相比，MusicSem捕捉到了更广泛的音乐语义范围，反映了听众如何以细腻且以人为中心的方式自然地描述音乐。为了结构化这些表达，我们提出了一个包含五种语义类别的分类法：描述性、氛围性、情境性、元数据相关的和上下文性的。 除了构建、分析并发布MusicSem之外，我们还利用该数据集评估了广泛的多模态模型在检索和生成任务中的表现，并强调了建模细粒度语义的重要性。总体而言，MusicSem作为一个新颖的语义感知资源，旨在支持未来关于与人类行为一致的多模态音乐表示学习的研究工作。
URL
https://arxiv.org/abs/2602.17769
