Abstract
We propose a pre-trained BERT-like model for symbolic music understanding that achieves competitive performance across a wide range of downstream tasks. To achieve this target, we design two novel pre-training objectives, namely token correction and pianoroll prediction. First, we sample a portion of note tokens and corrupt them with a limited amount of noise, and then train the model to denoise the corrupted tokens; second, we also train the model to predict bar-level and local pianoroll-derived representations from the corrupted note tokens. We argue that these objectives guide the model to better learn specific musical knowledge such as pitch intervals. For evaluation, we propose a benchmark that incorporates 12 downstream tasks ranging from chord estimation to symbolic genre classification. Results confirm the effectiveness of the proposed pre-training objectives on downstream tasks.
Abstract (translated)
我们提出了一种类似BERT的预训练模型，用于符号音乐的理解，并在广泛的下游任务中实现了具有竞争力的表现。为了达成这一目标，我们设计了两个新颖的预训练目标，即标记校正和钢琴卷帘预测。首先，我们在音符标记的一部分上添加少量噪声以进行数据扰动，然后训练模型去除这些被污染的标记中的噪声；其次，我们也训练模型从受干扰的音符标记中预测出小节级及局部钢琴卷帘衍生表示。我们认为，这些目标能够引导模型更好地学习特定的音乐知识，例如音高间隔等。 为了评估我们的模型，我们提出了一个包含12个下游任务的新基准测试集，这些任务涵盖了从和弦估计到符号风格分类等多个方面。实验结果证实了所提出的预训练目标在下游任务中的有效性。
URL
https://arxiv.org/abs/2507.04776
