Abstract
Since the introduction of Masked Autoencoders, various improvements to masking techniques have been explored. In this paper, we rethink masking strategies for audio representation learning using masked prediction-based self-supervised learning (SSL) on general audio spectrograms. While recent informed masking techniques have attracted attention, we observe that they incur substantial computational overhead. Motivated by this observation, we propose dispersion-weighted masking (DWM), a lightweight masking strategy that leverages the spectral sparsity inherent in the frequency structure of audio content. Our experiments show that inverse block masking, commonly used in recent SSL frameworks, improves audio event understanding performance while introducing a trade-off in generalization. The proposed DWM alleviates these limitations and computational complexity, leading to consistent performance improvements. This work provides practical guidance on masking strategy design for masked prediction-based audio representation learning.
Abstract (translated)
自掩码自编码器提出以来，研究者们对掩码技术进行了多种改进。本文针对通用音频频谱图，重新思考基于掩码预测的自监督学习（SSL）中的掩码策略。尽管近期信息感知掩码技术备受关注，但我们发现其计算开销显著。基于此观察，本文提出分散加权掩码（DWM）——一种轻量级掩码策略，利用音频内容频率结构中固有的频谱稀疏性。实验表明，近期SSL框架中常用的反向块掩码虽能提升音频事件理解性能，却以牺牲泛化能力为代价。所提出的DWM缓解了这些限制并降低计算复杂度，实现了稳定的性能提升。本研究为基于掩码预测的音频表征学习中的掩码策略设计提供了实用指导。
URL
https://arxiv.org/abs/2603.23810
