Abstract
Self-supervised visual pre-training methods face an inherent tension: contrastive learning (CL) captures global semantics but loses fine-grained detail, while masked image modeling (MIM) preserves local textures but suffers from "attention drift" due to semantically-agnostic random masking. We propose C2FMAE, a coarse-to-fine masked autoencoder that resolves this tension by explicitly learning hierarchical visual representations across three data granularities: semantic masks (scene-level), instance masks (object-level), and RGB images (pixel-level). Two synergistic innovations enforce a strict top-down learning principle. First, a cascaded decoder sequentially reconstructs from scene semantics to object instances to pixel details, establishing explicit cross-granularity dependencies that parallel decoders cannot capture. Second, a progressive masking curriculum dynamically shifts the training focus from semantic-guided to instance-guided and finally to random masking, creating a structured learning path from global context to local features. To support this framework, we construct a large-scale multi-granular dataset with high-quality pseudo-labels for all 1.28M ImageNet-1K images. Extensive experiments show that C2FMAE achieves significant performance gains on image classification, object detection, and semantic segmentation, validating the effectiveness of our hierarchical design in learning more robust and generalizable representations.
Abstract (translated)
自监督视觉预训练方法面临着固有的矛盾：对比学习（CL）能够捕捉全局语义信息，但会丢失细粒度的细节；而掩码图像建模（MIM）则能保留局部纹理特征，但由于采用无意义的随机掩码策略，会导致“注意力漂移”问题。为此，我们提出了一种名为C2FMAE（从粗到精的掩码自动编码器）的方法，通过显式学习三种数据粒度层次化的视觉表示来解决这一矛盾：语义掩码（场景级别）、实例掩码（对象级别）和RGB图像（像素级别）。为了严格贯彻自上而下的学习原则，C2FMAE引入了两个协同创新： 1. 级联解码器按顺序从场景语义重建至物体实例再到像素细节，建立了明确的跨粒度依赖关系，这是并行解码器所无法捕捉到的。 2. 动态掩码课程学习策略逐步将训练焦点由以语义指导为主转向以实例指导为主，最后转为随机掩码，从而构建了一个从全局上下文到局部特征的学习路径。 为了支持这一框架，我们还构建了大规模多粒度数据集，并为此数据集中所有128万张ImageNet-1K图像生成高质量的伪标签。广泛的实验结果表明，C2FMAE在图像分类、目标检测和语义分割任务上实现了显著性能提升，验证了我们的分层设计在学习更具鲁棒性和泛化能力表示的有效性。
URL
https://arxiv.org/abs/2603.09955
