Abstract
At the most basic level, pixels are the source of the visual information through which we perceive the world. Pixels contain information at all levels, ranging from low-level attributes to high-level concepts. Autoencoders represent a classical and long-standing paradigm for learning representations from pixels or other raw inputs. In this work, we demonstrate that autoencoder-based self-supervised learning remains competitive today and can produce strong representations for downstream tasks, while remaining simple, stable, and efficient. Our model, codenamed "Pixio", is an enhanced masked autoencoder (MAE) with more challenging pre-training tasks and more capable architectures. The model is trained on 2B web-crawled images with a self-curation strategy with minimal human curation. Pixio performs competitively across a wide range of downstream tasks in the wild, including monocular depth estimation (e.g., Depth Anything), feed-forward 3D reconstruction (i.e., MapAnything), semantic segmentation, and robot learning, outperforming or matching DINOv3 trained at similar scales. Our results suggest that pixel-space self-supervised learning can serve as a promising alternative and a complement to latent-space approaches.
Abstract (translated)
在最基本的层面上，像素是承载我们感知世界的视觉信息的来源。每个像素包含了从低级属性到高级概念的各种信息层次。自编码器是一种经典的、长期存在的范式，用于从像素或其它原始输入中学习表示方法。在这项工作中，我们展示了基于自编码器的自我监督学习在今天依然具有竞争力，并能为下游任务生成强大的表示方法，同时保持简单性、稳定性和效率。我们的模型名为“Pixio”，是一个增强版的掩码自动编码器（MAE），配备了更困难的预训练任务和更有能力的架构。该模型是在使用最小人工管理策略进行自我筛选的20亿张网络抓取图像上进行训练的。在各种下游任务中，包括单眼深度估计、前馈3D重建、语义分割以及机器人学习等方面，Pixio表现出了竞争力，并且在类似规模下训练的DINOv3模型上超过了或与之持平的表现。我们的结果表明，在像素空间中的自我监督学习可以作为潜在空间方法的一个有前途的替代和补充方案。
URL
https://arxiv.org/abs/2512.15715
