Abstract
Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purpose architectures and pretraining pipelines. As a result, Active-Vision Foundation Models (AVFMs) have remained unexplored. We introduce CanViT, the first task- and policy-agnostic AVFM. CanViT uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace, the canvas. Efficient interaction with this high-capacity working memory is supported by Canvas Attention, a novel asymmetric cross-attention mechanism. We decouple thinking (backbone-level) and memory (canvas-level), eliminating canvas-side self-attention and fully-connected layers to achieve low-latency sequential inference and scalability to large scenes. We propose a label-free active vision pretraining scheme, policy-agnostic passive-to-active dense latent distillation: reconstructing scene-wide DINOv3 embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths. We pretrain CanViT-B from a random initialization on 13.2 million ImageNet-21k scenes -- an order of magnitude more than previous active models -- and 1 billion random glimpses, in 166 hours on a single H100. On ADE20K segmentation, a frozen CanViT-B achieves 38.5% mIoU in a single low-resolution glimpse, outperforming the best active model's 27.6% with 19.5x fewer inference FLOPs and no fine-tuning, as well as its FLOP- or input-matched DINOv3 teacher. Given additional glimpses, CanViT-B reaches 45.9% ADE20K mIoU. On ImageNet-1k classification, CanViT-B reaches 81.2% top-1 accuracy with frozen teacher probes. CanViT generalizes to longer rollouts, larger scenes, and new policies. Our work closes the wide gap between passive and active vision on semantic segmentation and demonstrates the potential of AVFMs as a new research axis.
Abstract (translated)
主动视觉通过顺序、局部的注视点，有望实现高效且符合生物机理的感知，但一直缺乏可扩展的通用架构与预训练流程，导致主动视觉基础模型（AVFM）尚未被探索。本文提出首个任务与策略无关的AVFM——CanViT。其采用场景相对位置编码（scene-relative RoPE），将视网膜拓扑视觉Transformer主干与空间拓扑的全场景潜在工作空间（画布）相绑定。通过新型非对称交叉注意力机制——画布注意力（Canvas Attention），实现了与该高容量工作记忆的高效交互。通过解耦思考（主干层级）与记忆（画布层级），并移除画布端的自注意力与全连接层，实现了低延迟的顺序推理与对大规模场景的扩展性。我们提出了一种无标签的主动视觉预训练方案：策略无关的从被动到主动的稠密潜在蒸馏——即从随机位置、缩放级别与长度的低分辨率注视点序列中，重建全场景的DINOv3嵌入特征。我们在1320万张ImageNet-21k图像（比以往主动模型多一个数量级）及10亿个随机注视点上，使用单张H100显卡在166小时内完成了CanViT-B的随机初始化预训练。在ADE20K分割任务中，冻结的CanViT-B仅凭单次低分辨率注视即达到38.5% mIoU，不仅以19.5倍更少的推理浮点运算次数（无需微调）超越了最佳主动模型的27.6%，还超越了其浮点运算或输入匹配的DINOv3教师模型。增加注视次数后，CanViT-B可达45.9%的ADE20K mIoU。在ImageNet-1k分类任务中，CanViT-B使用冻结的教师探针达到81.2%的top-1准确率。CanViT可泛化至更长推理轨迹、更大场景与新策略。本研究弥合了被动与主动视觉在语义分割上的显著差距，并展示了AVFM作为新研究方向的潜力。
URL
https://arxiv.org/abs/2603.22570
